Explainable abnormal gait detection method and model based on multi-task graph fusion learning

Abstract

The application discloses an interpretable abnormal gait detection method and model based on multi-task graph fusion learning, and the method is: constructing a gait graph and arranging multiple gait graphs according to time to obtain a time sequence gait graph; constructing an LRP-MGG neural network model; extracting gait time dependence features through the cumulative GRU block of the LRP-MGG neural network model; obtaining local space features embedded in the time dependence feature space by using a deep GCN module; performing graph-based gait mode classification through a DiffPool module; learning interpretable abnormal gait information through a CNN deep convolutional neural network model by using gait space-time significant difference features; and calculating the importance score of each feature by using a hierarchical relevance propagation LRP. The application combines the training of multiple tasks, reduces the amount and complexity of data, and thus improves the generalization ability and prediction ability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to an interpretable abnormal gait detection method and model based on multi-task graph fusion learning. BACKGROUND

[0002] Gait is the posture feature of human walking, which is the external manifestation of the mutual coupling of the relevant bones, muscles and joints of the human movement system under the coordination of the human nervous system, and is closely related to physiological function factors such as human nervous function, motor function, cognitive function and visual function. If the related physiological function is diseased, the gait will be abnormal. Accurate identification of abnormal gait and its changes is conducive to early diagnosis, early prevention, diagnosis and treatment, and rehabilitation evaluation of related diseases such as senile degenerative diseases and falls of the elderly.

[0003] In gait analysis, how to accurately distinguish and interpret abnormal gait patterns has always been a challenging problem to be solved. More and more researches are committed to finding methods to quantitatively analyze gait changes to avoid low accuracy caused by qualitative evaluation such as visual inspection and professional experience of clinicians. Obtaining accurate and reliable robust gait data is the first important step in quantifying abnormal gait. In most related researches, three-dimensional gait analysis systems based on motion capture are widely used to collect gait data. The system is usually integrated with high-speed camera-based motion capture systems, force platforms of plantar pressure sensors and electromyography (EMG) acquisition systems to simultaneously obtain different gait parameters such as spatiotemporal kinematic data, kinetic data and EMG data. These parameters mainly include gait cycle, walking speed, gait rhythm, step length, walking time, support phase, swing phase, joint angle, joint moment and foot-ground reaction force, which can accurately, effectively and reliably quantitatively analyze abnormal gait. However, advanced three-dimensional gait analysis systems have some limitations such as high cost and special laboratory environment requirements, which make it impossible to monitor gait changes in outdoor environments in real time. In recent years, wearable collectors based on MEMS sensors (accelerometers, gyroscopes, etc.) are more commonly used in related researches because of their low cost, ease of operation and ability to be worn on relevant human movement anatomical structures, which can be collected in outdoor and daily home living environments. Current related researches mostly focus on identifying abnormal gait conditions based on wearable sensors.

[0004] Furthermore, advanced graph neural network (Graph Neural Network) models are widely used in various abnormal gait detection tasks. Unlike deep learning models based on Euclidean space, Graph Neural Networks can learn compensatory information hidden in gait movements from nodes and edges. In particular, Graph Neural Networks can effectively construct gait dynamics and kinetic chain compensation models from the structural and functional aspects of the entire lower limb, acquiring significant differences hidden in the gait kinetic chain in a non-Euclidean metric space, significantly enhancing the model's generalization performance and accurately distinguishing subtle differences between different gait patterns. However, despite the excellent generalization performance of Graph Neural Networks in gait anomaly recognition, the cost is a more complex model architecture, further exacerbating the black-box nature of the model, making it difficult to understand how the model makes predictions and interprets gait changes, necessitating the search for new technological solutions.

[0005] Currently, interpretability methods for deep learning models include pre-interpretability and post-interpretability methods. Pre-interpretability methods aim to consider model interpretability from the initial design stage of the neural network, creating interpretable models. This involves adjusting the network structure before training, such as introducing additional network structures or changing the training strategy. Pre-interpretability methods have strong interpretability performance, but they are often limited to traditional machine learning or shallow neural network models, unable to effectively explain complex neural networks. Deep neural networks have good fitting and expressive capabilities, suitable for complex real-world scenarios. While traditional shallow machine learning models have good interpretability, they are difficult to apply to complex scenarios. To adapt to complex deep learning network models, more related technical solutions focus on post-interpretability methods. The basic idea is to provide interpretable information related to model decisions by designing additional technical means, without sacrificing the predictive performance of the deep neural network. Therefore, most current abnormal gait recognition models based on inertial sensors mainly focus on post-interpretability methods of deep learning.

[0006] Currently, many gait recognition tasks utilize interpretability (XAI) techniques to construct a multi-task learning problem that can both classify abnormal gait and interpret prediction results, unlike the post-explanation methods used in abnormal gait recognition (such as LIME, SHAP, and LRP). Some studies use LRP to generate heatmaps of the importance of features throughout the gait cycle, consistently identifying peak and trough regions as important features, effectively assessing the consistency of previous research results. Other related studies have validated the use of SHAP and LRP techniques to assess the importance of gait parameters in wearable gait recognition; in particular, LRP is suitable for deep learning analysis of the regional importance of the entire gait pattern input. While these studies effectively utilize XAI to interpret model outputs, they do not consider the actual needs of users. Currently, most interpretive AI system development is developer-centric rather than user-centric. Compared to the post-explanation provided by the model, medical professionals and patients are more concerned with identifying areas of interest and regions of abnormal gait changes requiring treatment. Summary of the Invention

[0007] The purpose of this invention is to provide an interpretable abnormal gait detection method and model based on multi-task graph fusion learning. This method combines the training of multiple tasks, reduces the amount and complexity of data, and helps the model learn both gait recognition and interpretability tasks better. This makes the model more robust on different tasks, thereby improving the model's generalization and prediction capabilities.

[0008] The technical solution adopted in this invention is:

[0009] An interpretable abnormal gait detection method based on multi-task graph fusion learning includes the following steps:

[0010] Step 1: Construct a gait graph describing the topological structure of the human body according to the natural connection method of the human body, and arrange multiple gait graphs of the same gait pattern according to the time series to form a temporal gait graph;

[0011] Furthermore, the gait diagram is an unweighted graph describing the topological structure of the human body. Furthermore, it is an undirected graph constructed according to the natural connection patterns of the human body. It is a set of joint nodes. This indicates the number of lower limb joints, with each lower limb joint representing a node. It is a set of joint nodes. This indicates the number of lower limb joints, with each lower limb joint representing a node. This represents the set of edges connecting sensor nodes. This represents the number of edges in a gait graph; an equivalent representation of a gait graph is... , It is the temporal gait graph feature of all nodes in the lower limb nodes. An adjacency matrix that represents the connection relationships between recorded nodes.

[0012] Furthermore, the temporal gait diagram is as follows: The equivalent representation is The problem of abnormal gait recognition can be viewed as training a mapping function. , use sequence-gait-graph Predicting gait patterns is defined as follows:

[0013] (2-1)

[0014] in, This represents the abnormal gait pattern to be identified, where T represents the time series length of the gait plot, and t represents time t. This represents the temporal gait characteristics at time t.

[0015] Step 2: Construct the LRP-MGG neural network model, including an accumulative cGRU model, a deep GCN module (deeperGCN), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation explanation module (LRP) for post-event explanation of model decisions, and a CNN deep convolutional neural network prediction module.

[0016] Step 3: Using the temporal gait map as input features, extract gait time-dependent features through the accumulative cGRU block of the LRP-MGG neural network model. The specific expression is as follows:

[0017] (2-6)

[0018] in, This represents the gait time-dependent features extracted by the cumulative cGRU block. express At any moment The hidden state, each GRU cell of the accumulative cGRU block is regarded as a time window, and each hidden state Considered as each Output of the sliding window at any given time.

[0019] Step 4: Using gait time-dependent features as input, the deep GCN module is used to obtain local spatial features embedded in the time-dependent feature space;

[0020] The accumulative cGRU block and the deep GCN module work together to obtain significant spatiotemporal gait difference features hidden in the nonlinear kinetic chain of human gait movement;

[0021] Specifically, assuming gait-graph Gait time-dependent features exist. and adjacency matrix The GCN model can utilize the gait time dependence features and adjacency matrix in non-Euclidean space. Extract spatial dependency features between different joint nodes.

[0022] Furthermore, the deep GCN module is constructed using residual connections and pre-activation techniques. It includes two GCN blocks connected by residual connections, and the two GCN blocks are feature-added. Each GCN block includes layer normalization, ReLU function, Dropout, and generalized aggregation network (GEN) set sequentially. Layer normalization is used to normalize the input data; ReLU represents the ReLU activation function, and Dropout effectively alleviates the overfitting problem; the GEN of the first GCN block is used to pre-activate gait time-dependent features. It is connected to each residual block to ensure deep training. The Generalized Aggregation Network (GEN) layer is a GCN layer with enhanced message aggregation capabilities to efficiently train deep GCNs.

[0023] Step 5: Perform graph-based gait pattern classification using the DiffPool hierarchical network prediction module; that is, by learning differentiable node clustering assignments at each layer of the multi-level pooling network, all nodes are mapped to a set of clustering graph embeddings to perform graph-based abnormal gait pattern classification.

[0024] Specifically, the DiffPool hierarchical network prediction module is constructed to classify graph-based anomalous gait patterns. This module mainly consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). Then, the aggregated cluster is used as input to the fully connected layer for graph classification. Step 5 specifically includes the following steps:

[0025] Step 5-1: Calculate the input gait time dependency features in the deep GNN. and adjacency matrix It can generate an allocation matrix with differentiable clustering. The specific calculation formula is as follows:

[0026] (2-10)

[0027] Step 5-2, learn the allocation matrix By combining the convergent node features of differentiable clustering at each layer of a standard GNN, the embeddings of all aggregated joint nodes can be obtained. The specific calculation formula is as follows:

[0028] (2-11)

[0029] Step 5-3: Embed all six joint nodes (i.e., graph embedding) and input them into the fully connected layer to accurately predict graph-based gait patterns. Specifically,

[0030] (2-12)

[0031] in, This represents graph pooling. This represents the most anomalous and representative spatiotemporal gait features. This indicates that graph pooling outputs abnormal gait features. This indicates the output aggregate weight matrix. This indicates the row number of the weight matrix. This indicates the number of columns in the weight matrix. This represents the weight matrix of the activation function. Indicates the pooling network's first The clustering assignment matrix of the layer, The aggregation embedding matrix represents the graph nodes; This represents the number of graph-based gaits to be classified; Represents the Softmax function, based on The output dimension is used to obtain the maximum number of clusters at each layer; Labels representing the graph-based gait patterns to be predicted; express Activation function.

[0032] Step 6: Utilize the significant spatiotemporal differences in gait features, use a CNN deep convolutional neural network model to obtain the joint nodes most likely to be abnormal, and visualize the abnormal nodes in order to learn interpretable abnormal gait information contained in the spatiotemporal features.

[0033] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to perform dimensionality reduction, feature extraction, and local connectivity extraction on the given spatiotemporal features; the output of the last convolutional layer is then processed. The input is fed into a fully connected layer to obtain the node prediction vector; where, the CNN deep convolutional neural network model is given convolutional layers with different kernel sizes, and any nth node... Layered gait spatiotemporal features Defined as:

[0034] (2-13)

[0035] in, Represents the input matrix; This represents the activation function. Indicates the inner product. Represents the convolution kernel vector. Represents the offset matrix; The values ​​are 1, 2, and 3 respectively;

[0036] Output the last convolutional layer The input is fed into a fully connected layer to obtain the node prediction vector. Defined as

[0037] (2-14)

[0038] in , , , These are the weight matrix and kernel offset matrix of the two fully connected layers, respectively.

[0039] Step 7: The interpretable anomalous gait information in the spatiotemporal features is used to calculate the importance score of each input feature to the model output using the Hierarchical Relevance Propagation (LRP) module, in order to achieve anomalous gait detection; the specific calculation formula is as follows:

[0040] (2-15)

[0041] in, Representing the Feature importance scores for each hidden layer node It is the first The activation value of each hidden layer node. It is the connection of the first The hidden layer node and the first The weights of each output layer node. It is the first Feature importance scores for each output layer node.

[0042] Furthermore, in step 7, a small constant is added to the hierarchical correlation propagation interpretation module (LRP). This is applied to the ReLU output to prevent gradient vanishing. This ensures that importance scores are propagated correctly while maintaining a reasonable interpretation.

[0043] Furthermore, the LRP-MGG neural network model combines gait pattern prediction and gait node anomaly prediction tasks for multi-task learning. For each sub-task, cross-entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as...

[0044] (2-16)

[0045] Where K represents the total number of abnormal gait identification labels, and O represents the total number of abnormal nodes in the gait graph. and These represent the actual gait pattern and the corresponding predicted gait pattern, respectively. and These represent the actual abnormal nodes and the corresponding predicted abnormal nodes, respectively.

[0046] This invention also discloses an interpretable abnormal gait detection model based on multi-task graph fusion learning, which includes an accumulative cGRU model, a deep GCN module (deeperGCN, i.e., a deep graph convolutional network model), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation interpretation module (LRP) for post-event interpretation of model decisions, and a CNN deep convolutional neural network prediction module.

[0047] The cumulative cGRU model consists of two or more GRU cells that are added sequentially. Each GRU cell mainly consists of reset gates. and Update Gate Composition: The deep GCN module (deeperGCN) is constructed using residual connections and pre-activation techniques. It consists of two GCN blocks connected by residual connections, with features added between the two blocks. Each GCN block includes layer normalization, ReLU function, Dropout, and a generalized aggregation network (GEN) layer, arranged sequentially. Layer normalization normalizes the input data; ReLU represents the ReLU activation function; Dropout effectively alleviates overfitting; the generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation capabilities to effectively train the deep GCN. The GEN in the first GCN block is used to pre-activate gait time-dependent features. The DiffPool hierarchical network prediction module consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding), and the aggregated cluster is used as the input to the fully connected layer for graph-based gait pattern classification. The CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality from the spatiotemporal significant difference features of gait and visualizes the abnormal nodes in order to learn the interpretable abnormal gait information contained in the spatiotemporal features. The hierarchical correlation propagation interpretation module (LRP) is used to calculate the importance score of each input feature to the model output from the interpretable abnormal gait information in the spatiotemporal features.

[0048] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to perform dimensionality reduction, feature extraction, and local connectivity extraction on a given spatiotemporal feature. Given convolutional layers with different kernel sizes, the CNN deep convolutional neural network model can perform dimensionality reduction, feature extraction, and local connectivity extraction on any given convolutional layer. Layered gait spatiotemporal features Defined as:

[0049] (2-13)

[0050] in, Represents the input matrix; This represents the activation function. Indicates the inner product. Represents the convolution kernel vector. Represents the offset matrix; The values ​​are 1, 2, and 3 respectively;

[0051] Output the last convolutional layer The input is fed into a fully connected layer to obtain the node prediction vector. Defined as

[0052] (2-14)

[0053] in , , , These are the weight matrix and kernel offset matrix of the two fully connected layers, respectively.

[0054] Furthermore, the LRP-MGG neural network model combines gait pattern prediction and gait node anomaly prediction tasks for multi-task learning. For each sub-task, cross-entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as:

[0055] (2-16)

[0056] Where K represents the total number of abnormal gait identification labels, and O represents the total number of abnormal nodes in the gait graph. and These represent the actual gait pattern and the corresponding predicted gait pattern, respectively. and These represent the actual abnormal nodes and the corresponding predicted abnormal nodes, respectively.

[0057] This invention employs the above technical solution, using the developed cGRU-dGCN model as the backbone model. It outputs abnormal pattern classification features and abnormal node classification features through two different branches from graph convolutional features containing complex human compensatory information. The cumulativeGRU module possesses powerful capabilities for extracting abnormal gait temporal features. Through an additive design, it continuously superimposes the hidden features of multiple GRU temporal correlation calculation units, enriching the compensatory feature representation of abnormal gait. The complex graph convolutional features extracted by the deep GCN module (deepGCN) use two channel branches to obtain abnormal pattern classification features and abnormal node classification features respectively. The abnormal pattern classification features are used by the DiffPool hierarchical network prediction module to obtain gait abnormal pattern classification, thus providing a gait pattern classification while retaining the high generalization characteristics of the cGRU-dGCN model. The abnormal node classification features are used by the CNN deep convolutional neural network prediction module to obtain gait abnormal node classification, identifying possible abnormal joint nodes in the abnormal gait pattern, and thus providing a user-oriented explanation of possible disease nodes in abnormal gait. The gait abnormal pattern classification task and the abnormal node classification task, as different branches in multi-task learning, can simultaneously fit two different tasks. Finally, the hierarchical correlation propagation interpretation module (LRP) of the ex-post solvability method is used for ex-post interpretation. That is, based on the rule of allocating contribution values ​​according to the layer-by-layer backpropagation of the neural network decision output, each neuron in the neural network will have an impact on the model output. By quantifying the importance score of each neuron layer by layer until each wavelength range of the input sensor signal is calculated, an interpretation for the model decision is obtained.

[0058] This invention allows machine learning models to learn multiple tasks simultaneously, improving model performance and efficiency. Combining the training of multiple tasks reduces the amount and complexity of data, helping the model better learn both gait recognition and interpretability tasks. This results in better robustness across different tasks, thereby improving the model's generalization and predictive abilities. Attached Figure Description

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0060] Figure 1 This is a schematic diagram of the model framework of the interpretable abnormal gait detection method based on multi-task graph fusion learning of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0062] like Figure 1As shown, this invention discloses an interpretable abnormal gait detection method based on multi-task graph fusion learning. The gait anomaly pattern classification task and the anomaly node classification task are treated as different branches in the multi-task learning process. This allows for both high generalization of the model's abnormal gait recognition and the calculation of the importance of the model's prediction results for each signal interval in the gait signal to be interpreted. The model framework of this invention mainly consists of an accumulative cGRU model, a deep graph convolutional network model (deepGCN), a diffpool layered network prediction module, a layer-wise relevance propagation (LRP) module for post-event interpretation of model decisions, and a CNN deep convolutional neural network prediction module. Its structure is as follows: Figure 1 As shown, the technical approach is as follows: The developed cGRU-dGCN model is used as the backbone model. Graph convolutional features containing complex human compensatory information are output as abnormal pattern classification features and abnormal node classification features through two different branches. The cumulativeGRU module possesses powerful capabilities for extracting abnormal gait temporal features. Through an additive design, the hidden features of multiple GRU temporal correlation calculation units are continuously superimposed, enriching the compensatory feature representation of abnormal gait. The complex graph convolutional features extracted by the deeperGCN deep graph convolutional network model are used to obtain abnormal pattern classification features and abnormal node classification features through two channel branches. The abnormal pattern classification features are used to obtain gait abnormal pattern classification through the diffpool hierarchical network prediction module, thus providing a gait pattern classification while retaining the high generalization characteristics of the cGRU-dGCN model. The abnormal node classification features are used to obtain gait abnormal node classification through the CNN deep convolutional neural network prediction module, identifying possible abnormal joint nodes in the abnormal gait pattern, and thus providing a user-oriented explanation of possible disease nodes in abnormal gait. The gait abnormal pattern classification task and the abnormal node classification task, as different branches in multi-task learning, can simultaneously fit two different tasks. Finally, the hierarchical correlation propagation interpretation module (LRP) of the ex-post solvability method is used for ex-post interpretation. That is, based on the rule of allocating contribution values ​​according to the layer-by-layer backpropagation of the neural network decision output, each neuron in the neural network will have an impact on the model output. By quantifying the importance score of each neuron layer by layer until each wavelength range of the input sensor signal is calculated, an interpretation for the model decision is obtained.

[0063] It includes the following steps:

[0064] Step 1: Construct a gait graph describing the topological structure of the human body according to the natural connection method of the human body, and arrange multiple gait graphs of the same gait pattern according to the time series to form a temporal gait graph;

[0065] Furthermore, the gait diagram is an unweighted graph describing the topological structure of the human body. Furthermore, it is an undirected graph constructed according to the natural connection patterns of the human body. It is a set of joint nodes. This indicates the number of lower limb joints, with each lower limb joint representing a node. This represents the set of edges connecting sensor nodes. This represents the number of edges in a gait graph; an equivalent representation of a gait graph is... , It is the temporal gait graph feature of all nodes in the lower limb nodes. An adjacency matrix that represents the connection relationships between recorded nodes.

[0066] Furthermore, the temporal gait diagram is as follows: The equivalent representation is The problem of abnormal gait recognition can be viewed as training a mapping function. , use sequence-gait-graph Predicting gait patterns is defined as follows:

[0067] (2-1)

[0068] in, This represents the abnormal gait pattern to be identified, where T represents the time series length of the gait plot, and t represents time t. This represents the temporal gait characteristics at time t.

[0069] Step 2: Construct the LRP-MGG neural network model, including an accumulative cGRU model, a deep GCN module (deeperGCN), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation explanation module (LRP) for post-event explanation of model decisions, and a CNN deep convolutional neural network prediction module.

[0070] Step 3: Using the temporal gait map as input features, extract gait time-dependent features through the accumulative cGRU block of the LRP-MGG neural network model. The specific expression is as follows:

[0071] (2-6)

[0072] in, This represents the gait time-dependent features extracted by the cumulative cGRU block. express At any moment The hidden state, each GRU cell of the accumulative cGRU block is regarded as a time window, and each hidden state Considered as each Output of the sliding window at any given time.

[0073] Step 4: Using gait time-dependent features as input, the deep GCN module is used to obtain local spatial features embedded in the time-dependent feature space;

[0074] The accumulative cGRU block and the deep GCN module work together to obtain significant spatiotemporal gait difference features hidden in the nonlinear kinetic chain of human gait movement;

[0075] Specifically, assuming gait-graph Gait time-dependent features exist. and adjacency matrix The GCN model can utilize the gait time dependence features in non-Euclidean space. and adjacency matrix Extract spatial dependency features between different joint nodes.

[0076] Furthermore, the deep GCN module is constructed using residual connections and pre-activation techniques. It includes two GCN blocks connected by residual connections, and the two GCN blocks are feature-added. Each GCN block includes layer normalization, ReLU function, Dropout, and generalized aggregation network (GEN) set sequentially. Layer normalization is used to normalize the input data; ReLU represents the ReLU activation function, and Dropout effectively alleviates the overfitting problem; the GEN of the first GCN block is used to pre-activate gait time-dependent features. It is connected to each residual block to ensure deep training. The Generalized Aggregation Network (GEN) layer is a GCN layer with enhanced message aggregation capabilities to efficiently train deep GCNs.

[0077] Step 5: Perform graph-based gait pattern classification using the DiffPool hierarchical network prediction module; that is, by learning differentiable node clustering assignments at each layer of the multi-level pooling network, all nodes are mapped to a set of clustering graph embeddings to perform graph-based abnormal gait pattern classification.

[0078] Specifically, the DiffPool hierarchical network prediction module is constructed to classify graph-based anomalous gait patterns. This module mainly consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding). Then, the aggregated cluster is used as input to the fully connected layer for graph classification. Step 5 specifically includes the following steps:

[0079] Step 5-1: Calculate the input gait time dependency features in the deep GNN. and adjacency matrix It can generate an allocation matrix with differentiable clustering. The specific calculation formula is as follows:

[0080] (2-10)

[0081] Step 5-2, learn the allocation matrix By combining the convergent node features of differentiable clustering at each layer of a standard GNN, the embeddings of all aggregated joint nodes can be obtained. The specific calculation formula is as follows:

[0082] (2-11)

[0083] Step 5-3: Embed all six joint nodes (i.e., graph embedding) and input them into the fully connected layer to accurately predict graph-based gait patterns. Specifically,

[0084] (2-12)

[0085] in, This represents graph pooling. This represents the most anomalous and representative spatiotemporal gait features. This indicates that graph pooling outputs abnormal gait features. This indicates the output aggregate weight matrix. This indicates the row number of the weight matrix. This indicates the number of columns in the weight matrix. This represents the weight matrix of the activation function. Indicates the pooling network's first The clustering assignment matrix of the layer, The aggregation embedding matrix represents the graph nodes; This represents the number of graph-based gaits to be classified; Represents the Softmax function, based on The output dimension is used to obtain the maximum number of clusters at each layer; Labels representing the graph-based gait patterns to be predicted; This represents the ReLU activation function.

[0086] Step 6: Utilize the significant spatiotemporal differences in gait features, use a CNN deep convolutional neural network model to obtain the joint nodes most likely to be abnormal, and visualize the abnormal nodes in order to learn interpretable abnormal gait information contained in the spatiotemporal features.

[0087] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to perform dimensionality reduction, feature extraction, and local connectivity extraction on the given spatiotemporal features; the output of the last convolutional layer is then processed. The input is fed into a fully connected layer to obtain the node prediction vector; where, the CNN deep convolutional neural network model is given convolutional layers with different kernel sizes, and any nth node... Layered gait spatiotemporal features Defined as:

[0088] (2-13)

[0089] in, Represents the input matrix; This represents the activation function. Indicates the inner product. Represents the convolution kernel vector. Represents the offset matrix; The values ​​are 1, 2, and 3 respectively;

[0090] Output the last convolutional layer The input is fed into a fully connected layer to obtain the node prediction vector. Defined as

[0091] (2-14)

[0092] in , , , These are the weight matrix and kernel offset matrix of the two fully connected layers, respectively.

[0093] Step 7: The interpretable anomalous gait information in the spatiotemporal features is used to calculate the importance score of each input feature to the model output using the Hierarchical Relevance Propagation (LRP) module, in order to achieve anomalous gait detection; the specific calculation formula is as follows:

[0094] (2-15)

[0095] in, Representing the Feature importance scores for each hidden layer node It is the first The activation value of each hidden layer node. It is the connection of the first The hidden layer node and the first The weights of each output layer node. It is the first Feature importance scores for each output layer node.

[0096] Furthermore, in step 7, a small constant is added to the hierarchical correlation propagation interpretation module (LRP). This is applied to the ReLU output to prevent gradient vanishing. This ensures that importance scores are propagated correctly while maintaining a reasonable interpretation.

[0097] Furthermore, the LRP-MGG neural network model combines gait pattern prediction and gait node anomaly prediction tasks for multi-task learning. For each sub-task, cross-entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as...

[0098] (2-16)

[0099] Where K represents the total number of abnormal gait identification labels, and O represents the total number of abnormal nodes in the gait graph. and These represent the actual gait pattern and the corresponding predicted gait pattern, respectively. and These represent the actual abnormal nodes and the corresponding predicted abnormal nodes, respectively.

[0100] This invention also discloses an interpretable abnormal gait detection model based on multi-task graph fusion learning, which includes an accumulative cGRU model, a deep GCN module (deeperGCN, i.e., a deep graph convolutional network model), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation interpretation module (LRP) for post-event interpretation of model decisions, and a CNN deep convolutional neural network prediction module.

[0101] The cumulative cGRU model consists of two or more GRU cells that are added sequentially. Each GRU cell mainly consists of reset gates. and Update Gate Composition: The deep GCN module (deeperGCN) is constructed using residual connections and pre-activation techniques. It consists of two GCN blocks connected by residual connections, with features added between the two blocks. Each GCN block includes layer normalization, ReLU function, Dropout, and a generalized aggregation network (GEN) layer, arranged sequentially. Layer normalization normalizes the input data; ReLU represents the ReLU activation function; Dropout effectively alleviates overfitting; the generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation capabilities to effectively train the deep GCN. The GEN in the first GCN block is used to pre-activate gait time-dependent features. The DiffPool hierarchical network prediction module consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster (i.e., graph embedding), and the aggregated cluster is used as the input to the fully connected layer for graph-based gait pattern classification. The CNN deep convolutional neural network model is used to obtain the joint nodes with the highest probability of abnormality from the spatiotemporal significant difference features of gait and visualizes the abnormal nodes in order to learn the interpretable abnormal gait information contained in the spatiotemporal features. The hierarchical correlation propagation interpretation module (LRP) is used to calculate the importance score of each input feature to the model output from the interpretable abnormal gait information in the spatiotemporal features.

[0102] Furthermore, the CNN deep convolutional neural network model uses three convolutional layers to perform dimensionality reduction, feature extraction, and local connectivity extraction on a given spatiotemporal feature. Given convolutional layers with different kernel sizes, the CNN deep convolutional neural network model can perform dimensionality reduction, feature extraction, and local connectivity extraction on any given convolutional layer. Layered gait spatiotemporal features Defined as:

[0103] (2-13)

[0104] in, Represents the input matrix; This represents the activation function. Indicates the inner product. Represents the convolution kernel vector. Represents the offset matrix; The values ​​are 1, 2, and 3 respectively;

[0105] Output the last convolutional layer The input is fed into a fully connected layer to obtain the node prediction vector. Defined as

[0106] (2-14)

[0107] in , , , These are the weight matrix and kernel offset matrix of the two fully connected layers, respectively.

[0108] Furthermore, the LRP-MGG neural network model combines gait pattern prediction and gait node anomaly prediction tasks for multi-task learning. For each sub-task, cross-entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as:

[0109] (2-16)

[0110] Where K represents the total number of abnormal gait identification labels, and O represents the total number of abnormal nodes in the gait graph. and These represent the actual gait pattern and the corresponding predicted gait pattern, respectively. and These represent the actual abnormal nodes and the corresponding predicted abnormal nodes, respectively.

[0111] The specific principles of this invention will be explained in detail below:

[0112] Define abnormal gait patterns:

[0113] Definition 1: Gait graph This invention uses an unweighted graph (gait-graph). To describe the topological structure of the human body, and gait-graph It is an undirected graph constructed according to the natural connection patterns of the human body. This invention treats each lower limb joint as a node. It is a set of joint nodes. This indicates the number of joints in the lower limbs. Represents the set of edges connecting sensor nodes, where Represents gait-graph The number of edges. Based on the properties of human gait motion, the edge relationships of the graph are constructed according to the natural connection method. Furthermore, since motion compensation is interactive, the gait-graph... It is an undirected graph. The accelerometer and gyroscope data collected by each node serve as the node's data. Attributes and characteristics , This indicates the number of node attribute features. (As a feasible implementation, a single sensor uses a 3-axis accelerometer and a 3-axis gyroscope to collect data, therefore...) =6). Given a gait-graph An adjacency matrix can be used This indicates the connection relationship between the record nodes. Represents a node and nodes The connection between them Represents a node and nodes There is no connection between them. Represents a node and nodes There are connections between them. gait-graph It can be equivalently represented as , It is the temporal gait graph feature of all nodes of the lower limb nodes.

[0114] Definition 2: Sequence-gait graph This invention uses a multi-sensor network to construct a sequence-gait-graph. Gait patterns are represented by sequence-gait graphs, which consist of multiple gait graphs of the same gait pattern arranged in a time series, and are represented as follows: The graph structure of human motion is fixed, meaning the adjacency matrix remains unchanged during motion; only the node attributes change. Therefore, a sequence-gait graph can be equivalently represented as... The problem of abnormal gait recognition can be viewed as training a mapping function. , use sequence-gait-graph Predicting gait patterns is defined as follows:

[0115] (2-1)

[0116] in, This represents the abnormal gait pattern to be identified, where T represents the time series length of the gait plot, and t represents time t. This represents the temporal gait characteristics at time t.

[0117] The LRP-MGG neural network model is constructed, including an accumulative cGRU model, a deep GCN module (deeperGCN), a DiffPool hierarchical network prediction module, a hierarchical correlation propagation explanation module (LRP) for post-event explanation of model decisions, and a CNN deep convolutional neural network prediction module.

[0118] Accumulating GRU Module: Gated Recurrent Neural Networks (GRUs) are powerful tools based on Recurrent Neural Networks (RNNs) for capturing gait time-dependent features in time-series data. In a GRU model, previous computations can be used recursively. GRUs can address problems such as vanishing gradients, exploding gradients, and long-term memory loss. Unlike Long Short-Term Memory (LSTM) units, which include input, forget, and output gates, GRU units primarily consist of reset gates. and Update Gate Composition. The GRU model has a relatively simple structure and fewer parameters, which facilitates model training and testing. In this chapter, the present invention develops a method for processing sequence-gait graphs. An additive GRU model is used to extract temporal dependencies. For each GRU cell at the current time t, it is assumed that there is a hidden state at a previous time t-1 containing gait temporal dependency features calculated previously. and sequence-gait-graph Gait characteristics Input, reset gate function and update gate function It is used to obtain relevant temporary information about gait characteristics between the current time t and the previous time t-1 by calculating equations (2-1) and (2-2).

[0119] (2-2)

[0120] (2-3)

[0121] in, This represents the sigmoid activation function; and It is the bias vector learned during the training process; and This is the weight matrix learned during training. Then, based on the features obtained from the reset gate function, candidate hidden states are... We will obtain the result from the equation (3-4) defined below at the current time t. Useful temporal information of gait features:

[0122] (2-4)

[0123] in, Represents the hyperbolic tangent activation function; It is the Hadamard product; and These are the weighting matrix and bias vector learned during training, respectively. Next, based on the candidate hidden states... and from the updated gate The obtained features are used to define the updated candidate hidden state by calculating the following equations (3-5). To capture the current moment and the previous moment Valuable gait transient dependency features:

[0124] (2-5)

[0125] It is important to note here that when When the value is close to 1, more previously temporary dependency feature information is remembered; conversely, when... When the value is close to 0, irrelevant previous time feature information disappears. Therefore, by using time series... The GRU unit is calculated at each time step within the time series, which can be obtained from time-series multi-sensor gait data. The study uncovered an intrinsic long-term dependency between multiple lower limb joints associated with abnormal gait changes.

[0126] Because the interaction between multiple joints in the lower limbs is synchronized under the control of the central nervous system, each Moment Hidden state It may contain useful temporary dependency features related to anomalous gait changes. To obtain the embeddings... To provide more useful temporary dependencies, this invention constructs an accumulative GRU block, where each GRU unit is considered a time window, and each hidden state... Considered as each The output of the sliding window at any given time, all At any moment Hidden state By definition

[0127] (2-6)

[0128] in, This represents the gait time-dependent features extracted by the cumulative cGRU block. express At any moment The hidden state, each GRU cell of the accumulative cGRU block is regarded as a time window, and each hidden state Considered as each Output of the sliding window at any given time.

[0129] The deeperGCN module: Graph Convolutional Networks (GCNs) are one of the most representative graph neural network models, capable of applying the representation learning capabilities of convolutional neural networks to irregular graph data. Its advantage lies in extending convolutional operations to non-Euclidean graph data for spatial feature extraction. In the GCN model, graph convolutional operations generate richer feature representations for the target vertex by aggregating the features of the target vertex and its neighboring vertices. The definition of graph convolutional operations in spatial GCNs depends on the spatial relationships between graph nodes. This invention constructs a GCN model to extract spatial features based on the topological relationships between joint nodes and their neighboring nodes. Features can be extracted by aggregating the features of the joint node itself and the features of two neighboring nodes within the GCN model. That is, assuming a gait-graph... Gait time-dependent features exist. and adjacency matrix The GCN model can utilize the gait time dependence features in non-Euclidean space. and adjacency matrix Extracting spatial dependency features between different joint nodes. In a GCN model constructed by stacking multiple convolutional layers, the solution for spatial feature extraction can be defined as:

[0130] (2-7)

[0131] (2-8)

[0132] (2-9)

[0133] in, It is an adjacency matrix with added self-connections. It is an identity matrix. It is a degree matrix, and all elements except the diagonal are 0. It is the feature map of each layer, when the GCN layer When =0, . This is the weight matrix that needs to be learned during training. Through a multi-layer graph convolutional network, the temporal dependencies of the joint nodes can be learned. Embedded spatial features. When a GCN model developed during training has optimal learning capabilities, more useful spatial features for each joint node can be discovered by aggregating neighboring joint nodes. However, existing GCN architectures can only train shallow GCNs. Increasing the number of graph convolutional layers leads to problems such as vanishing gradients, oversmoothing, and overfitting. This can result in a decrease in the learning ability of shallow GCN models to explore the most representative spatial features. Therefore, it is necessary to develop deeper GCNs with more powerful learning performance.

[0134] To this end, this invention utilizes the nonlinear spatial feature learning properties of the deep graph convolutional network (DCN) to facilitate the model's capture of nonlinear kinetic chain information hidden in human gait motion. Specifically, the temporal dependency features extracted by the GRU are used as input to deeper GCN modules to mine local spatial features embedded in the temporal dependency feature space. This can be achieved from the sequence-gait graph. It captures implicit spatiotemporal gait dynamics. Given the gait graph topology, the cumulative GRU will extract sequence-gait graph... Aggregated into a gait-graph Here, the deeper GCN module is constructed using residual connections and pre-activation techniques to achieve gait-graph processing. The module features excellent learning capabilities for more useful spatial dependency features. In this module, the Generalized Aggregation Network (GEN) layer is a GCN layer with enhanced message aggregation capabilities to effectively train the deep GCN. Based on GEN, the deeper GCN consists of a sequence of layer normalization → ReLU → GEN → Addition, where layer normalization is used to normalize the input data. ReLU represents the ReLU activation function. Dropout effectively alleviates the overfitting problem. The first GEN is used to pre-activate gait time-dependent features. This is then connected to each residual block to ensure deep training. In this way, significant spatiotemporal gait difference features hidden in the nonlinear kinetic chain of human gait motion can be obtained.

[0135] DiffPool hierarchical network predicts anomalous joint nodes: After extracting the most representative spatiotemporal features from all joint nodes, a graph-based gait classification task is performed. In recent related research, the DiffPool technique has been successfully applied to high-quality graph classification tasks. A DiffPool hierarchical network prediction module is constructed to perform graph-based gait pattern classification. The aim is to map all nodes into a set of clusters by learning differentiable node cluster assignments at each layer of a multi-level pooling network. Its advantage lies in aggregating the feature information of all nodes into a graph embedding through a fully connected layer for graph classification. In the model of this invention, the DiffPool hierarchical network prediction module is constructed for classifying graph-based anomalous gait patterns. This module mainly consists of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a single cluster (i.e., a graph embedding). Then, the aggregated cluster is used as input to the fully connected layer for graph classification. The above solution can be obtained by calculating the equations defined below.

[0136] (2-10)

[0137] (2-11)

[0138] (2-12)

[0139] in, This represents graph pooling. This represents the most anomalous and representative spatiotemporal gait features. This indicates that graph pooling outputs abnormal gait features. This indicates the output aggregate weight matrix. This indicates the row number of the weight matrix. This indicates the number of columns in the weight matrix. This represents the weight matrix of the activation function. Indicates the pooling network's first The clustering assignment matrix of the layer, This represents the aggregation embedding matrix of graph nodes. This represents the number of graph-based gaits to be classified. Represents the Softmax function, based on The output dimension can obtain the maximum number of clusters in each layer. Labels representing the graph-based gait patterns to be predicted. Let represent the ReLU activation function. Based on equation (2-11), the input gait time dependency features in the deep GNN are calculated. and adjacency matrix It can generate an allocation matrix with differentiable clustering. Then, according to equation (2-11), the allocation matrix is ​​learned. By combining the convergent node features of differentiable clustering at each layer of a standard GNN, the embeddings of all aggregated joint nodes can be obtained. Finally, based on equation (2-12), two fully connected layers were constructed to accurately predict graph-based gait patterns, while all six joint nodes were embedded (i.e., graph embedding) as input to the fully connected layers.

[0140] Predicting Abnormal Joint Nodes Using Deep Convolutional Neural Networks (CNNs): Unlike previous sensor-based abnormal gait recognition (XAI) tasks, this invention utilizes a deep convolutional neural network (CNN) model to obtain the most likely abnormal joint nodes and visualizes these nodes, facilitating the model's output of easily understandable and interpretable abnormal gait information. This invention leverages the significant spatiotemporal differences in gait obtained from cGRU and dGCN to further learn the interpretable information embedded in these spatiotemporal features. CNNs are excellent motion data feature extractors, capable of solving sensor data classification problems. This invention uses shallow layers to facilitate the model's learning to identify relatively simple node classification features, reducing the risk of overfitting. This invention designs three convolutional layers to perform dimensionality reduction, feature extraction, and local connectivity extraction on given spatiotemporal features. That is, given convolutional layers with different kernel sizes... Layered gait spatiotemporal features Defined as:

[0141] (2-13)

[0142] in, Represents the input matrix; This represents the activation function. Indicates the inner product. Represents the convolution kernel vector. Represents the offset matrix; The values ​​are 1, 2, and 3 respectively;

[0143] Output the last convolutional layer The input is fed into a fully connected layer to obtain the node prediction vector. Defined as

[0144] (2-14)

[0145] in , , , These are the weight matrix and kernel offset matrix of the two fully connected layers, respectively.

[0146] Hierarchical Relevance Propagation (LRP) Explanation Module for Model Decision Making: This invention employs the Hierarchical Relevance Propagation (LRP) module to predict interpretations. As a propagation-based post-hoc approach, it provides interpretations within the input space, which is typically the space where experts interpret signals in clinical practice. The backpropagation process of correlation is shown in the figure. LRP traverses the hierarchical structure of the LSTM model backward to generate an interpretation. LRP calculates the importance score of each input feature to the model output through backpropagation. It propagates the importance of the model output layer by layer backward, from the output layer to the input layer, assigning scores to each input feature. In this way, LRP can tell this invention which input signal regions play a positive or negative role in the model's decision-making. However, the interpretations obtained by LRP are overly complex and may contain a lot of noise. To ignore less important neurons, this invention uses LRP- (Layer-wise Relevance Propagation with epsilon). In LRP- In, by adding a small constant This is applied to the ReLU output to prevent gradient vanishing. This ensures that importance scores are propagated correctly while maintaining a reasonable interpretation, defined as...

[0147] (2-15)

[0148] in, Representing the Feature importance scores for each hidden layer node It is the first The activation value of each hidden layer node. It is the connection of the first The hidden layer node and the first The weights of each output layer node. It is the first Feature importance scores for each output layer node.

[0149] Multi-task model loss function: The multi-task learning approach designed in this invention allows machine learning models to learn multiple tasks simultaneously, improving model performance and efficiency. Combining the training of multiple tasks reduces the amount and complexity of data, helping the model better learn both gait recognition and interpretability tasks. This results in better robustness across different tasks, thereby improving the model's generalization and predictive abilities. Multi-task learning is achieved by combining gait pattern prediction and gait node anomaly prediction tasks. For each sub-task, cross-entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as...

[0150] (2-16)

[0151] Where K represents the total number of abnormal gait identification labels, and O represents the total number of abnormal nodes in the gait graph. and These represent the actual gait pattern and the corresponding predicted gait pattern, respectively. and These represent the actual abnormal nodes and the corresponding predicted abnormal nodes, respectively.

[0152] This invention employs the above technical solution, using the developed cGRU-dGCN model as the backbone model. It outputs abnormal pattern classification features and abnormal node classification features through two different branches from graph convolutional features containing complex human compensatory information. The cumulativeGRU module possesses powerful capabilities for extracting abnormal gait temporal features. Through an additive design, it continuously superimposes the hidden features of multiple GRU temporal correlation calculation units, enriching the compensatory feature representation of abnormal gait. The complex graph convolutional features extracted by the deep GCN module (deepGCN) use two channel branches to obtain abnormal pattern classification features and abnormal node classification features respectively. The abnormal pattern classification features are used by the DiffPool hierarchical network prediction module to obtain gait abnormal pattern classification, thus providing a gait pattern classification while retaining the high generalization characteristics of the cGRU-dGCN model. The abnormal node classification features are used by the CNN deep convolutional neural network prediction module to obtain gait abnormal node classification, identifying possible abnormal joint nodes in the abnormal gait pattern, and thus providing a user-oriented explanation of possible disease nodes in abnormal gait. The gait abnormal pattern classification task and the abnormal node classification task, as different branches in multi-task learning, can simultaneously fit two different tasks. Finally, the hierarchical correlation propagation interpretation module (LRP) of the ex-post solvability method is used for ex-post interpretation. That is, based on the rule of allocating contribution values ​​according to the layer-by-layer backpropagation of the neural network decision output, each neuron in the neural network will have an impact on the model output. By quantifying the importance score of each neuron layer by layer until each wavelength range of the input sensor signal is calculated, an interpretation for the model decision is obtained.

[0153] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. The components of the embodiments of this application described and illustrated herein can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. An interpretable abnormal gait detection method based on multi-task graph fusion learning, characterized in that: It comprises the following steps: Step 1, construct a gait chart describing the topological structure of the human body according to the natural connection mode of the human body, and arrange a plurality of gait charts of the same gait mode in time sequence to obtain a time sequence gait chart; Step 2, construct an LRP-MGG neural network model, including an accumulative cGRU model, a deep GCN module, a DiffPool hierarchical network prediction module, a hierarchical relevance propagation explanation module LRP, and a CNN deep convolutional neural network prediction module, Step 3, taking the time sequence gait chart as the input feature, the accumulative cGRU block of the LRP-MGG neural network model is used to extract the gait time dependence feature; Step 4, taking the gait time dependence feature as the input, the deep GCN module is used to obtain the local space feature embedded in the time dependence feature space; Step 5, the DiffPool hierarchical network prediction module is used for graph-based gait mode classification; that is, by learning the differentiable node clustering assignment at each layer of the multi-level pooling network, all nodes are mapped into a set of cluster graph embeddings for graph-based abnormal gait mode; Step 6, using the gait spatiotemporal significant difference feature, the CNN deep convolutional neural network model is used to obtain the joint node most likely to occur abnormally, and the abnormal node is visualized to learn the explainable abnormal gait information contained in the spatiotemporal feature; Step 7, the explainable abnormal gait information in the spatiotemporal feature is calculated by the hierarchical relevance propagation explanation module LRP to obtain the importance score of each input feature to the model output, so as to realize abnormal gait detection; wherein the gait graph is an unweighted graph describing the topology of the human body , and is an undirected graph constructed according to the natural connection of the human body, is a set of joint nodes, represents the number of lower limb joints, each of which is a node; represents a set of connecting edges between sensor nodes, represents the number of edges of the gait graph; the gait graph is equivalent to , is a time sequence gait graph feature of all nodes of the lower limb nodes, represents an adjacency matrix recording the connection relationship of the nodes; The time series gait graph is , and the equivalent representation is ; the abnormal gait recognition problem is regarded as training the mapping function , using the predicted gait pattern, defined as: (2-1) wherein, represents an abnormal gait pattern to be identified, T represents a time series length of a time series gait graph, t represents a t moment, represents a time series gait graph feature at the t moment.

2. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 1, characterized in that: The specific expression of the gait time dependence feature extracted in step 3 is as follows: (2-6) wherein, denotes the gait time-dependent features extracted by the cumulative cGRU block, denotes the hidden state of at time step , each GRU unit of the cumulative cGRU block is considered as a time window, each hidden state is considered as the output of a sliding window at each time step.

3. The method of claim 1, wherein the method is based on multi-task graph fusion learning. The deep GCN module in step 4 is constructed by using a residual connection and a pre-activation technique, including two GCN blocks with a residual connection, and the two GCN blocks are used for feature addition; the GCN model uses the gait time dependence feature in a non-Euclidean space and an adjacency matrix to extract the spatial dependence features between different joint nodes; each GCN block includes layer normalization, a ReLU function, Dropout and a generalized aggregation network GEN arranged in sequence, and the layer normalization is used for normalizing the input data; the ReLU represents a ReLU activation function, and the Dropout is used for relieving the overfitting problem; The GEN of the first GCN block is used to pre-activate the gait time-dependent features and connected with each residual block to ensure deep training.

4. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 1, characterized in that: The DiffPool hierarchical network prediction module in step 5 is constructed for graph-based abnormal gait mode classification, and the DiffPool hierarchical network prediction module is composed of a pooling network and a fully connected layer. The pooling network aggregates all six lower limb joint nodes into a cluster, and the aggregated cluster is used as the input of the fully connected layer for graph classification. The specific steps include the following steps: Step 5-1, computing input gait time-dependent features in the depth GNN and adjacency matrix , generating an assignment matrix with differentiable clustering ; specifically, the formula is as follows: (2-10) Step 5-2, learning the assignment matrix and the differential clustering of the aggregated node features of each layer of the standard GNN, obtaining all the aggregated joint node embeddings ; specifically, the calculation formula is as follows: (2-11) Step 5-3, embed all six joint nodes into the fully connected layer to accurately predict the graph-based gait mode, specifically, (2-12) wherein, represents graph pooling, represents most abnormal representative spatiotemporal gait graph features, represents graph pooling output abnormal gait features, represents output aggregation weight matrix, represents the number of rows of the weight matrix, represents the number of columns of the weight matrix, represents the weight matrix of the activation function, represents the clustering assignment matrix of the pooling network first layer, represents the aggregated embedding matrix of the graph nodes; represents the number of graph-based gaits to be classified; represents the Softmax function, according to the output dimension of , the maximum number of clusters is obtained in each layer; represents the label of the graph-based gait pattern to be predicted; represents the activation function.

5. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 1, characterized in that: The CNN deep convolutional neural network model adopts three layers of convolutional layers to reduce dimension, extract features and extract local connectivity for the given spatiotemporal feature layers, and outputs the last layer of convolution to a fully connected layer to obtain a node prediction vector; wherein the CNN deep convolutional neural network model is given a convolutional layer with different kernel sizes, and any ith layer of gait spatiotemporal features is defined as: wherein, the CNN deep convolutional neural network model is given a convolutional layer with different kernel sizes, and any ith layer of gait spatiotemporal features is defined as: wherein, the CNN deep convolutional neural network model is given a convolutional layer with different kernel sizes, and any ith layer of gait spatiotemporal features is defined as: ​ (2-13) wherein, represents an input matrix; represents an activation function, represents an inner product, represents a convolution kernel vector, represents a bias matrix; the values of are 1, 2, 3, respectively. The output of the last convolution layer is input to a fully connected layer to obtain a node prediction vector The output of the last convolution layer is input to a fully connected layer to obtain a node prediction vector is defined as (2-14) wherein , , , are the weight matrix and the bias matrix of the two fully connected layers, respectively.

6. The interpretable abnormal gait detection method based on multi-task graph fusion learning according to claim 1, characterized in that: Adding a small constant to the output of ReLU in step 7 to prevent vanishing gradient; specifically, the formula is as follows: to the output of ReLU to prevent vanishing gradient; specifically, the formula is as follows: (2-15) wherein, represents a feature importance score for a node of a first hidden layer, is an activation value for a node of a first hidden layer, is a weight connecting a node of a first hidden layer and a node of a first output layer, is a feature importance score for a node of a first output layer.

7. The interpretable abnormal gait detection model based on multi-task graph fusion learning according to any one of claims 1 to 6, characterized in that: The model includes an accumulative cGRU model, a deep GCN module, a DiffPool hierarchical network prediction module, a hierarchical relevance propagation explanation module LRP, and a CNN deep convolutional neural network prediction module; The cumulative cGRU model comprises two or more GRU units sequentially accumulated, each GRU unit mainly comprises a reset gate and an update gate ; the deep GCN module is constructed by using a residual connection and a pre-activation technique, comprises two GCN blocks using a residual connection, and the two GCN blocks add features; each GCN block comprises sequentially arranged layer normalization, a ReLU function, Dropout and a generalized aggregation network GEN, the layer normalization is used for normalizing input data; the ReLU represents a ReLU activation function, and the Dropout effectively alleviates an overfitting problem. The generalized aggregation network (GEN) layer is a GCN layer with enhanced message aggregation function to effectively train a deep GCN, and the GEN of the first GCN block is used to pre-activate the gait time-dependent features And each residual block is connected to ensure deep training; the DiffPool hierarchical network prediction module is composed of a pooling network and a fully connected layer, the pooling network aggregates all six lower limb joint nodes into a cluster, and the aggregated cluster is used as the input of the fully connected layer for graph-based gait pattern classification; the CNN deep convolutional neural network model is used to obtain the joint node with the most probable abnormality from the gait spatiotemporal significant difference features, and the abnormal node is visualized to learn the interpretable abnormal gait information contained in the spatiotemporal features; The hierarchical relevance propagation explanation module LRP is used to calculate the importance score of each input feature to the model output from the explainable abnormal gait information in the spatiotemporal feature.

8. The interpretable abnormal gait detection model based on multi-task graph fusion learning of claim 7, wherein: The CNN deep convolutional neural network model adopts three layers of convolutional layers to reduce dimension, extract features and extract local connectivity for the given spatiotemporal feature accumulation layer. The CNN deep convolutional neural network model is given a convolutional layer with different kernel sizes, and any i Layer spatiotemporal features is defined as: (2-13) wherein, represents an input matrix; represents an activation function, represents an inner product, represents a convolution kernel vector, represents an offset matrix; the values of are 1, 2, 3, respectively. Output the last convolutional layer The input is fed into a fully connected layer to obtain the node prediction vector. Defined as (2-14) wherein , , , are respectively the weight matrix kernel bias matrix of two fully connected layers.

9. The interpretable abnormal gait detection model based on multi-task graph fusion learning of claim 7, wherein: The LRP-MGG neural network model combines the gait mode prediction task and the gait node abnormal prediction task to perform multi-task learning; For the subtask, cross entropy is used as the loss function to evaluate the minimum error between the true gait label and the corresponding label to be predicted during training and testing, defined as (2-16) wherein K represents the total number of abnormal gait recognition labels, O represents the total number of gait graph abnormal nodes, and respectively represent the real gait pattern and the corresponding predicted gait pattern; and respectively represent the real abnormal node and the corresponding predicted abnormal node.

Images