Gait analysis method based on multi-scale cross-dimensional common attention mechanism

Abstract

The invention discloses a gait analysis method based on a multi-scale cross-dimensional common attention mechanism. The method comprises the following steps: (1) constructing a space-time diagram convolutional network ST-GCN based on skeleton action recognition; (2) in order to better capture and fuse multi-scale and multi-dimensional gait features, a multi-scale cross-dimensional common attention mechanism, which is called MS-CDCA for short, is introduced on the basis of ST-GCN, and an MCC-GCN model for abnormal gait recognition is obtained; (3) an abnormal gait data set GIST is used for training MCC-GCN, the data set comprises one normal gait and five pathological abnormal gaits, and the data set is obtained through simulation of 10 subjects; and (4) using a kinect v2 depth camera to collect three-dimensional gait data of a real abnormal gait patient, normalizing the three-dimensional gait data, and inputting the normalized three-dimensional gait data into the pre-trained MCC-GCN to obtain a diagnosis result. By means of the method, discrimination and robustness of gait feature representation are improved, a new tool is provided for clinical diagnosis of abnormal gaits, and the method has wide research prospects and important application value.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a gait analysis method based on a multi-scale cross-dimensional joint attention mechanism. Background Art

[0002] Gait is the movement pattern of the human body when walking. Normal gait shows the characteristics of coordination, balance and fluency. Abnormal gait refers to a walking style that deviates from the normal gait movement pattern and shows abnormal characteristics. It is usually caused by diseases, injuries or dysfunctions of the nervous system, skeletal muscle system or other parts of the body. Gait analysis can identify abnormal gait by analyzing the characteristics of gait.

[0003] Abnormal gait analysis based on deep learning can use rich gait data to automatically learn gait features and identify abnormal gait, providing doctors with auxiliary diagnosis basis and improving diagnostic efficiency and accuracy. Therefore, it is of great practical significance to develop a more accurate, convenient and clinically appropriate abnormal gait diagnosis method.

[0004] Gait is a complex and dynamic process, containing rich characteristic information at multiple scales and dimensions. Abnormal gait often manifests as multiple levels of motion deviations, which need to be fully understood through observation and analysis at different scales. From the microscopic scale, abnormal gait manifests as subtle changes in specific joint angles or abnormalities in the speed and acceleration of skeletal points. These abnormalities are reflected in the details of the relative position relationship and motion trajectory between skeletal points, and may be early warning signs of certain specific diseases or physical conditions. From the mesoscopic scale, abnormal gait manifests as significant changes in the gait cycle and overall pattern, or disturbances in the coordination of limb movements. These changes are reflected in the synchronization and consistency of skeletal point movements between different limbs, which may reveal functional abnormalities or diseases in a specific part of the body. From the macroscopic scale, abnormal gait manifests as deviations in the overall motion trajectory, and even presents special motion patterns. These deviations are reflected in the overall change trend of the skeletal point motion trajectory throughout the gait cycle, and are often the external manifestations of systemic problems that affect whole-body movement.

[0005] Abnormal gait is often caused by multiple factors, which are reflected in different dimensions of gait. For example, a patient with knee injury will show abnormal changes in knee angle, abnormal knee velocity and acceleration in the bone point dimension, prolonged gait cycle and slower step frequency in the time dimension, and shift of body center of gravity and decreased limb movement coordination in the space dimension. Existing abnormal gait analysis methods tend to focus only on features of a single scale, ignoring the complex interactions between features of different scales, resulting in an incomplete understanding of abnormal gait. In addition, many studies mainly focus on feature extraction in the time and space dimensions, while in-depth analysis of the bone point dimension is relatively insufficient. This emphasis may lead to the neglect of some important local information, thus affecting the comprehensiveness and accuracy of the analysis.

[0006] Therefore, effectively capturing multi-scale information and establishing long-distance dependencies are crucial for abnormal gait analysis. Developing a gait analysis method that can effectively capture gait information of different scales and dimensions and perform deep fusion will provide a new tool for the clinical diagnosis of abnormal gait, which has broad research prospects and important application value. Summary of the invention

[0007] The present invention proposes a gait analysis method based on a multi-scale cross-dimensional joint attention mechanism to improve the diagnostic efficiency and accuracy of patients with abnormal gait, provide doctors with more accurate diagnostic basis, and provide more effective guidance for the treatment of patients.

[0008] The technical solution adopted by the present invention is as follows: Gait analysis method based on multi-scale cross-dimensional joint attention mechanism:

[0009] (1) Build a spatiotemporal graph convolutional network ST-GCN for skeleton action recognition;

[0010] 1.1) The skeleton sequence is represented by the 2D or 3D coordinates of each joint of the human body in each frame. The skeleton sequence is hierarchically represented using a spatiotemporal graph to construct a skeleton graph. An undirected spatiotemporal graph G = (V, E) is constructed on a skeleton sequence with N joints and T frames. It captures the connections between joints in the body while considering the temporal association between frames. The joint nodes are divided into multiple subsets, including centripetal, eccentric, and root nodes, and different weights are assigned to each subset.

[0011] 1.2) Based on the characteristics of action recognition, a graph partitioning method is introduced. The distance from the root node to the center of gravity is used as the benchmark, marked as label = 0. For all adjacent nodes, they are classified according to the comparison between their distance to the center of gravity and the benchmark value: if the distance is less than the benchmark value, it is marked as a centripetal node, label = 1; if the distance is greater than the benchmark value, it is marked as a centrifugal node, label = 2;

[0012] 1.3) Graph convolution uses edges as weights to calculate the weighted average of node features. The graph convolution formula used is:

[0013]

[0014] Among them, z ti (v tj )=|{v tk |l ti (v tk )=l ti (v tj )}| is a normalization term used to balance the contribution of different subsets, w is a weight function that provides a weight matrix for inner product operation with the input sampling features, and v tj v ti The adjacent nodes of ti (v tj ) is the adjacent node v tj The label to which it belongs, l ti is the mapping relationship of the subset;

[0015] 1.4) Use the temporal convolutional network (TCN) to extract the temporal information of joint point changes. TCN is a one-dimensional convolution of size [1, t]. Its convolution dimension is the time dimension. The convolution kernel is a structure of one row and multiple columns. Convolution is performed along the time dimension to extract the temporal features of the skeleton points.

[0016] (2) Based on ST-GCN, a multi-scale cross-dimensional joint attention mechanism is introduced, referred to as MS-CDCA, which captures and integrates multiple 2.1) The multi-scale cross-dimensional joint attention mechanism has a small number of parameters and moderate computational overhead. ST-GCN contains 10 layers of spatiotemporal graph convolutional layers, and a multi-scale cross-dimensional joint attention mechanism is deployed between every two layers of spatiotemporal graph convolutional layers;

[0017] 2.2) The gait sequence feature F is input into ST-GCN and processed by the spatiotemporal graph convolution layer to obtain the feature F input , the calculation process is:

[0018] F input =ST-GCN(F)

[0019] Among them, ST-GCN(·) represents the spatiotemporal graph convolution layer before the attention module, and the obtained F in[ut It is passed to the multi-scale cross-dimensional joint attention mechanism, and the calculation process is:

[0020] F atten =MS-CDCA(F input )

[0021] Among them, MCDCA(·) represents the multi-scale cross-dimensional joint attention mechanism, which converts F inputand the output F of the multi-scale cross-dimensional joint attention mechanism atten Connect along the channel dimension to get F output , the calculation process is:

[0022] F output =[F input ,F at(en ]

[0023] Among them, […] represents residual connection;

[0024] 2.3) F output It will be passed to the next spatiotemporal graph convolution layer and attention module to obtain the final feature representation F final , the linear layer is used to replace the pooling layer of ST-GCN, and the final prediction result Output is obtained through the fully connected layer. The calculation process is:

[0025] Output=FCN(Linear(F final ))

[0026] Among them, Linear(·) represents the linear layer and FCN(·) represents the fully connected layer.

[0027] The scale and multi-dimensional gait features are used to obtain the MCC-GCN model for abnormal gait recognition.

[0028] (3) The abnormal gait dataset GIST was used to train MCC-GCN, which contains 1 normal gait and 5 pathological abnormal gaits simulated by 10 subjects;

[0029] The GIST dataset divides gait into one normal gait and five abnormal gaits; among them, abnormal gaits include analgesic gait, stride gait, stagger gait, stiff leg gait and gluteus medius gait; analgesic gait is manifested as lameness and reduced walking speed according to the injured part; stride gait is walking with toes touching the ground and lifting the lower limbs to start; stagger gait is manifested as swaying of the body on both sides when walking, similar to the walk of a duck; stiff leg gait has the phenomenon of circling gait, foot drop and hyperextension of knee joint; gluteus medius gait has one high and one low pelvis when walking; this dataset uses 6 cameras to collect three-dimensional bone information, with a total of 7200 samples. The dataset is divided into a training set: test set ratio of 8:2 and the MCC-GCN model is trained to obtain a pre-trained MCC-GCN model.

[0030] (4) The kinect v2 depth camera is used to collect the three-dimensional gait data of real patients with abnormal gait, which is normalized and input into the pre-trained MCC-GCN to obtain the diagnosis results.

[0031] 4.1) In actual application scenarios, for the abnormal gait data based on 3D skeleton points of real abnormal gait patients, the posture alignment of the gait cycle and the normalization of the world coordinate system are first performed to achieve unification in time and space dimensions; the complete gait cycle is identified and extracted in the continuous motion sequence, and the postures in different sequences are aligned;

[0032] The coordinates of the bone points in the three-dimensional space are converted into the radian representation of the joint rotation, the Euclidean distance between each pair of vectors in the radian matrix is ​​calculated, and the inter-frame distance constraint is introduced. The best matching action is identified by finding the local minimum Euclidean distance in the distance matrix, and the start frame and end frame of the gait cycle are located. The cycle is defined as from the left heel to the next left heel touchdown; the extracted complete gait cycle data is uniformly sampled, the number and time order of key frames in different gait data are unified, and time normalization is performed by frame cutting;

[0033] 4.2) Perform spatial normalization on the acquired gait data, define the reference height as 1.6874 meters, calculate the ratio of the target height to the reference height, start from the central bone point of the hip, and gradually adjust the relative position of each bone point outward to maintain the integrity of the bone structure, so that all the bone points of the characters are at the same scale; unify the coordinate system standard by adjusting the coordinate axis, adjust the Y axis according to the height of the left and right ankles, so that the character is located in a coordinate system with the ground as the origin; use the human physiological information calculation method to obtain the spatial information and physiological information of the character, and calculate the Z axis offset in combination with the pelvic position of the character to eliminate the influence of camera distance and viewing angle;

[0034] 4.3) After eliminating the differences in data collection through 4.1) and 4.2), the final dimension is R N×C×T×V The gait sequence feature F is obtained, where N is the number of gait cycles, C is the number of channels, T is the number of gait cycle frames, and V is the number of three-dimensional bone points. The gait sequence F is input into the pre-trained MCC-GCN model to obtain the abnormal gait type diagnosis of the patient to whom the gait sequence feature F belongs.

[0035] The beneficial effects of the invention are: only by using an ordinary optical camera or a depth camera to shoot the walking process of a patient with abnormal gait, the abnormal gait type of the patient can be identified using the obtained three-dimensional skeleton gait data. The model is relatively lightweight and does not change the characteristic dimension, which does not burden or make operation difficult for medical personnel and patients. At present, the global aging trend is intensifying, and the gait problem of the elderly has become a topic of concern to the whole society. This method can play an important role in medical diagnosis, rehabilitation assessment, sports training and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 : Flow chart of the method of the present invention;

[0037] Figure 2 : Schematic diagram of the overall structure of MCC-GCN;

[0038] Figure 3 : Schematic diagram of the start frame and end frame obtained using the key frame selection algorithm;

[0039] Figure 4 : Detailed structural diagram of the multi-scale cross-dimensional joint attention mechanism. DETAILED DESCRIPTION

[0040] Step 1: Build a spatiotemporal graph convolutional network (ST-GCN) for skeleton action recognition;

[0041] The skeleton sequence is represented by the 2D or 3D coordinates of each joint of the human body in each frame. The skeleton sequence is hierarchically represented using a space-time graph to construct a skeleton graph. An undirected space-time graph G = (V, E) is constructed on a skeleton sequence with N joints and T frames. It captures the connection between the joints in the body while considering the temporal association between frames. The joint nodes are divided into multiple subsets, including centripetal, eccentric, and root nodes, and different weights are assigned to each subset. Based on the characteristics of action recognition, a graph partitioning method is introduced. The distance from the root node to the center of gravity is used as the benchmark, marked as label = 0. For all adjacent nodes, they are classified according to the comparison of their distance to the center of gravity with the benchmark value: if the distance is less than the benchmark value, it is marked as a centripetal node, label = 1; if the distance is greater than the benchmark value, it is marked as a centrifugal node, label = 2. Graph convolution uses edges as weights to calculate the weighted average of node features. The graph convolution formula used is:

[0042]

[0043] Among them, z ti (v tj )=|{v tk |l ti (v tk )=l ti (v tj )}| is a normalization term used to balance the contribution of different subsets, w is a weight function that provides a weight matrix for inner product operation with the input sampling features, and v tj v ti The adjacent nodes of ti (v tj ) is the adjacent node v tj The label to which it belongs, l tiThe temporal convolution network (TCN) is used to extract the temporal information of the joint point changes. TCN is a one-dimensional convolution of size [1, t], whose convolution dimension is the time dimension. The convolution kernel is a structure of one row and multiple columns. Convolution is performed along the time dimension to extract the temporal features of the skeleton points.

[0044] Step 2: In order to better capture and integrate multi-scale and multi-dimensional gait features, a multi-scale cross-dimensional joint attention mechanism, referred to as MS-CDCA, is introduced on the basis of ST-GCN to obtain the MCC-GCN model for abnormal gait recognition.

[0045] The multi-scale cross-dimensional joint attention mechanism has a small number of parameters and moderate computational overhead. ST-GCN contains 10 layers of spatiotemporal graph convolutional layers. This method deploys a multi-scale cross-dimensional joint attention mechanism between every two spatiotemporal graph convolutional layers. The gait sequence feature F is input into ST-GCN and processed by the spatiotemporal graph convolutional layer to obtain the feature F. input The calculation process can be expressed as:

[0046] F input =ST-GCN(F)

[0047] Among them, ST-GCN(·) represents the spatiotemporal graph convolution layer before the attention module. The obtained F input is passed to the multi-scale cross-dimensional joint attention mechanism. The calculation process can be expressed as:

[0048] F atten =MS-CDCA(F input )

[0049] Among them, MCDCA(·) represents the multi-scale cross-dimensional joint attention mechanism. This method converts F input and the output F of the multi-scale cross-dimensional joint attention mechanism atten Connect along the channel dimension to get F output The calculation process can be expressed as:

[0050] F output =[F input ,F atten ]

[0051] Where [...] represents the residual connection. output It will be passed to the next spatiotemporal graph convolution layer and attention module to obtain the final feature representation F final In order to avoid reducing the accuracy of features, this method replaces the pooling layer of ST-GCN with a linear layer, and then obtains the final prediction result Output through a fully connected layer. The calculation process can be expressed as:

[0052] Output=FCN(Linear(Ffinasl ))

[0053] Among them, Linear(·) represents the linear layer and FCN(·) represents the fully connected layer.

[0054] Step 3: Use the abnormal gait dataset GIST to train MCC-GCN. The dataset contains 1 normal gait and 5 pathological abnormal gaits, which are simulated by 10 subjects.

[0055] The GIST dataset divides gait into one normal gait and five abnormal gaits; among them, abnormal gaits include analgesic gait, stride gait, stagger gait, stiff leg gait and gluteus medius gait; analgesic gait is manifested as lameness and reduced walking speed according to the injured part; stride gait is walking with toes touching the ground and lifting the lower limbs to start; stagger gait is manifested as swaying of the body on both sides when walking, similar to the walk of a duck; stiff leg gait has the phenomenon of circling gait, foot drop and hyperextension of knee joint; gluteus medius gait has one high and one low pelvis when walking; this dataset uses 6 cameras to collect three-dimensional bone information, with a total of 7200 samples. The dataset is divided into a training set: test set ratio of 8:2 and the MCC-GCN model is trained to obtain a pre-trained MCC-GCN model.

[0056] Step 4: Use the Kinect v2 depth camera to collect the 3D gait data of real abnormal gait patients, normalize it and input it into the pre-trained MCC-GCN to obtain the diagnosis result;

[0057] In practical application scenarios, for the abnormal gait data based on three-dimensional skeleton points of real abnormal gait patients collected, it is necessary to first align the posture of the gait cycle and normalize the world coordinate system to achieve unification in time and space dimensions. First, the complete gait cycle is identified and extracted in the continuous motion sequence, and the postures in different sequences are aligned for comparison and analysis. Specifically, inspired by Dynamic Time Warping (DTW), this method designs a key frame selection algorithm. The algorithm first converts the coordinates of the skeleton points in three-dimensional space into a more comparable joint rotation radian representation. Then, the Euclidean distance between each pair of vectors in the radian matrix is ​​calculated, and the inter-frame distance constraint is introduced. The best matching action is identified by finding the local minimum Euclidean distance in the distance matrix, so as to accurately locate the start frame and end frame of the gait cycle. The gait cycle of this method is defined as from the left heel to the next left heel touchdown. Then, this method uniformly samples the extracted complete gait cycle data, unifies the number and time order of key frames in different gait data, and performs time normalization by cutting frames. Next, this method performs spatial normalization on the obtained gait data. This method defines a reference height of 1.6874 meters. By calculating the ratio of the target height to the reference height, starting from the central bone point of the hip, the relative position of each bone point is gradually adjusted outward to maintain the integrity of the bone structure, so that the bone points of all characters are at the same scale. Then, this method unifies the coordinate system standard by adjusting the coordinate axis. This method adjusts the Y axis according to the height of the left and right ankles so that the character is located in a coordinate system with the ground as the origin. Then, the human physiological information calculation method is used to obtain the spatial information and physiological information of the character, and the Z-axis offset is calculated in combination with the pelvic position of the character to eliminate the influence of camera distance and viewing angle. This method eliminates the mutual differences in data acquisition through these preprocessing steps, and finally obtains a dimension of R N×C×T×V The gait sequence feature F is obtained, where N is the number of gait cycles, C is the number of channels, T is the number of gait cycle frames, and V is the number of three-dimensional bone points. The gait sequence F is input into the pre-trained MCC-GCN model to obtain the abnormal gait type of the patient to which the gait sequence feature F belongs.

[0058] Embodiment 1:

[0059] In order to improve the diagnostic efficiency and accuracy of patients with abnormal gait, provide doctors with more accurate diagnostic evidence, and provide more effective guidance for patients' treatment, the embodiment of the present invention provides an abnormal gait analysis method based on a multi-scale cross-dimensional joint attention mechanism, see Figure 1 As shown, it is a flowchart of an abnormal gait analysis method based on a multi-scale cross-dimensional joint attention mechanism provided by an embodiment of the present invention, the method includes steps 1 to 5, wherein:

[0060] Step 1: Build a spatiotemporal graph convolutional network (ST-GCN) for skeleton action recognition;

[0061] Step 2: In order to better capture and integrate multi-scale and multi-dimensional gait features, a multi-scale cross-dimensional joint attention mechanism, referred to as MS-CDCA, is introduced on the basis of ST-GCN to obtain the MCC-GCN model for abnormal gait recognition.

[0062] Step 3: Use the abnormal gait dataset GIST to train MCC-GCN. The dataset contains 1 normal gait and 5 pathological abnormal gaits, which are simulated by 10 subjects.

[0063] Step 4: Use the Kinect v2 depth camera to collect the three-dimensional gait data of real patients with abnormal gait, normalize it and input it into the pre-trained MCC-GCN to obtain the diagnosis results.

[0064] Different body parts have different scales, and even the same part in different frames may appear at different positions and scales. Therefore, the embodiment of the present invention uses a multi-scale cross-dimensional joint attention mechanism (MS-CDCA), which consists of three parallel branches to extract gait features for skeletal points, time series, and spatial configuration dimensions respectively. Each branch uses multi-scale one-dimensional convolution to encode multi-level context information along a specific dimension, and then applies cross-attention to achieve feature fusion between branches to construct a cross-dimensional joint attention structure. This design can effectively integrate multi-scale and multi-dimensional gait information and improve the discriminability and robustness of feature representation. The spatiotemporal graph convolutional network (ST-GCN) is a convolutional neural network based on a spatiotemporal graph, which not only considers the connection relationship between nodes, but also considers the temporal order relationship of nodes, so it is suitable for processing time series data. The embodiment of the present invention introduces MS-CDCA on the basis of ST-GCN, and this new network model is called MCC-GCN.

[0065] In order to efficiently acquire, process and fuse the rich features of abnormal gait, the embodiment of the present invention proposes a novel MCC-GCN model, which can effectively capture gait information of different scales and dimensions and perform deep fusion. Figure 2 , which is a schematic diagram of the overall structure of MCC-GCN provided by an embodiment of the present invention.

[0066] The keyframe selection algorithm first converts the coordinates of the skeleton points in 3D space into a more comparable joint rotation radian representation. Then, the Euclidean distance between each pair of vectors in the radian matrix is ​​calculated, and the inter-frame distance constraint is introduced. The best matching action is identified by finding the local minimum Euclidean distance in the distance matrix, thereby accurately locating the start and end frames of the gait cycle, see Figure 3As shown, the present invention implements and applies this method to analyze the gait data of seven real abnormal gait patients, and successfully identifies the starting frame and ending frame of the gait cycle of each patient.

[0067] Effectively capturing multi-scale information and establishing long-distance dependencies are crucial for abnormal gait analysis. Axial attention can effectively capture global context while avoiding introducing excessive computational complexity. Through multi-scale convolution and multiple cross attention, the embodiments of the present invention can achieve long-distance interaction and better capture multi-dimensional gait features, thereby more comprehensively analyzing the spatial structure information of the action and improving the accuracy of the analysis. Figure 4 As shown in FIG, a detailed structural diagram of a multi-scale cross-dimensional joint attention mechanism provided by an embodiment of the present invention. The attention mechanism is divided into three parallel branches, which extract features along the spatial dimension, the temporal dimension, and the skeleton point dimension respectively. Each branch consists of three 1D convolutions with different kernel sizes, which encode multi-scale context information along one dimension and then use cross attention to aggregate with features from the other two dimensions. Figure 4 Taking the spatial dimension branch at the top as an example, the specific operation of the multi-scale cross-dimensional joint attention mechanism is explained.

[0068] The first is the multi-scale convolution module. The dimension of the spatiotemporal graph convolution layer output is R N×C×T×V Gait characteristics F input Input to 1×1 convolution for channel dimension reduction, and its output F ′ With F input The same number of channels. The calculation process can be expressed as:

[0069] F ′ =Conv 1×1 (F input )

[0070] Among them, Conv 1×1 (·) represents 1×1 convolution. The embodiment of the present invention adopts the Gaussian Error Linear Unit (GELU) activation function as the post-activation function of all convolution operations, but in order to avoid redundant description, the activation function is omitted in the formula and model diagram of the embodiment of the present invention, and the following description of the model structure will also omit the activation function step.

[0071] In the embodiment of the present invention, F ′ Send it into a d×d convolution to get the preliminary feature F″. The calculation process can be expressed as:

[0072] F″=Conv d×d (F′)

[0073] Among them, Conv d×d(·) represents a d×d convolution. Next, three parallel one-dimensional convolutions are used to encode the feature F″. The outputs of the three convolutional layers are summed and fused with F″ and sent to a 1×1 convolution to obtain the spatial feature F of the gait. V The calculation process can be expressed as:

[0074]

[0075] in, Represents a one-dimensional convolution along the spatial dimension. For the time dimension branch in the middle and the bone point dimension branch at the bottom, the corresponding time feature F T and skeleton point feature F C It can be obtained by:

[0076]

[0077] in, represents a one-dimensional convolution along the time dimension, Represents a one-dimensional convolution along the dimension of the bone point. Given the three-dimensional gait feature F V 、F T and F C , the embodiment of the present invention passes them to the feature fusion module.

[0078] The embodiment of the present invention adopts a differentiated convolution kernel design strategy. For attention modules at different positions, their convolution kernel size parameters: a, b, c and d should be different to capture features of different scales. The d parameter is usually set to be slightly smaller than a, b, c, which helps to reduce computational complexity while maintaining the receptive field. As the network level deepens, the convolution kernel size should show a trend of gradually decreasing, which is conducive to extracting more refined and abstract features in deep networks. The specific parameter configuration should be carefully adjusted according to the characteristics of the data set to achieve the best performance.

[0079] The second is the feature fusion module. In order to better utilize the multi-scale features from three dimensions, the embodiment of the present invention calculates F V 、F T and F C Specifically, still taking the top branch as an example, the embodiment of the present invention converts F V As the key and value matrix, F T and F C The residual is connected and used as the query matrix. The calculation process can be expressed as:

[0080] F x =MHCA V ([F T ,F C ],F V,F V )

[0081] Among them, MHCA V (·,·,·) represents multi-head cross attention along the spatial dimension. For the middle branch and the bottom branch, the embodiment of the present invention uses a similar method to perform feature fusion along the time dimension and the skeleton point dimension. The calculation process can be expressed as:

[0082] F y =MHCA T ([F V ,F C ],F T ,F T )

[0083] F z =MHCA C ([F T ,F V ],F C ,F C )

[0084] Among them, MHCA T (·,·,·) represents multi-head cross attention along the time dimension, MHCA C (·,·,·) represents the multi-head cross attention along the dimension of the skeleton point. Given the three-dimensional fused gait feature F x 、F y and F z , the output of the proposed multi-scale cross-dimensional joint attention mechanism can be expressed as:

[0085] F atten =[F input ,Conv 1x1 ([F inpu( ,Conv 1x1 (F x ),Conv 1x1 (F y ),Conv 1x1 (F z )])]

[0086] Among them, the obtained fusion feature F atten The dimension is still R N×C×T×V .

[0087] In order to verify the performance of the proposed MCC-GCN model in the abnormal gait classification task, it was compared with the cutting-edge abnormal gait classification models and some traditional models on GIST:pathological gait. The cutting-edge models include the GRU model, ST-GCN model and AGS-GCN model used by the Gwangju Institute of Science and Technology in South Korea, and the traditional models include LSTM classifier, basic RNN classifier, convolutional neural network (CNN) classifier, deep neural network (DNN) classifier, k-nearest neighbor (k-NN) classifier, k-means classifier and random forest classifier. The model accuracy experimental results are shown in Table 1. It can be seen that the MCC-GCN model performs well on the public dataset GIST, and its accuracy is better than other comparison models.

[0088] Table 1: Accuracy table of the MCC-GCN model compared with other traditional models and models for abnormal gait classification.

[0089]

[0090] In order to comprehensively evaluate the effectiveness of the MCC-GCN model, Table 2 shows in detail the performance of the model on the three evaluation indicators of Sensitivity, Specificity and Precision. The results show that the MCC-GCN model performs well in terms of Sensitivity and Precision, which is better than the GRU model. In disease diagnosis, Sensitivity and Precision are more important because we hope to minimize missed diagnoses (FN) to avoid delayed diagnosis and treatment, which may lead to worsening of the disease. We also hope to avoid misdiagnosis (FP) as much as possible to avoid unnecessary examinations or treatments for patients.

[0091] Table 2: Performance accuracy table of MCC-GCN model and GRU model in terms of Sensitivity, Specificity and Precision indicators.

[0092]

[0093] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A gait analysis method based on a multi-scale cross-dimensional joint attention mechanism, characterized by: (1) Build a spatiotemporal graph convolutional network ST-GCN based on skeleton action recognition; (2) Based on ST-GCN, a multi-scale cross-dimensional joint attention mechanism, referred to as MS-CDCA, is introduced to capture and fuse multi-scale and multi-dimensional gait features to obtain the MCC-GCN model for abnormal gait recognition; (3) MCC-GCN was trained using the abnormal gait dataset GIST, which contains 1 normal gait and 5 pathological abnormal gaits simulated by 10 subjects; (4) The kinect v2 depth camera is used to collect the three-dimensional gait data of real patients with abnormal gait, which is normalized and input into the pre-trained MCC-GCN to obtain the diagnosis results.

2. The gait analysis method based on the multi-scale cross-dimensional joint attention mechanism according to claim 1 is characterized in that: In the above (1), the specific method is: 1.1) The skeleton sequence is represented by the 2D or 3D coordinates of each joint of the human body in each frame. The skeleton sequence is hierarchically represented using a spatiotemporal graph to construct a skeleton graph. An undirected spatiotemporal graph G = (V, E) is constructed on a skeleton sequence with N joints and T frames. It captures the connections between joints in the body while considering the temporal association between frames. The joint nodes are divided into multiple subsets, including centripetal, eccentric, and root nodes, and different weights are assigned to each subset. 1.2) Based on the characteristics of action recognition, a graph partitioning method is introduced. The distance from the root node to the center of gravity is used as the benchmark, marked as label = 0. For all adjacent nodes, they are classified according to the comparison between their distance to the center of gravity and the benchmark value: if the distance is less than the benchmark value, it is marked as a centripetal node, label = 1; if the distance is greater than the benchmark value, it is marked as a centrifugal node, label = 2; 1.3) Graph convolution uses edges as weights to calculate the weighted average of node features. The graph convolution formula used is: Among them, z ti (v tj )=|{v tk |l ti (v tk )=l ti (v tj )}| is a normalization term used to balance the contribution of different subsets, w is a weight function that provides a weight matrix for inner product operation with the input sampling features, and v tj v ti The adjacent nodes of ti (v tj ) is the adjacent node v tj The label to which it belongs, l ti is the mapping relationship of the subset; 1.4) Use the temporal convolutional network (TCN) to extract the temporal information of joint point changes. TCN is a one-dimensional convolution of size [1, t]. Its convolution dimension is the time dimension. The convolution kernel is a structure of one row and multiple columns. Convolution is performed along the time dimension to extract the temporal features of the skeleton points.

3. The gait analysis method based on multi-scale cross-dimensional joint attention mechanism according to claim 1, characterized in that: In the above (2), the specific method is: 2.1) The multi-scale cross-dimensional joint attention mechanism has a small number of parameters and moderate computational overhead. ST-GCN contains 10 layers of spatiotemporal graph convolutional layers, and a multi-scale cross-dimensional joint attention mechanism is deployed between every two spatiotemporal graph convolutional layers; 2.2) The gait sequence feature F is input into ST-GCN and processed by the spatiotemporal graph convolution layer to obtain the feature F input , the calculation process is: F input =ST-GCN(F) Among them, ST-GCN(·) represents the spatiotemporal graph convolution layer before the attention module, and the obtained F input It is passed to the multi-scale cross-dimensional joint attention mechanism, and the calculation process is: F atten =MS-CDCA(F input ) Among them, MCDCA(·) represents the multi-scale cross-dimensional joint attention mechanism, which converts F input and the output F of the multi-scale cross-dimensional joint attention mechanism atten Connect along the channel dimension to get F output , the calculation process is: F output =[F input ,F atten ] Among them, […] represents residual connection; 2.3) F output It will be passed to the next spatiotemporal graph convolution layer and attention module to obtain the final feature representation F final , the linear layer is used to replace the pooling layer of ST-GCN, and the final prediction result Output is obtained through the fully connected layer. The calculation process is: Output=FCN(Linear(F final )) Among them, Linear(·) represents the linear layer and FCN(·) represents the fully connected layer.

4. The gait analysis method based on multi-scale cross-dimensional joint attention mechanism according to claim 1, characterized in that: In the above (3), the specific method is: The GIST dataset divides gait into one normal gait and five abnormal gaits; among them, abnormal gaits include analgesic gait, stride gait, stagger gait, stiff leg gait and gluteus medius gait; analgesic gait is manifested as lameness and reduced walking speed according to the injured part; stride gait is walking with toes touching the ground and lifting the lower limbs to start; stagger gait is manifested as swaying of the body on both sides when walking, similar to the walk of a duck; stiff leg gait has the phenomenon of circling gait, foot drop and hyperextension of knee joint; gluteus medius gait has one high and one low pelvis when walking; this dataset uses 6 cameras to collect three-dimensional bone information, with a total of 7200 samples. The dataset is divided into a training set: test set ratio of 8:2 and the MCC-GCN model is trained to obtain a pre-trained MCC-GCN model.

5. The gait analysis method based on multi-scale cross-dimensional joint attention mechanism according to claim 1, characterized in that: In the above (4), the specific method is: 4.1) In actual application scenarios, for the abnormal gait data based on 3D skeleton points of real abnormal gait patients, the posture alignment of the gait cycle and the normalization of the world coordinate system are first performed to achieve unification in time and space dimensions; the complete gait cycle is identified and extracted in the continuous motion sequence, and the postures in different sequences are aligned; The coordinates of the bone points in the three-dimensional space are converted into the radian representation of the joint rotation, the Euclidean distance between each pair of vectors in the radian matrix is ​​calculated, and the inter-frame distance constraint is introduced. The best matching action is identified by finding the local minimum Euclidean distance in the distance matrix, and the start frame and end frame of the gait cycle are located. The cycle is defined as from the left heel to the next left heel touchdown; the extracted complete gait cycle data is uniformly sampled, the number and time order of key frames in different gait data are unified, and time normalization is performed by frame cutting; 4.2) Perform spatial normalization on the acquired gait data, define the reference height as 1.6874 meters, calculate the ratio of the target height to the reference height, start from the center bone point of the hip, and gradually adjust the relative position of each bone point outward to maintain the integrity of the bone structure, so that all the bone points of the characters are at the same scale; unify the coordinate system standard by adjusting the coordinate axis, adjust the Y axis according to the height of the left and right ankles, so that the character is located in the coordinate system with the ground as the origin; The human body physiological information calculation method is used to obtain the spatial information and physiological information of the character, and the Z-axis offset is calculated in combination with the character's pelvic position to eliminate the influence of camera distance and viewing angle; 4.3) After eliminating the differences in data collection through 4.1) and 4.2), the final dimension is R N×C×T×V The gait sequence feature F is obtained, where N is the number of gait cycles, C is the number of channels, T is the number of gait cycle frames, and V is the number of three-dimensional bone points. The gait sequence F is input into the pre-trained MCC-GCN model to obtain the abnormal gait type diagnosis of the patient to whom the gait sequence feature F belongs.

Images