Multi-head contextual attention gait phase classification method based on TCN network

Abstract

The present invention relates to the technical field of gait analysis, and in particular to a multi-head contextual attention gait phase classification method based on a TCN network, which is as follows: construct a model based on a TCN network and a multi-head contextual attention module, input the data collected by the sensor into the TCN network of the model for feature extraction, and obtain features; construct a multi-head contextual attention module, input the features into the module, and obtain fused features; input the fused features into a channel attention module to obtain comprehensive features; calculate the total loss of the model; input the comprehensive features into a linear layer classifier to obtain the final phase classification prediction score, and the category corresponding to the phase classification prediction score with the highest score is the predicted final category. The present invention can more accurately extract the temporal context features of sensor data and improve the accuracy of gait phase classification.

Description

Technical Field

[0001] The present invention relates to the technical field of gait analysis, and in particular to a multi-head contextual attention gait phase classification method based on a TCN network. Background Art

[0002] Current gait phase classification technologies mainly use computer vision and machine learning algorithms to determine gait phases by analyzing the changing trends of gait data. These technologies have important applications in the medical and rehabilitation fields, helping medical staff to more objectively and timely evaluate walking status and monitor abnormal gait behavior.

[0003] At present, artificial intelligence algorithms represented by deep learning have achieved outstanding results in the fields of sensors, signal processing, gait analysis, etc. with various algorithms and powerful computing resources. For gait phase classification, many scholars have proposed a large number of excellent evaluation algorithms, including recurrent neural networks, convolutional neural networks, etc. These algorithms can classify gait phases according to the changing rules of gait signals collected by sensors, which can reduce the workload of medical staff to a certain extent, and timely evaluate walking status and monitor abnormal gait behavior. However, there are still some problems in this field. On the one hand, gait is a dynamic process, and there is obvious temporal continuity between each stage. The characteristics of different gait phases usually depend on the previous and subsequent states, and the long-term dependence characteristics of the data are difficult to capture; on the other hand, gait data usually comes from multiple sensor channels (such as accelerometers and gyroscopes), each channel captures different types of motion information, and different channels may show unique characteristics in different gait stages. The feature differences between different channels are difficult to capture. Paying attention to the long-term dependence features and channel feature differences in sensor data is very important for gait phase classification. To this end, in-depth research on the long-term dependence features and channel features in sensor data is a necessary development.

[0004] Therefore, the present invention proposes a multi-head contextual attention gait phase classification method based on TCN network to solve the above problems. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a multi-head contextual attention gait phase classification method based on TCN network. By introducing the MHMA multi-head contextual attention module and the SE module on the basis of the existing TCN network, the improved TCN network can more accurately extract the temporal context features of sensor data and improve the accuracy of gait phase classification.

[0006] The technical solution to the technical problem of the present invention is a multi-head contextual attention gait phase classification method based on TCN network, which is as follows:

[0007] S1. Build a model based on TCN network and multi-head context attention module, input the data collected by the sensor into the TCN network of the model for feature extraction, and obtain the features of accelerometer and gyroscope data ;

[0008] S2. Construction Multi-head contextual attention module, The module consists of a linear layer and four parallel layers. Contextual Attention Module, The module consists of a temporal attention unit and a channel attention unit set in parallel. Input to Module, get fusion features ;

[0009] S3. Fusion features Input to The channel attention module extracts the transition features between gait phases and finally obtains the comprehensive features. ;

[0010] S4. Calculation is added based on the original TCN network Modules and The total loss of the improved TCN network after the channel attention module. The improved TCN network is optimized. The total loss includes the cross entropy loss. and difference loss ;

[0011] S5. Comprehensive features Input to the linear layer classifier to get the final phase classification prediction score , the category corresponding to the phase classification prediction score with the highest score is the final category predicted.

[0012] S1 is as follows:

[0013] The model based on TCN network and multi-head context attention module includes TCN network, MHMA multi-head context attention module, Channel attention module and linear layer classifier;

[0014] The accelerometer and gyroscope data collected by the sensor are input into the TCN network of the model for feature extraction to obtain the feature , where the TCN network consists of six residual blocks connected in series.

[0015] S2 is as follows:

[0016] S2.1. Features extracted from the TCN network Input to Module, first through the linear layer feature Processing to obtain features , the specific process is as follows:

[0017] ,

[0018] in, represents the linear layer operation, feature and Features The dimensions of , express Batch size, Indicates the number of channels;

[0019] S2.2. Connect four context attention modules in parallel Recorded as , , and , four contextual attention modules , , and Both include a temporal attention unit and a channel attention unit;

[0020] The characteristics Enter into four Modules , , , In the above example, we obtain the features , , and ,feature , , and The dimensions of ;

[0021] S2.3, through the partition constraint loss function to four Modules , , and Constraints are calculated as follows:

[0022] ,

[0023] Where N represents the total number of input samples, L represents the total number of categories, and L=4 is set. Indicates The sample in The confidence score on the class ranges from , Indicates The sample in The variance of the predicted distribution over the class, K represents a learnable hyperparameter;

[0024] For four Modules , , and After the constraints are applied, the module , , , Focus on one gait category respectively, and then get the features , , and , means that we only focus on the features of gait phase category 1. Indicates that only the features of gait phase category 2 are concerned. means that we only focus on the features of gait phase category 3. It means that we only focus on the features of gait phase category 4;

[0025] S2.4 will feature , , and Perform weighted fusion, the specific calculation is as follows:

[0026] ,

[0027] in, , , and Represents four different weight parameters, ,and , , , are greater than 0, Represents the fusion feature containing time information and channel information .

[0028] S2.2 is as follows:

[0029] S2.2.1、Characterize Enter the first Modules After that, it passes through the time attention unit and the channel attention unit respectively;

[0030] The temporal attention unit consists of two network layers, both of which are dilated convolutional layers with a convolution kernel size of 1×1, a step size of 1, and a dilation rate of 4. The second network layer is followed by a network component, Activation function;

[0031] The channel attention unit consists of three network layers. The first network layer is an adaptive average pooling layer. The second and third network layers are both linear layers. The third network layer is followed by a network component. Activation function;

[0032] In the temporal attention unit, the feature First, the feature is obtained by passing the dilated convolution layer of the first network layer. , and then the feature Input to the dilated convolutional layer of the second network layer to obtain features , and then the feature Input to the ReLU activation function to get the attention score of the temporal attention unit , and finally the input features of the temporal attention unit With attention score Multiply to get the time feature ;

[0033] In the channel attention unit, the feature First, the adaptive average pooling layer of the first network layer is used to obtain the feature , and then the feature Input to the linear layer of the second network layer to obtain the features , and then the feature Input to the linear layer of the third network layer to obtain the feature , and then the feature Input to the ReLU activation function to get the attention score of the channel attention unit , and finally the input features of the channel attention unit With attention score Multiply to get channel features ;

[0034] The time feature and channel characteristics Add and get the features ;

[0035] S2.2.2, Similarly, the feature Enter the second Modules After that, the feature is obtained through the operation in step S2.2.1 ;

[0036] The characteristics Enter the third Modules After that, the feature is obtained through the operation in step S2.2.1 ;

[0037] The characteristics Enter the fourth Modules After that, the feature is obtained through the operation in step S2.2.1 .

[0038] S3 is as follows:

[0039] The fusion features Input to Module, in the fusion feature Based on this, we continue to extract the transition features between different gait phases to obtain comprehensive features including gait data time features, channel features, and transition features. .

[0040] S4 is as follows:

[0041] Calculate the fusion features through the mean square error loss MSE and comprehensive characteristics The difference between them is used to obtain the fusion feature and comprehensive characteristics The difference between the losses , the smaller the difference loss, the The more accurate the module calculation, the more accurate the specific process is as follows:

[0042] ,

[0043] in, express and The dimension of Indicates the index of the dimension;

[0044] Cross Entropy Loss in Multi-Classification Tasks The calculation method is:

[0045] ,

[0046] in, Represents the true category label of the input data, expressed in one-hot encoding form, with the true category represented as 1 and other categories represented as 0. represents the phase category index, represents the predicted probability, Indicates the total number of phase categories, set ;

[0047] The total loss of the model based on the TCN network and the multi-head contextual attention module The specific calculation is as follows:

[0048] .

[0049] S5 is as follows:

[0050] The comprehensive characteristics obtained Input to the linear layer classifier, the total loss The training is carried out under the supervision of , and finally the phase classification score predicted by the improved TCN network is obtained ,in, represents the phase category index, represents the total number of phase categories, , , the phase classification score is calculated as follows:

[0051] ,

[0052] in, Represents a linear layer operation;

[0053] Finally, the scores predicted from the phase classification The phase classification score with the highest score is selected, and the category corresponding to the phase classification score is the final category predicted.

[0054] The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0055] The present invention provides a multi-head contextual attention gait phase classification method based on TCN network, which can simultaneously focus on the time dimension features and channel dimension features of sensor data, and pay attention to the main features of gait phase in the time context to a great extent, so that the network can more accurately extract the time context features of sensor data; at the same time, the four-head contextual attention module is used to enable the model to learn feature information from different feature angles of the input, and then aggregate the information extracted from different feature angles, so as to capture more comprehensive feature information, thereby improving the model's ability to distinguish different gait phases, and greatly improving the accuracy of gait phase classification;

[0056] The multi-head contextual attention module proposed in the present invention pays great attention to the temporal context characteristics of the sensor data, thereby improving the model's ability to distinguish different gait phases and greatly improving the accuracy of gait phase classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0058] Figure 1 It is a schematic diagram of the network flow in the present invention.

[0059] Figure 2 Schematic diagram of the multi-head contextual attention module MHMA in the present invention.

[0060] Figure 3 It is the context attention module TCF in the present invention.

[0061] Figure 4 The gait phase classification result of the fused acceleration sensor data waveform. DETAILED DESCRIPTION

[0062] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below.

[0063] Example 1

[0064] Combination Figures 1 to 3 As shown, the present invention provides a multi-head contextual attention gait phase classification method based on TCN network, which is as follows:

[0065] S1. Build a model based on TCN network and multi-head context attention module, input the data collected by the sensor into the TCN network of the model for feature extraction, and obtain the features of accelerometer and gyroscope data ;

[0066] S2. Construction Multi-head contextual attention module, The module consists of a linear layer and four parallel layers. Contextual Attention Module, The module consists of a temporal attention unit and a channel attention unit set in parallel. Input to Module, get fusion features ;

[0067] S3. Fusion features Input to The channel attention module extracts the transition features between gait phases and finally obtains the comprehensive features. ;

[0068] S4. Calculation is added based on the original TCN network Modules and The total loss of the improved TCN network after the channel attention module. The improved TCN network is optimized. The total loss includes the cross entropy loss. and difference loss ;

[0069] S5. Comprehensive features Input to the linear layer classifier to get the final phase classification prediction score , the category corresponding to the phase classification prediction score with the highest score is the final category predicted.

[0070] S1 is as follows:

[0071] The model based on TCN network and multi-head context attention module includes TCN network, MHMA multi-head context attention module, Channel attention module and linear layer classifier;

[0072] The accelerometer and gyroscope data collected by the sensor are input into the TCN network of the model for feature extraction to obtain the feature , where the TCN network consists of six residual blocks connected in series.

[0073] S2 is as follows:

[0074] S2.1. Features extracted from the TCN network Input to Module, first through the linear layer feature Processing to obtain features , the specific process is as follows:

[0075] ,

[0076] in, represents the linear layer operation, feature and Features The dimensions of , express Batch size, Indicates the number of channels;

[0077] S2.2. Connect four context attention modules in parallel Recorded as , , and , four contextual attention modules , , and Both include a temporal attention unit and a channel attention unit;

[0078] The characteristics Enter into four Modules , , , In the above example, we obtain the features , , and ,feature , , and The dimensions of ;

[0079] S2.3, through the partition constraint loss function to four Modules , , and Constraints are calculated as follows:

[0080] ,

[0081] Where N represents the total number of input samples, L represents the total number of categories, and L=4 is set. Indicates The sample in The confidence score on the class ranges from , Indicates The sample in The variance of the predicted distribution over the class, K represents a learnable hyperparameter;

[0082] For four Modules , , and After the constraints are applied, the module , , , Focus on one gait category respectively, and then get the features , , and , means that we only focus on the features of gait phase category 1. Indicates that only the features of gait phase category 2 are concerned. Indicates that only the features of gait phase category 3 are concerned. It means that we only focus on the features of gait phase category 4;

[0083] S2.4 will feature , , and Perform weighted fusion, the specific calculation is as follows:

[0084] ,

[0085] in, , , and Represents four different weight parameters, ,and , , , are greater than 0, Represents the fusion feature containing time information and channel information .

[0086] S2.2 is as follows:

[0087] S2.2.1、Characterize Enter the first Modules After that, it passes through the time attention unit and the channel attention unit respectively;

[0088] The temporal attention unit consists of two network layers, both of which are dilated convolutional layers with a convolution kernel size of 1×1, a step size of 1, and a dilation rate of 4. The second network layer is followed by a network component, Activation function;

[0089] The channel attention unit consists of three network layers. The first network layer is an adaptive average pooling layer. The second and third network layers are both linear layers. The third network layer is followed by a network component. Activation function;

[0090] In the temporal attention unit, the feature First, the feature is obtained by passing the dilated convolution layer of the first network layer. , and then the feature Input to the dilated convolutional layer of the second network layer to obtain features , and then the feature Input to the ReLU activation function to get the attention score of the temporal attention unit , and finally the input features of the temporal attention unit With attention score Multiply to get the time feature ;

[0091] In the channel attention unit, the feature First, the adaptive average pooling layer of the first network layer is used to obtain the feature , and then the feature Input to the linear layer of the second network layer to obtain the features , and then the feature Input to the linear layer of the third network layer to obtain the feature , and then the feature Input to the ReLU activation function to get the attention score of the channel attention unit , and finally the input features of the channel attention unit With attention score Multiply to get channel features ;

[0092] The time feature and channel characteristics Add and get the features ;

[0093] S2.2.2, Similarly, the feature Enter the second Modules After that, the feature is obtained through the operation in step S2.2.1 ;

[0094] The characteristics Enter the third Modules After that, the feature is obtained through the operation in step S2.2.1 ;

[0095] The characteristics Enter the fourth Modules After that, the feature is obtained through the operation in step S2.2.1 .

[0096] S3 is as follows:

[0097] The fusion features Input to Module, in the fusion feature Based on this, we continue to extract the transition features between different gait phases to obtain comprehensive features including gait data time features, channel features, and transition features. .

[0098] S4 is as follows:

[0099] Calculate the fusion features through the mean square error loss MSE and comprehensive characteristics The difference between them is used to obtain the fusion feature and comprehensive characteristics The difference between the losses , the smaller the difference loss, the The more accurate the module calculation, the more accurate the specific process is as follows:

[0100] ,

[0101] in, express and The dimension of Indicates the index of the dimension;

[0102] Cross Entropy Loss in Multi-Classification Tasks The calculation method is:

[0103] ,

[0104] in, Represents the true category label of the input data, expressed in one-hot encoding form, with the true category represented as 1 and other categories represented as 0. represents the phase category index, represents the predicted probability, Indicates the total number of phase categories, set ;

[0105] The total loss of the model based on the TCN network and the multi-head contextual attention module The specific calculation is as follows:

[0106] .

[0107] S5 is as follows:

[0108] The comprehensive characteristics obtained Input to the linear layer classifier, the total loss The training is carried out under the supervision of , and finally the phase classification score predicted by the improved TCN network is obtained ,in, represents the phase category index, represents the total number of phase categories, , , the phase classification score is calculated as follows:

[0109] ,

[0110] in, Represents a linear layer operation;

[0111] Finally, the scores predicted from the phase classification The phase classification score with the highest score is selected, and the category corresponding to the phase classification score is the final category predicted.

[0112] Example 2

[0113] In order to demonstrate the beneficial effects of the present invention, the method of the present invention is experimentally compared with other existing gait phase classification methods. As shown in Table 1, the experimental results are compared on the classification accuracy Acc. The higher the classification accuracy Acc value, the better the performance of the method used. It can be seen from Table 1 that the method of the present invention achieves the best result in the comparison of this indicator. Compared with other methods, the present invention shows a strong generalization ability and can classify gait phases well. The present invention has a wide range of applications in clinical medicine and can help medical staff to evaluate walking status more objectively and timely and monitor abnormal gait behavior.

[0114] Table 1 Comparison of the results of the method of the present invention with other existing gait classification methods

[0115]

[0116] Example 3

[0117] In order to prove the progress made by the method of the present invention, ablation experiments were performed based on the data collected by the sensors to verify the classification accuracy of the TCN network, TCN network + SE module and the method of the present invention in gait phase classification. The method of the present invention is based on the TCN network and integrates the SE module and the multi-head contextual attention module MHMA. It can be seen from the results shown in Table 2 that the method of the present invention achieves a higher classification accuracy, which can improve the network's ability to capture the long-term dependencies of sensor data, thereby improving the model's gait phase classification ability.

[0118] Table 2. Comparison results of ablation experiments

[0119]

[0120] Example 4

[0121] like Figure 4 As shown, the present invention provides the result of gait phase classification on the gait data set GEDS by the method of the present invention. Through the processing of the method of the present invention, the fused acceleration sensor data waveform graph input in the gait data set GEDS is classified, such as Figure 4 As shown in the figure, the waveform is divided into four gait phase types, namely the first contact, weight-bearing reaction period, the end of the support phase, and the real position of the early swing phase, corresponding to Figure 4 The curve in the box is Figure 4It can be seen from the classification results that the network model in the present invention performs satisfactorily in gait phase classification and shows strong stability and consistency.

[0122] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-head contextual attention gait phase classification method based on TCN network, characterized by: The following steps are involved: S1. Build a model based on TCN network and multi-head context attention module, input the data collected by the sensor into the TCN network of the model for feature extraction, and obtain the features of accelerometer and gyroscope data ; S2. Construction Multi-head contextual attention module, The module consists of a linear layer and four parallel layers. Contextual Attention Module, The module consists of a temporal attention unit and a channel attention unit set in parallel. Input to Module, get fusion features ; S3. Fusion features Input to The channel attention module extracts the transition features between gait phases and finally obtains the comprehensive features. ; S4. Calculation is added based on the original TCN network Modules and The total loss of the improved TCN network after the channel attention module. The improved TCN network is optimized. The total loss includes the cross entropy loss. and difference loss ; S5. Comprehensive features Input to the linear layer classifier to get the final phase classification prediction score , the category corresponding to the phase classification prediction score with the highest score is the final category predicted.

2. The multi-head contextual attention gait phase classification method based on TCN network according to claim 1 is characterized in that: S1 is as follows: The model based on TCN network and multi-head context attention module includes TCN network, MHMA multi-head context attention module, Channel attention module and linear layer classifier; The accelerometer and gyroscope data collected by the sensor are input into the TCN network of the model for feature extraction to obtain the feature , where the TCN network consists of six residual blocks connected in series.

3. The multi-head contextual attention gait phase classification method based on TCN network according to claim 2 is characterized in that: S2 is as follows: S2.

1. Features extracted from the TCN network Input to Module, first through the linear layer feature Processing to obtain features , the specific process is as follows: ， in, represents the linear layer operation, feature and Features The dimensions of , express batch size, Indicates the number of channels; S2.

2. Connect four context attention modules in parallel Recorded as , , and , four contextual attention modules , , and Both include a temporal attention unit and a channel attention unit; The characteristics Enter into four Modules , , , In the above example, we obtain the features , , and ,feature , , and The dimensions of ; S2.3, through the partition constraint loss function to four Modules , , and Constraints are calculated as follows: ， Where N represents the total number of input samples, L represents the total number of categories, and L=4 is set. Indicates The sample in The confidence score on the class ranges from , Indicates The sample in The variance of the predicted distribution over the class, K represents a learnable hyperparameter; For four Modules , , and After the constraints are applied, the module , , , Focus on one gait category respectively, and then get the features , , and , means that we only focus on the features of gait phase category 1. Indicates that only the features of gait phase category 2 are concerned. means that we only focus on the features of gait phase category 3. It means that we only focus on the features of gait phase category 4; S2.4 will feature , , and Perform weighted fusion, the specific calculation is as follows: ， in, , , and Represents four different weight parameters, ,and , , , are greater than 0, Represents the fusion feature containing time information and channel information .

4. The multi-head contextual attention gait phase classification method based on TCN network according to claim 3 is characterized in that: S2.2 is as follows: S2.2.1、Characterize Enter the first Modules After that, it passes through the time attention unit and the channel attention unit respectively; The temporal attention unit consists of two network layers, both of which are dilated convolutional layers with a convolution kernel size of 1×1, a step size of 1, and a dilation rate of 4. The second network layer is followed by a network component, Activation function; The channel attention unit consists of three network layers. The first network layer is an adaptive average pooling layer. The second and third network layers are both linear layers. The third network layer is followed by a network component. Activation function; In the temporal attention unit, the feature First, the feature is obtained by passing the dilated convolution layer of the first network layer. , and then the feature Input to the dilated convolutional layer of the second network layer to obtain features , and then the feature Input to the ReLU activation function to get the attention score of the temporal attention unit , and finally the input features of the temporal attention unit With attention score Multiply to get the time feature ; In the channel attention unit, the feature First, the adaptive average pooling layer of the first network layer is used to obtain the feature , and then the feature Input to the linear layer of the second network layer to obtain the features , and then the feature Input to the linear layer of the third network layer to obtain the feature , and then the feature Input to the ReLU activation function to get the attention score of the channel attention unit , and finally the input features of the channel attention unit With attention score Multiply to get channel features ; The time feature and channel characteristics Add and get the features ; S2.2.2, Similarly, the feature Enter the second Modules After that, the feature is obtained through the operation in step S2.2.1 ; The characteristics Enter the third Modules After that, the feature is obtained through the operation in step S2.2.1 ; The characteristics Enter the fourth Modules After that, the feature is obtained through the operation in step S2.2.1 .

5. The multi-head contextual attention gait phase classification method based on TCN network according to claim 4 is characterized in that S3 The details are as follows: The fusion features Input to Module, in the fusion feature Based on this, we continue to extract the transition features between different gait phases to obtain comprehensive features including gait data time features, channel features, and transition features. .

6. The multi-head contextual attention gait phase classification method based on TCN network according to claim 5 is characterized in that: S4 is as follows: Calculate the fusion features through the mean square error loss MSE and comprehensive characteristics The difference between them is used to obtain the fusion feature and comprehensive characteristics The difference between the losses , the smaller the difference loss, the The more accurate the module calculation, the more accurate the specific process is as follows: ， in, express and The dimension of Indicates the index of the dimension; Cross Entropy Loss in Multi-Classification Tasks The calculation method is: ， in, Represents the true category label of the input data, expressed in one-hot encoding form, with the true category represented as 1 and other categories represented as 0. represents the phase category index, represents the predicted probability, Indicates the total number of phase categories, set ; The total loss of the model based on the TCN network and the multi-head contextual attention module The specific calculation is as follows: 。 7. The multi-head contextual attention gait phase classification method based on TCN network according to claim 6 is characterized in that: S5 is as follows: The comprehensive characteristics obtained Input to the linear layer classifier, the total loss The training is carried out under the supervision of , and finally the phase classification score predicted by the improved TCN network is obtained ,in, represents the phase category index, represents the total number of phase categories, , , the phase classification score is calculated as follows: ， in, Represents a linear layer operation; Finally, the scores predicted from the phase classification The phase classification score with the highest score is selected, and the category corresponding to the phase classification score is the final category predicted.

Images