Taijiquan action recognition method of hybrid cascade architecture

Abstract

The invention belongs to the technical field of action recognition, and particularly relates to a shadowboxing action recognition method of a hybrid cascade architecture. According to the technical scheme provided by the invention, the processed electromyographic signals are mainly subjected to Grubrum angle field processing, time sequence information is enriched, feature extraction is further carried out by utilizing a Grubrum angle field time sequence, the time sequence information of the electromyographic signals is convenient to capture, and the recognition stability and precision are improved; and then full extraction of high-dimensional features is performed through a double-flow CNN structure, and consumption of computing resources is reduced while high-dimensional feature extraction efficiency is maintained by means of the local receptive field advantage of the CNN. And finally, providing a cascade fusion transformer structure, and effectively fusing high-dimensional features obtained by an electromyographic signal diagram branch emphasizing space information and a Gramer angle field time sequence diagram branch emphasizing time sequence information to obtain rich high-dimensional feature representation. According to the method, the Tai Chi action recognition precision is guaranteed under the condition that the requirement for computing resources is low.

Description

Technical Field

[0001] The present invention belongs to the technical field of motion recognition, and in particular to a Tai Chi motion recognition method with a hybrid cascade architecture. Background Art

[0002] Tai Chi is a traditional martial art originating from China. It is an internal martial art that emphasizes the principle of using softness to overcome hardness and using stillness to overcome motion. Tai Chi is widely believed to enhance muscle flexibility and strength, improve balance, relieve stress and anxiety, promote cardiovascular health, improve concentration and focus, and enhance the immune system. These benefits make Tai Chi a popular healthy exercise suitable for people of all ages. With the rapid development of wearable devices, Tai Chi motion recognition can be achieved by combining electromyographic signal data and deep learning models, providing more possibilities for fitness assistance, sports training and other fields.

[0003] The patent with publication number CN116304833A provides a Tai Chi action recognition method based on native wavelet. First, the signal data is collected and preprocessed, and then the fixed-scale empirical mode decomposition is used to obtain the connotation modal component, and the connotation modal components of the same scale are fused, and then the amplitude-frequency component threshold filtering and fusion are performed to obtain the native wavelet, and multi-channel fusion smooth reconstruction is performed to obtain the fused native wavelet, and finally the fused native wavelet is used to perform Tai Chi action recognition from the continuous signal.

[0004] The patent with publication number CN116340824A provides a method for myoelectric signal action recognition based on convolutional neural network. First, the original data is enhanced by amplitude skew and white noise, and the signal is converted into a multi-channel two-dimensional image sample using a space filling curve algorithm to enhance the feature extraction ability of the model. On the basis of the convolutional neural network, a long short-term memory module is added to further extract the time series characteristics of the signal. Finally, the action recognition is performed through the extracted time series characteristics.

[0005] The above-mentioned method for Tai Chi movement recognition based on electromyographic signals does not extract high-dimensional features, and the feature extraction is insufficient, which affects the accuracy of Tai Chi movement recognition. Summary of the invention

[0006] To solve the above problems, the present invention proposes a Tai Chi movement recognition method based on a hybrid cascade architecture, which uses a dual-stream CNN to extract dual-modal high-dimensional features, and then proposes a cascade fusion transformer to fuse high-dimensional features to obtain rich and effective feature representation, which helps to improve the accuracy of Tai Chi movement recognition.

[0007] The technical solution of the present invention is:

[0008] A Tai Chi action recognition method based on a hybrid cascade architecture, characterized in that it comprises the following steps:

[0009] S1, collecting data sets, specifically, collecting electromyographic signals generated during Tai Chi exercise through electromyographic sensors, taking the data collected by each sensor as a channel data, taking all channel data obtained at each observation time as the observation data obtained at the current time, and collecting data at the set time from the start time to the end time of the Tai Chi exercise, so as to obtain a data set of a whole set of Tai Chi moves;

[0010] S2, processing the data set, including denoising the electromyographic signal of each channel to obtain electromyographic signal data, and then obtaining the electromyographic signal sequence diagram of each channel according to the acquisition time series of each channel data;

[0011] The denoised electromyographic signal data is used to obtain the time series characteristic graph after Gram angle fieldization;

[0012] S3, build a Tai Chi action recognition model based on a hybrid cascade architecture, including a CNN feature extraction module, a cascade fusion transformer, a fully connected layer, a Softmax layer, and a Tai Chi action recognition module;

[0013] The CNN feature extraction module includes two branches with the same structure, which respectively extract features from the electromyographic signal sequence diagram and the time series feature diagram;

[0014] The cascade fusion transformer consists of two multi-head attention layers and one fully connected layer. The two features output by the CNN feature extraction module are fed to different multi-head attention layers as different roles in the cross attention mechanism. The attention calculation result of each feature is obtained through the multi-head attention mechanism, and then complementary fusion is performed to obtain two fusion outputs respectively. Finally, the two fusion outputs are connected along the last dimension and input into the fully connected layer, and the fully connected layer outputs multimodal features.

[0015] The obtained multimodal features are sequentially passed through the fully connected layer and the Softmax layer to obtain the probability values ​​of different action categories for the single-channel features. Finally, the probability values ​​corresponding to each single-channel feature are added in the Tai Chi action recognition module to obtain the probability value of each category predicted by the full-channel features, and the category with the highest probability value is the identified Tai Chi action category.

[0016] S4, using the obtained data set to train the constructed Tai Chi action recognition model, using cross entropy as the loss function in the training, and obtaining a trained Tai Chi action recognition model;

[0017] S5. Input the acquired target action into the trained Tai Chi action recognition model for recognition.

[0018] Furthermore, when collecting data in S1, a total of 30 channels of data are collected, corresponding to 30 muscles of the upper and lower limbs; at the same time, it is set that there are 24 moves in Tai Chi, and the upper and lower limbs are collected for each move, and each move is collected 7 times, and the signal sampling rate is set to 1000Hz; the collected electromyographic signal is represented as S i ,1≤i≤30, there are n observation moments in total.

[0019] Furthermore, the specific method of S2 is:

[0020] For S i Filtering is performed to remove power line interference noise through a 50Hz notch filter, and motion artifacts and high-frequency noise are eliminated through a 20Hz-450Hz bandpass filter. Full-wave rectification is then performed to obtain a time series of n observations:

[0021] S i ={s i1 ,s i2 ,…,s in}

[0022] Finally, use the python CV library to convert it into an EMG signal sequence diagram;

[0023] Based on the processed electromyographic signal data S i Establishing the Gram angle field, that is, establishing the time series characteristic diagram:

[0024] Definitions ij represents the jth observation value of the i-th channel, i represents the i-th channel, j represents the j-th time observation value, 1≤j≤n, for S i Normalize so that all values ​​fall within [-1,1] to obtain normalized data definition Represents the jth observation value of the i-th channel after normalization:

[0025]

[0026]

[0027] Polar coordinates are used to preserve the absolute time relationship of the sequence, encoding the value as the cosine of the angle and the timestamp as the radius:

[0028]

[0029]

[0030] φ={φ i1 ,φ i2 ,…,φ in}

[0031] Among them, φ ij and r ij is the angle and radius in polar coordinates obtained by converting the signal data into polar coordinates, t j For time series The timestamp in is the time corresponding to the jth observation value. Since 1≤j≤n, the time series There are n timestamps in total, N is the difference between the last time point and the start time point, that is, N = t n -t1;

[0032] Finally, the Gram angle field GASF is expressed as follows:

[0033] GASF=[cos(φ ip +φ iq )]

[0034] Among them, GASF is an n×n matrix, φ ip Represents the radian value converted by the arc cosine formula from the pth observation value of the time series of the ith channel after normalization, φ iq Represents the radian value converted from the qth observation value of the time series of the ith channel after normalization by the arc cosine formula, 1≤p≤n, 1≤q≤n;

[0035] The processed GASF matrix data is converted into a time series feature map using the Python CV library.

[0036] Furthermore, the two branches in the CNN feature extraction module are composed of three convolutional layers. After the convolution operation, each convolutional layer is processed using an activation function, and then a batch normalization operation is performed. The two branches are defined to output features respectively. and a,b=1,2,a≠b.

[0037] Furthermore, the specific method of the cascade fusion transformer for data fusion is:

[0038] For features is d x dimensional vector, the input vector query Q of each head h h 、key K h and value V h Computed from the learned linear projection:

[0039]

[0040] in, is the parameter matrix, d h Set to d x / H, H represents the number of multiple heads; the query-key-value vector is mapped to the head h through the proportional dot product z On the vector:

[0041]

[0042] in, represents the self-attention operation;

[0043] The outputs of all heads are connected and connected by weights The linear transformation of is projected to obtain the multi-head self-attention calculation result:

[0044]

[0045] The same reason Processing, and finally the two input features and Complementary fusion is performed to obtain two fusion outputs and

[0046]

[0047]

[0048] The fusion output of the two cross-attention layers and Connect along the last dimension to obtain the merged multimodal features Optimization is performed using residual connections and layer normalization functions:

[0049]

[0050] Furthermore, The output after inputting into the fully connected layer is:

[0051]

[0052] Among them, q is a vector of length C, C is the total number of categories, corresponding to the moves of Tai Chi, W is the weight matrix of the fully connected layer, and b is the bias vector;

[0053] The Softmax layer obtains the probability values ​​of different action categories for single-channel features as follows:

[0054]

[0055] Among them, q iis the i-th element of the fully connected layer output, representing the unnormalized score of the i-th category, and pi is the probability value of the i-th category predicted by the single channel feature;

[0056] The Tai Chi action recognition module adds the probability values ​​corresponding to each single channel feature to obtain the probability value P of the i-th category predicted by the full channel feature. i , select the category with the highest probability as the prediction output of the model, which is achieved by finding the index of the maximum value in the probability distribution P. The category with the highest probability value is the identified Tai Chi action category:

[0057] y=argmax 1≤i≤C (P i ).

[0058] The beneficial effect of the present invention is that, compared with the existing technology, the technical solution proposed by the present invention enriches the timing information of the electromyographic signal by performing Gram angle fieldization on the electromyographic signal, and the dual-branch data fusion of the hybrid cascade architecture obtains a better feature representation, thereby ensuring the accuracy of Tai Chi movement recognition with low requirements on computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is the overall flow chart of the present invention.

[0060] Figure 2 The figure is a flow chart of hybrid cascade architecture.

[0061] Figure 3 Schematic diagram of the cascade fusion transformer structure. DETAILED DESCRIPTION

[0062] The technical principle and scheme of the present invention are described in detail below with reference to the accompanying drawings:

[0063] like Figure 1 As shown, the present invention mainly comprises the following steps:

[0064] 1) EMG data collection and data set construction

[0065] The subjects wore electromyographic sensors while performing Tai Chi exercises, collected the electromyographic signals generated during each move, and constructed a data set for the entire set of Tai Chi moves.

[0066] This dataset was collected using the Delsys Trigno device, with 30 channels of sEMG data collected, including 30 muscles in the upper and lower limbs. Data from 22 healthy subjects were collected, with an age range of 20±3 years, a height range of 170±10cm, and a weight range of 65±10kg. The subjects were required to be non-obese and non-oily. All subjects had no fractures, sprains, muscle strains, or other injuries that may affect motor function one month before the collection, no mental illness, no strenuous exercise one day before the collection (i.e., no symptoms such as muscle soreness, discomfort, and weakness), and kept their skin clean during data collection. All subjects were given a unique number. A total of 24 simplified 24-posture Tai Chi moves were collected, with each move collected separately for the upper and lower limbs, and each move collected 7 times, with the signal sampling rate set to 1000Hz.

[0067] Each subject collected data in two stages. In the first stage, all 16 sensors were used to collect the electromyographic signals of the upper limbs during Tai Chi training, and 16 channels of data were obtained. The corresponding situation is shown in Table 1. In the second stage, 14 sensors were used to collect the electromyographic signals of the lower limbs during Tai Chi training, and 14 channels of data were obtained. The corresponding situation is shown in Table 2.

[0068] The overall size of the dataset is: 22 (people) × 24 (moves) × 7 (times) × 30 (channels) = 110,880 data items. Without counting the number of channels, the overall size of the dataset is 22 (people) × 24 (moves) × 7 (times) = 3,696 data items.

[0069] Table 1 Correspondence between muscle and collector numbers in the first collection

[0070] Collector number Muscle Name 1 Left deltoid 2 Right deltoid 3 Left biceps 4 Right biceps 5 Left triceps 6 Right triceps 7 Left extensor digitorum 8 Right extensor digitorum 9 left trapezius 10 Right trapezius 11 Left erector spinae 12 right erector spinae 13 Left tensor fasciae latae 14 Right tensor fasciae latae 15 Left gluteus maximus 16 Right gluteus maximus

[0071] Table 2 Correspondence between muscle and collector numbers in the second collection

[0072] Collector number Muscle Name 1 Left rectus femoris 2 Right rectus femoris 3 Left vastus lateralis 4 Right vastus lateralis 5 Left vastus medialis 6 Right vastus medialis 7 Left biceps femoris 8 Right biceps femoris 9 Left tibialis anterior 10 Right tibialis anterior muscle 11 Left gastrocnemius lateralis 12 Right gastrocnemius lateralis 13 Left gastrocnemius medialis 14 Right gastrocnemius medialis

[0073] 2) Denoising and enhancement of original electromyographic signals

[0074] 2.1 Processing and denoising of raw EMG signals

[0075] With a single channel electromyographic signal S i (i represents the channel number, 1≤i≤30) as an example to illustrate the processing process.

[0076] The collected EMG data was pre-processed by filtering, using a 50Hz notch filter to remove power line interference noise, a 20Hz-450Hz bandpass filter to eliminate motion artifacts (<20Hz) and high-frequency noise (>450Hz), and full-wave rectification. The processed EMG signal data is a time series containing n observations, expressed as:

[0077] S i ={s i1 ,s i2 ,…,si n} (1)

[0078] The processed electromyographic signal data S i Use Python's CV library to convert it into an EMG signal sequence diagram.

[0079] 2.2 Gram angle field conversion of electromyographic signal data

[0080] Based on the processed electromyographic signal data S i To establish the Gram Angle Field (GASF), that is, to establish the timing characteristic diagram, the steps are as follows:

[0081] s ij represents the jth observation value of the i-th channel, S i represents the set of all observations of the ith channel. First, i Normalize so that all values ​​fall within [-1,1] to obtain normalized data in Represents the jth observation value of the i-th channel after normalization, i represents the i-th channel, 1≤i≤30, j represents the j-th time observation value, 1≤j≤n.

[0082]

[0083]

[0084] Then, polar coordinates are used to preserve the absolute time relationship of the sequence, and the value is encoded as the cosine of the angle and the timestamp is encoded as the radius. This encoding method ensures that the result obtained in polar coordinates is unique.

[0085]

[0086]

[0087] φ={φ i1 ,φ i2 ,…,φ in} (6)

[0088] Among them, φ ij and r ijis the angle and radius in polar coordinates obtained by converting the signal data into polar coordinates, t j For time series The timestamp in is the time corresponding to the jth observation value. Since 1≤j≤n, the time series There are n timestamps in total. N is the difference between the last time point and the start time point, that is, N = t n -t1.

[0089] Finally, the Gram angle field GASF is expressed as follows:

[0090] GASF=[cos(φ ip +φ iq )] (7)

[0091] Among them, GASF is an n×n matrix, φ ip Represents the radian value converted by the arc cosine formula from the pth observation value of the time series of the ith channel after normalization, φ iq Represents the qth observation value of the time series of the ith channel after normalization, converted to the radian value using the arc cosine formula, 1≤p≤n, 1≤q≤n.

[0092] The processed GASF matrix data is converted into a time series feature map using the Python CV library.

[0093] 3) Hybrid cascade architecture for feature extraction

[0094] Taking single-channel data as an example, the feature extraction process is introduced. The EMG signal sequence diagram obtained in 2.1 and 2.2 and the time series feature diagram after Gram angle fieldization are input into the hybrid cascade architecture (structure includes CNN and Transformer) for feature extraction. The model structure diagram is shown in Figure 2 .

[0095] The model adopts a dual-stream architecture as the baseline model, which consists of two identical and independent CNN feature extraction backbones, respectively extracting the temporal features of the electromyographic signal sequence graph and the temporal features of the Gram angle field temporal feature graph. Then, the cascade fusion transformer structure proposed in the present invention is used to fuse and interact the dual-stream features.

[0096] 3.1 CNN feature extraction backbone

[0097] The CNN feature extraction backbone of this scheme adopts a dual-stream structure to process the EMG signal sequence graph and the Gram angle field temporal feature graph respectively, and the two CNN branches adopt the same structure.

[0098] The CNN structure has three convolutional layers. After the convolution operation, each convolutional layer is processed using an activation function (GELU activation function) and then a batch normalization (BN) operation is performed.

[0099] Take the branch that processes the electromyographic signal sequence diagram as an example (the other branch operates in the same way):

[0100] First, the EMG signal sequence graph processed in step 2.1 is input into the CNN network.

[0101] Convolution layer: The EMG signal sequence graph is convolved through a 3×3 convolution kernel with a stride of 2.

[0102] GELU activation function: Use GELU (Gaussian Error Linear Unit) activation function to introduce nonlinearity, so that the network can learn and simulate complex function mappings.

[0103] Batch Normalization (BN): Stabilizes the learning process through normalization, speeds up training, and helps reduce internal covariate shift.

[0104] Through the feature extraction backbone network, the time series information in the input EMG signal sequence diagram is converted into a high-level feature vector

[0105] Similarly, another branch is used to extract features from the Gram angle field time series feature map to obtain the feature vector

[0106] 3.2 Modal Fusion Interaction Based on Transformer Network Model

[0107] Based on the Transformer model, this paper proposes a cascade fusion transformer (CFT) structure for feature fusion and interaction in the dual-stream baseline feature extraction process. Figure 3 shown.

[0108] The Cascaded Fusion Transformer consists of two multi-head attention layers and a fully connected (FC) layer.

[0109] Different branches of the two-stream baseline are processed by the CNN feature extraction backbone to obtain high-level features. and (a, b = 1, 2, a ≠ b), as different roles (Q or K, V) are fed to different multi-head attention layers in the cross-attention mechanism, a, b = 1, 2, a ≠ b.

[0110] The self-attention (SA) layer is an important part of the cascade fusion transformer structure. Input the high-level feature vector obtained in step 3.1, i.e., 2 d x dimensional vector, and Here, the high-dimensional features obtained from the electromyographic signal branch are For example, the input vector query Q of each head h is h 、key K h and value V h Computed from the learned linear projection:

[0111]

[0112] in, is the parameter matrix, d h Set to d x / H, where H represents the number of heads, which can be obtained through experiments. The value of this solution is 16. Then the query-key-value vector is mapped to the head h through proportional dot product z On the vector:

[0113]

[0114] in, Represents the self-attention operation.

[0115] Then the outputs of all heads are connected and again passed through the weights The linear transformation is projected to obtain the multi-head self-attention calculation result:

[0116]

[0117] Features of the input double branch and Complementary fusion is performed to obtain two fusion outputs and

[0118]

[0119]

[0120] Then the fusion output of the two cross-attention layers is and Connect along the last dimension and use it as the input of the FC layer to obtain the merged multimodal features. Residual connection and layer normalization (LN) function are used for optimization:

[0121]

[0122] In the formula, a, b = 1, 2, a ≠ b. The features are gradually enriched and propagated across scales by cascading fusion transformers.

[0123] 4) Tai Chi movement recognition

[0124] 4.1 Fully Connected Layer

[0125] The feature vector output by the cascade fusion transformer is mapped to a 24-dimensional space (there are 24 types of Tai Chi movements). The output of the fully connected layer is a 24-dimensional vector, so the output q of the fully connected layer can be expressed as:

[0126]

[0127] Where q is a vector of length C, C is the total number of categories (C = 24 in this solution), W is the weight matrix of the fully connected layer, and b is the bias vector.

[0128] 4.2 Softmax

[0129] The output of the fully connected layer is sent to the Softmax function for classification to obtain the probability values ​​of different action categories for single channel features:

[0130]

[0131] where q i is the i-th element of the fully connected layer output, representing the unnormalized score of the i-th category. i It is the probability value of the i-th category predicted by the single channel feature.

[0132] 4.3 Tai Chi Movement Recognition

[0133] Finally, the probability values ​​corresponding to each single channel feature are added together to obtain the probability value P of the i-th category predicted for all channel features. i , select the category with the highest probability as the predicted output of the model. This can be achieved by finding the index of the maximum value in the probability distribution P. The category with the highest probability value is the Tai Chi action category identified by this model:

[0134] y=argmax 1≤i≤C (P i )

[0135] 4.4 Loss Function

[0136] The solution of the present invention uses cross entropy as the loss function, where Batch is the number of batch samples selected in batch stochastic gradient descent, z The calculated predicted probability distribution for the zth sample, y z is the true probability distribution of the zth sample, expressed as follows:

[0137]

[0138] The network is learned using the back error propagation algorithm. Gradient descent is used to update the parameters, and the iterations are repeated until the loss function converges. The hyperparameters are set to learning rate α = 0.001 and batch_size = 50.

[0139] In summary, in the solution of the present invention, the processed electromyographic signal is converted into a Gram angle field to enrich the time series information. The Gram angle field time series is further used for feature extraction, so as to capture the time series information of the electromyographic signal and improve the stability and accuracy of recognition.

[0140] The present invention proposes a hybrid cascade architecture for extracting features and realizing action recognition, which mainly includes:

[0141] (1) Design a two-stream CNN structure to fully extract high-dimensional features. By leveraging the local receptive field advantage of CNN, the consumption of computing resources can be reduced while maintaining the efficiency of high-dimensional feature extraction.

[0142] (2) A cascaded fusion transformer structure is proposed to accurately and effectively fuse the high-dimensional features obtained from the electromyographic signal graph branch that focuses on spatial information and the Gram angular field timing graph branch that focuses on timing information, thereby obtaining a rich and effective high-dimensional feature representation.

Claims

1. A Tai Chi action recognition method based on a hybrid cascade architecture, characterized in that: The following steps are involved: S1, collecting data sets, specifically, collecting electromyographic signals generated during Tai Chi exercise through electromyographic sensors, taking the data collected by each sensor as a channel data, taking all channel data obtained at each observation time as the observation data obtained at the current time, and collecting data at the set time from the start time to the end time of the Tai Chi exercise, so as to obtain a data set of a whole set of Tai Chi moves; S2, processing the data set, including denoising the electromyographic signal of each channel to obtain electromyographic signal data, and then obtaining the electromyographic signal sequence diagram of each channel according to the acquisition time series of each channel data; The denoised electromyographic signal data is used to obtain the time series characteristic graph after Gram angle fieldization; S3, build a Tai Chi action recognition model based on a hybrid cascade architecture, including a CNN feature extraction module, a cascade fusion transformer, a fully connected layer, a Softmax layer, and a Tai Chi action recognition module; The CNN feature extraction module includes two branches with the same structure, which respectively extract features from the electromyographic signal sequence diagram and the time series feature diagram; The cascade fusion transformer consists of two multi-head attention layers and one fully connected layer. The two features output by the CNN feature extraction module are fed to different multi-head attention layers as different roles in the cross attention mechanism. The attention calculation result of each feature is obtained through the multi-head attention mechanism, and then complementary fusion is performed to obtain two fusion outputs respectively. Finally, the two fusion outputs are connected along the last dimension and input into the fully connected layer, and the fully connected layer outputs multimodal features. The obtained multimodal features are sequentially passed through the fully connected layer and the Softmax layer to obtain the probability values ​​of different action categories for the single-channel features. Finally, the probability values ​​corresponding to each single-channel feature are added in the Tai Chi action recognition module to obtain the probability value of each category predicted by the full-channel features, and the category with the highest probability value is the identified Tai Chi action category. S4, using the obtained data set to train the constructed Tai Chi action recognition model, using cross entropy as the loss function in the training, and obtaining a trained Tai Chi action recognition model; S5. Input the acquired target action into the trained Tai Chi action recognition model for recognition.

2. The Tai Chi action recognition method of a hybrid cascade architecture according to claim 1, characterized in that: When collecting data in S1, a total of 30 channels of data are collected, corresponding to 30 muscles of the upper and lower limbs. At the same time, it is set that there are 24 Tai Chi moves, and the upper and lower limbs are collected for each move, 7 times for each move, and the signal sampling rate is set to 1000Hz. The collected electromyographic signal is represented as S i ,1≤i≤30, there are n observation moments in total.

3. The Tai Chi action recognition method of a hybrid cascade architecture according to claim 2, characterized in that: The specific method of S2 is: For S i Filtering is performed to remove power line interference noise through a 50Hz notch filter, and motion artifacts and high-frequency noise are eliminated through a 20Hz-450Hz bandpass filter. Full-wave rectification is then performed to obtain a time series of n observations: S i ={s i1 ,s i2 ,…,s in } Finally, use the python CV library to convert it into an EMG signal sequence diagram; Based on the processed electromyographic signal data S i Establishing the Gram angle field, that is, establishing the time series characteristic diagram: Definitions ij represents the jth observation value of the i-th channel, i represents the i-th channel, j represents the j-th time observation value, 1≤j≤n, for S i Normalize so that all values ​​fall within [-1,1] to obtain normalized data definition Represents the jth observation value of the i-th channel after normalization: Polar coordinates are used to preserve the absolute time relationship of the sequence, encoding the value as the cosine of the angle and the timestamp as the radius: φ={φ i1 ,f i2 ,…,f in } Among them, φ ij and r ij is the angle and radius in polar coordinates obtained by converting the signal data into polar coordinates, t j For time series The timestamp in is the time corresponding to the jth observation value. Since 1≤j≤n, the time series There are n timestamps in total, N is the difference between the last time point and the start time point, that is, N = t n -t1; Finally, the Gram angle field GASF is expressed as follows: GASF=[cos(φ ip +φ iq )] Among them, GASF is an n×n matrix, φ ip Represents the radian value converted by the arc cosine formula from the pth observation value of the time series of the ith channel after normalization, φ iq Represents the radian value converted from the qth observation value of the time series of the ith channel after normalization by the arc cosine formula, 1≤p≤n, 1≤q≤n; The processed GASF matrix data is converted into a time series feature map using the Python CV library.

4. The Tai Chi action recognition method of a hybrid cascade architecture according to claim 3 is characterized in that: The two branches in the CNN feature extraction module are composed of three convolutional layers. After the convolution operation, each convolutional layer is processed using an activation function and then a batch normalization operation is performed. The two branches are defined to output features respectively. and a,b=1,2,a≠b.

5. The Tai Chi action recognition method of a hybrid cascade architecture according to claim 4, characterized in that: The specific method of the cascade fusion transformer for data fusion is: For features is d x dimensional vector, the input vector query Q of each head h h 、key K h and value V h Computed from the learned linear projection: in, is the parameter matrix, d h Set to d x / H, H represents the number of multiple heads; the query-key-value vector is mapped to the head h through proportional dot product z On the vector: in, represents the self-attention operation; The outputs of all heads are connected and connected by weights The linear transformation of is projected to obtain the multi-head self-attention calculation result: The same reason Processing, and finally the two input features and Complementary fusion is performed to obtain two fusion outputs and The fusion output of the two cross-attention layers and Connect along the last dimension to obtain the merged multimodal features Optimization is performed using residual connections and layer normalization functions:

6. The Tai Chi action recognition method of a hybrid cascade architecture according to claim 5, characterized in that: The output after inputting into the fully connected layer is: Among them, q is a vector of length C, C is the total number of categories, corresponding to the moves of Tai Chi, W is the weight matrix of the fully connected layer, and b is the bias vector; The Softmax layer obtains the probability values ​​of different action categories for single-channel features as follows: Among them, q i is the i-th element of the fully connected layer output, representing the unnormalized score of the i-th category, p i It is the probability value of the i-th category predicted by the single channel feature; The Tai Chi action recognition module adds the probability values ​​corresponding to each single channel feature to obtain the probability value P of the i-th category predicted by the full channel feature. i , select the category with the highest probability as the prediction output of the model, which is achieved by finding the index of the maximum value in the probability distribution P. The category with the highest probability value is the identified Tai Chi action category: y=argmax 1≤i≤C (P i )。

Images