A deep learning-based upper limb action recognition method and system

Abstract

The application provides a kind of upper limb action recognition method and system based on deep learning, it is related to upper limb action recognition technical field, method includes: according to the sequence data of different upper limb action of multiple sensors acquisition construction, the frequency spectrum is converted according to the preprocessed data to the preprocessed sequence data, and the preprocessed data includes the preprocessed time domain sequence;The time sequence feature vector and the frequency spectrum feature vector corresponding to the preprocessed time domain sequence and the frequency spectrum are respectively extracted using deep learning network to output;According to attention mechanism, combine sensor real-time reliability evaluation value, the time sequence feature vector and the frequency spectrum feature vector of multi-sensor are fused to obtain dynamic weighted fusion feature, and the target model is obtained by training the fusion feature through the preset model, to identify upper limb action according to target model.The upper limb action recognition of target model obtained by the application, feature extraction is accurate, greatly improves the similar action recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of upper limb movement recognition technology, and in particular to an upper limb movement recognition method and system based on deep learning. Background Technology

[0002] With the development of human-computer interaction technology and rehabilitation medicine, utilizing upper limb movements to achieve human-computer interaction has become one of the current research directions, and the demand is becoming increasingly urgent. Most methods rely on a single sensor to collect data, which can only reflect local upper limb movement information and cannot cover the coordinated movement characteristics of multiple parts such as the upper arm and forearm, resulting in low accuracy of movement recognition.

[0003] Action recognition involves collecting activity data through sensors, classifying different actions by labels, and then inferring human behavior and predicting human intentions. However, the collected action data is mostly stored in the form of continuous time series, and how to efficiently extract feature signals from it remains a pressing problem to be solved in this field.

[0004] Traditional methods only use time-series data to extract time-series features, or only use frequency-domain data to extract frequency features, without fully integrating complementary information from the time and frequency domains, making it difficult to capture the dynamic time-frequency variation patterns of actions.

[0005] Current multi-sensor fusion methods mostly remain at the data-level, lacking feature-level deep fusion mechanisms. Furthermore, they fail to consider sensor reliability differences during fusion decision-making, making the fusion results prone to deviations when some sensors are affected by noise. How to efficiently extract feature signals from data remains a pressing problem to be solved in this field.

[0006] To address the aforementioned issues, this invention proposes an upper limb movement recognition method that integrates multi-sensor time-frequency domain feature collaborative extraction, feature-level deep fusion, and reliability-weighted decision fusion, thereby improving the recognition performance of upper limb movements. Summary of the Invention

[0007] Therefore, the purpose of this invention is to provide a deep learning-based upper limb movement recognition method and system to solve the technical problems of difficulty in feature extraction and low accuracy of similar movement recognition in the prior art.

[0008] This invention provides a deep learning-based upper limb movement recognition method, comprising:

[0009] Sequence data is constructed by collecting different upper limb movements from multiple sensors. The sequence data is preprocessed to obtain a spectrogram based on the preprocessed data. The preprocessed data includes a preprocessed time-domain sequence.

[0010] Deep learning networks are used to extract features from the preprocessed time-domain sequence and spectrogram to output corresponding time-series feature vectors and spectrogram feature vectors. Both the time-series feature vectors and the spectrogram feature vectors are structured feature vectors that can be trained and fused by the model.

[0011] Based on the attention mechanism and the real-time reliability evaluation value of the sensor, the temporal feature vector and spectral feature vector of multiple sensors are fused to obtain the dynamically weighted fused feature. The fused feature is then trained by a preset model to obtain the target model, which is used to identify upper limb movements.

[0012] The aforementioned deep learning-based upper limb action recognition method utilizes a temporal convolutional network to obtain local feature relationships in the preprocessed temporal sequence, and a bidirectional gated recurrent unit captures long-term dependencies in the sequence from bidirectional time steps. Secondly, based on an attention mechanism combined with real-time reliability assessment values ​​from sensors, it fuses temporal feature vectors and spectral feature vectors from multiple sensors to obtain dynamically weighted fused features. Finally, a Transformer feature fusion model is introduced to calculate the correlation degree at different locations, and the fused features are used to train the model to obtain the target model. Upper limb action recognition is performed using the obtained target model, resulting in accurate feature extraction and significantly improved accuracy in similar action recognition. This solves the technical problems of difficult feature extraction and low accuracy in similar action recognition in existing technologies for upper limb action recognition.

[0013] In addition, the deep learning-based upper limb movement recognition method according to the present invention may also have the following additional technical features:

[0014] Furthermore, in the step of using a deep learning network to extract features from the preprocessed time-domain sequence and spectrogram to output the corresponding time-series feature vector and spectrogram feature vector, the method for extracting the time-series feature vector includes:

[0015] Preprocessed three-dimensional acceleration data is obtained and input into a temporal feature extraction model composed of a deep learning network that integrates a temporal convolutional network and a bidirectional gated recurrent unit to achieve automatic extraction of temporal feature vectors.

[0016] Among them, the temporal convolutional network takes the residual block as its core and obtains the local feature relationship of the preprocessed temporal sequence through causal convolution in the residual block;

[0017] The bidirectional gated loop unit extracts the temporal sequence features of the data from both the forward and reverse directions of the preprocessed time-domain sequence, and eliminates the phase offset of the bidirectional extraction through the time-series alignment module, which is used to extract long-term dependencies in the time-domain sequence.

[0018] The temporal feature extraction model includes an input adaptation layer, a temporal convolutional network layer, a dimension transformation layer, a bidirectional gated recurrent unit layer, an adaptive average pooling layer, and a fully connected layer.

[0019] The input adaptation layer is used to adjust the preprocessed temporal sequence to fit the input of the temporal convolutional network layer.

[0020] The temporal convolutional network layer contains two residual blocks. Each residual block is configured with two one-dimensional convolutional layers with kernel sizes of 3 and 5 in parallel. The output of each residual block is fused with the output of the residual branch and then activated by the ReLU activation function to output 32 channels of local temporal features.

[0021] The dimension transformation layer is used to adjust the order of feature dimensions to match the input requirements of the bidirectional gated recurrent unit layer;

[0022] The bidirectional gated loop unit layer has two layers, which cycle bidirectionally in both the forward and reverse directions of the timing sequence. The number of output channels in the hidden layer is 64 and 32, respectively.

[0023] The adaptive average pooling layer is a pooling layer with a kernel size of 1. After feature flattening, the input dimension is 32 and the output is a 64-dimensional temporal feature vector.

[0024] Furthermore, in the step of using a deep learning network to extract features from the preprocessed time-domain sequence and spectrogram to output the corresponding time-series feature vector and spectrogram feature vector, the method for extracting the spectrogram feature vector includes:

[0025] The amplitude of the spectrogram is numerically normalized, and the spectrograms of different data segments are unified to a fixed size through bilinear interpolation. Then, the spectrograms of multiple sensors are arranged along the channel dimension to form a multi-channel spectrogram tensor. In the bilinear interpolation process, a multi-sensor channel alignment strategy is introduced to ensure the spatial consistency of the spectrogram data of different parts of the sensor.

[0026] The local time-frequency texture of the spectrogram is extracted by a two-dimensional convolutional neural network. The feature map after dimensionality compression and feature selection is flattened in pixel order and input into a two-dimensional spatial long short-term memory network to model the global dependency relationship between time and frequency dimensions.

[0027] Principal component analysis is performed on the multidimensional spectral feature vector output by the two-dimensional spatial long short-term memory network to remove redundant features and select key features, including the proportion of high-frequency components and time-frequency entropy, and output the corresponding spectral feature vector. The principal component analysis process includes: combining a time-frequency entropy weighted screening mechanism and dynamically adjusting a preset variance threshold based on prior knowledge of upper limb movement categories to retain principal components whose cumulative variance meets the variance threshold.

[0028] Furthermore, based on the attention mechanism and combined with the real-time reliability evaluation values ​​of the sensors, the steps of fusing the temporal feature vectors and spectral feature vectors of multiple sensors to obtain dynamically weighted fused features, and training the fused features using a preset model to obtain the target model include:

[0029] The time-series feature vector and the spectral feature vector are respectively input into the fully connected layer to generate a time-domain view descriptor and a frequency-domain view descriptor with a unified dimension; layer normalization is performed on the time-domain view descriptor and the frequency-domain view descriptor to eliminate the numerical scale differences of different view features;

[0030] The time-domain view descriptor and frequency-domain view descriptor, after being normalized by the layer, are concatenated and input into a fully connected layer for feature interaction. Then, the dynamic attention score of the time-domain view descriptor and frequency-domain view descriptor is calculated by the Softmax attention calculation layer of the attention mechanism fusion module, combined with the real-time monitoring value of sensor noise. Based on the attention score, the time-series features and frequency-domain features are weighted and summed to generate multi-dimensional single-sensor fusion features. This fusion feature is input into the Transformer feature fusion model to concatenate the fusion features of multiple sensors. Through its multi-head self-attention module, the correlation features between sensors and between time and frequency domains are captured and global feature fusion training is performed to obtain the target model. Here, the time-series features are the feature component values ​​of each dimension of the time-series feature vector, and the frequency-domain features are the feature component values ​​of each dimension of the spectral feature vector.

[0031] Furthermore, the step of constructing sequence data based on different upper limb movements collected by multiple sensors, and preprocessing the sequence data to obtain a spectrogram from the preprocessed data includes:

[0032] The three-dimensional acceleration data of different upper limb movements are acquired by the sensor and preprocessed. The preprocessed three-dimensional acceleration data is then converted into a two-dimensional spectrum using short-time Fourier transform.

[0033] The data preprocessing methods include:

[0034] The magnitude of the three-dimensional acceleration of each sensor is calculated to convert the three-dimensional acceleration data into one-dimensional data, thereby creating a one-dimensional time series view to reduce the complexity of the sequence data;

[0035] One-dimensional data is input into a multi-order Butterworth low-pass filter for denoising. The multi-order Butterworth low-pass filter adopts an adaptive cutoff frequency adjustment mechanism and dynamically matches the filtering parameters according to the action type, so as to retain the subtle features of the action while denoising.

[0036] Perform data standardization on the denoised one-dimensional data.

[0037] The standardized data is segmented into continuous data segments by using an adaptive window size adjustment mechanism and a sliding window technique that dynamically adjusts the sliding window based on the upper limb movement rate.

[0038] Furthermore, the steps of converting the preprocessed three-dimensional acceleration data into a two-dimensional spectrogram using short-time Fourier transform include:

[0039] Each data segment is divided into consecutive overlapping frames based on the adaptive overlap rate of the action cycle, and a Hanning window with a window width dynamically adjusted according to the frequency distribution of the signal within the frame is applied to each frame to suppress spectral leakage caused by frame truncation.

[0040] Perform a short-time Fourier transform on each windowed signal frame, retain positive frequency components, extract the single-sided spectrum, double the amplitude compensation for non-DC and quister frequency points, and eliminate conjugate symmetric amplitude distortion.

[0041] The frequency and amplitude information of each frame are arranged in chronological order to form a time-frequency matrix. The time-frequency matrix includes time, frequency and amplitude, and the matrix element values ​​correspond to the signal amplitude at a specific time and frequency point.

[0042] A two-dimensional spectrogram is generated by mapping the image based on the time-frequency matrix.

[0043] Another aspect of the present invention provides a deep learning-based upper limb movement recognition system, comprising:

[0044] The data acquisition module is used to construct sequence data based on different upper limb movements collected by multiple sensors, and to preprocess the sequence data to obtain a spectrogram based on the preprocessed data. The preprocessed data includes a preprocessed time-domain sequence.

[0045] The feature extraction module is used to extract features from the preprocessed time-domain sequence and spectrogram using a deep learning network to output corresponding time-series feature vectors and spectrogram feature vectors. The time-series feature vectors and the spectrogram feature vectors are both structured feature vectors that can be trained and fused by the model.

[0046] The model training module is used to combine the attention mechanism with the real-time reliability evaluation value of the sensor, fuse the temporal feature vectors and spectral feature vectors of multiple sensors to obtain dynamically weighted fusion features, and train the fusion features through a preset model to obtain the target model, so as to recognize upper limb movements based on the target model.

[0047] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based upper limb movement recognition method described above.

[0048] In another aspect, the present invention provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deep learning-based upper limb movement recognition method described above. Attached Figure Description

[0049] Figure 1 This is an architecture diagram of the overall structure of the model proposed in this invention;

[0050] Figure 2 This is a flowchart of the deep learning-based upper limb movement recognition method in the first embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the improved TCN network structure of the present invention;

[0052] Figure 4 This is a schematic diagram of the TCN-BiGRU model structure and corresponding parameters of the present invention;

[0053] Figure 5 This is a schematic diagram of the 2D CNN-LSTM model structure and corresponding parameters of the present invention;

[0054] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0055] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0057] To address the challenges of feature extraction and low accuracy in recognizing similar upper limb movements, this invention proposes a method integrating multi-sensor collaborative data acquisition, dual-path time-frequency feature adaptive extraction, attention-driven weighted fusion, and Transformer global modeling. This method achieves deep fusion of local features, long-term temporal features, and multi-dimensional spectral features, thereby improving the accuracy of recognizing similar upper limb movements. Specifically: First, a Temporal Convolutional Network (TCN) is used to obtain local feature relationships in the preprocessed time-domain sequence, and a Bidirectional Gated Recurrent Unit (BiGRU) captures long-term dependencies in the sequence from bidirectional time steps. Second, based on the attention mechanism and combined with real-time reliability assessment values ​​of the sensors, the temporal feature vectors and spectral feature vectors of multiple sensors are fused to obtain dynamically weighted fused features. Finally, a Transformer feature fusion model is introduced to calculate the correlation degree at different locations, and the fused features are used to train the model to obtain the target model. Using the obtained target model for upper limb movement recognition results in accurate feature extraction and significantly improved accuracy in recognizing similar movements.

[0058] Specifically, such as Figure 1 The diagram shows the overall architecture of the proposed model. TCN and BiGRU are used to capture temporal features, while a 2D Convolutional Neural Network (2D CNN) and a 2D Long Short-Term Memory (2D LSTM) are used to extract frequency domain features. An attention mechanism is introduced to weightedly fuse the two types of features. Finally, the fused features are trained using a Transformer feature fusion model, and the fully connected layer outputs the recognition accuracy for six typical upper limb movements. The proposed model was evaluated on a self-made dataset, and human-computer interaction experiments involving upper limbs and robotic arms were conducted. The proposed algorithm achieved an upper limb movement recognition accuracy of 99.64%, which is more than 4% higher than the CNN-BiGRU and CNN-Attention algorithms, meeting the requirements of human-computer interaction applications.

[0059] To facilitate understanding of the present invention, several embodiments are given below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0060] Example 1

[0061] Please see Figure 2The figure shows a deep learning-based upper limb movement recognition method according to the first embodiment of the present invention, the method including steps S101 to S103:

[0062] S101. Construct sequence data based on different upper limb movements collected by multiple sensors, and preprocess the sequence data to obtain a spectrum diagram based on the preprocessed data.

[0063] As a specific example, in the upper limb motion information acquisition stage, three inertial measurement unit sensors were used. The sensors were fixed to the wrist, forearm, and upper arm of the human upper limb, respectively. A fixed sampling rate of 50Hz was set to collect motion data of six typical upper limb movements. Each movement was repeated 12 times by 6 healthy volunteers, with each movement lasting 2 to 5 seconds, recording the complete motion process of multiple types of movements. Specifically, the six typical upper limb movements included: elbow horizontal flexion, elbow horizontal extension, shoulder horizontal adduction, shoulder horizontal abduction, arm vertical elevation, and arm vertical lowering.

[0064] In this embodiment, the preprocessed data includes a preprocessed time-domain sequence. Further, based on the three-dimensional acceleration data of different upper limb movements acquired by the sensor and preprocessed, the preprocessed three-dimensional acceleration data is converted into a two-dimensional spectrogram using a short-time Fourier transform.

[0065] The data preprocessing method includes: First, calculating the magnitude of the three-dimensional acceleration data for each sensor to convert the three-dimensional acceleration data into one-dimensional data, thereby creating a one-dimensional time series view to reduce the complexity of the sequence data; Second, inputting the one-dimensional data into a multi-order Butterworth low-pass filter for denoising, wherein the multi-order Butterworth low-pass filter adopts an adaptive cutoff frequency adjustment mechanism, dynamically matching the filter parameters according to the action type, preserving subtle action features while denoising; Specifically, the multi-order Butterworth low-pass filter is set to a cutoff frequency of 10Hz to remove environmental interference and inherent noise of the equipment in the original signal; Third, performing data standardization on the denoised one-dimensional data; Specifically, calculating the mean and standard deviation of the denoised one-dimensional data to limit the data to a specific range, reducing the overall data amplitude while maintaining the original characteristics. Furthermore, an adaptive window size adjustment mechanism is used, and a sliding window technique that dynamically adjusts according to the upper limb movement rate is used to segment the standardized data into continuous data segments. Specifically, a fixed window size of 512 sampling frames is used, and a 50% segmentation overlap rate is set to segment the standardized data into continuous data segments.

[0066] Furthermore, the step of converting the preprocessed three-dimensional acceleration data into a two-dimensional spectrogram using short-time Fourier transform includes: First, dividing each data segment into consecutive overlapping frames based on an adaptive overlap rate according to the action period, and applying a Hanning window with a window width dynamically adjusted according to the signal frequency distribution within the frame to suppress spectral leakage caused by frame truncation. In this embodiment, the preset overlap rate is 50%, meaning that each data segment is divided into consecutive overlapping frames at 50%. Second, performing a short-time Fourier transform on the windowed signal of each frame, retaining positive frequency components, extracting the single-sided spectrum, and doubling the amplitude compensation for non-DC and quiescent frequency points to eliminate conjugate symmetric amplitude distortion. Then, arranging the frequency and amplitude information of each frame in chronological order to form a time-frequency matrix, wherein the time-frequency matrix includes time, frequency, and amplitude, and the matrix element values ​​correspond to the signal amplitude at specific time and frequency points. Furthermore, a two-dimensional spectrum is generated based on the time-frequency matrix mapping image; specifically, based on the time-frequency matrix mapping image, the horizontal axis represents time, the vertical axis represents frequency, and the color intensity represents the amplitude of the corresponding time and frequency points, thereby generating a two-dimensional spectrum.

[0067] S102. Use a deep learning network to extract features from the preprocessed time-domain sequence and spectrogram to output the corresponding time-series feature vector and spectrogram feature vector.

[0068] In this embodiment, both the temporal feature vector and the spectral feature vector are structured feature vectors that can be trained and fused by the model. As a specific example, the method for extracting the temporal feature vector includes: acquiring preprocessed three-dimensional acceleration data, and inputting it into a temporal feature extraction model composed of a deep learning network fused with a temporal convolutional network and bidirectional gated recurrent units to achieve automatic extraction of the temporal feature vector. The TCN network structure is as follows: Figure 3 As shown, where, y T This is the output data corresponding to the T-th time step. x 0、 x 1, ... x T-1 , x T This represents the input data at different time steps. TCN uses residual blocks as its core, acquiring local feature relationships of the preprocessed temporal sequence through causal convolutions within the residual blocks. Specifically, causal convolutions expand their receptive field by dilating the convolution kernel, dynamically adjusting the dilation coefficient based on the local feature density of the temporal sequence. This enhances the ability to differentiate and capture local dynamic features of different actions. Specifically, the dilation coefficient is dynamically allocated based on the local feature variance of the current data segment. A larger variance indicates more drastic action changes, requiring a smaller dilation coefficient to ensure no loss of local detail features; a smaller variance indicates smoother actions, requiring a larger dilation coefficient to expand the receptive field and capture long-term dependencies. Residual blocks are used to accelerate the convergence speed of the TCN network between each layer.

[0069] Specifically, such as Figure 4 As shown, the temporal feature extraction model includes an input adaptation layer, a temporal convolutional network layer, a dimension transformation layer, a bidirectional gated recurrent unit layer, an adaptive average pooling layer, and a fully connected layer. The input adaptation layer adjusts the preprocessed temporal sequence to fit the input of the temporal convolutional network layer. The temporal convolutional network layer contains two residual blocks, residual block 1 and residual block 2. Each residual block is configured with two one-dimensional convolutional layers with kernel sizes of 3 and 5 in parallel. Their outputs are fused with the outputs of the residual branches and then activated by the ReLU activation function to output 32-channel local temporal features. The dimension transformation layer adjusts the feature dimension order to match the input requirements of the bidirectional gated recurrent unit layer. The bidirectional gated recurrent unit layer has two layers, cycling bidirectionally along the temporal sequence, with hidden layer output channels of 64 and 32 respectively. The adaptive average pooling layer is a pooling layer with a kernel size of 1. After feature flattening, the input dimension is 32, and the output is a 64-dimensional temporal feature vector.

[0070] Furthermore, BiGRU extracts the temporal sequence features of the data from both the forward and reverse directions of the time-domain sequence, and eliminates the phase offset of the bidirectional extraction through the time-series alignment module, which is used to extract long-term dependencies in the time-domain sequence.

[0071] In this embodiment, the method for extracting the spectral feature vector includes:

[0072] The amplitude of the spectrogram is numerically normalized, and bilinear interpolation is used to unify the spectrograms of different data segments to a fixed size. During bilinear interpolation, a multi-sensor channel alignment strategy is introduced to ensure spatial consistency of the spectrogram data from different sensor locations. The spectrograms from multiple sensors are then arranged along the channel dimension to form a multi-channel spectrogram tensor. A 2D Convolutional Neural Network (2D CNN) is used to extract local time-frequency texture from the spectrogram. The feature map, after dimensionality compression and feature selection, is flattened pixel-wise and input into a 2D Long Short-Term Memory (2D LSTM) network to model the global dependencies between time and frequency dimensions. Principal component analysis (PCA) is performed on the multi-dimensional spectral feature vectors output by the 2D LSTM to remove redundant features and select key features such as the proportion of high-frequency components and time-frequency entropy, outputting the corresponding spectral feature vectors. The PCA process includes: combining a time-frequency entropy weighted selection mechanism with prior knowledge of upper limb movement categories to dynamically adjust a preset variance threshold, retaining principal components whose cumulative variance meets the threshold.

[0073] Furthermore, the spectral feature vector is extracted through a spectral graph feature extraction model. As a specific example, a deep learning network that integrates a two-dimensional convolutional neural network and a two-dimensional spatial long short-term memory network is used to form a spectral graph feature extraction model to achieve automatic extraction of spectral feature vectors.

[0074] Specifically, such as Figure 5 As shown, the spectrogram feature extraction model includes an input adaptation layer, a two-dimensional convolutional neural network layer, a dimension transformation layer, a two-dimensional spatial long short-term memory network layer, an adaptive average pooling layer, a feature selection layer, and a principal component analysis layer. Among them, the input adaptation layer matches the construction of the multi-channel spectrogram tensor after bilinear interpolation, the two-dimensional convolutional neural network layer extracts the local time-frequency texture features of the spectrogram, the feature selection layer performs a time-frequency entropy weighted selection mechanism, and the principal component analysis layer realizes principal component analysis with dynamic adjustment of variance threshold.

[0075] Specifically: The input adaptation layer adjusts the format of the multi-channel spectrogram tensor to adapt to the input of the 2D convolutional neural network layer; the 2D convolutional neural network layer contains 3 convolutional blocks, each block is configured with 2 2D convolutional layers with a kernel size of 3 in parallel, and the number of output channels of each block are 16, 32 and 64 respectively. After convolution, it is activated by the ReLU activation function and reduced in dimensionality by the max pooling layer; the dimension transformation layer adjusts the order of feature dimensions to match the input requirements of the 2D spatial long short-term memory network layer; the 2D spatial long short-term memory network layer has 2 layers, in the time and frequency two-dimensional space, The model establishes a global dependency between two dimensions, with hidden layer output channels of 128 and 64 respectively. The adaptive average pooling layer is a pooling layer with a kernel size of 1, which downsamples and compresses the dimensions of the sequence features. After flattening, the features are obtained as a one-dimensional feature vector. The feature selection layer combines a time-frequency entropy weighted selection mechanism to select key features and outputs a 128-dimensional selected feature vector. The key features include the proportion of high-frequency components and time-frequency entropy. The principal component analysis layer combines prior knowledge of upper limb movements to dynamically adjust the variance threshold, retain the core principal components, and outputs a 64-dimensional spectral feature vector.

[0076] S103. Based on the attention mechanism and the real-time reliability evaluation value of the sensor, the temporal feature vector and spectral feature vector of multiple sensors are fused to obtain the dynamically weighted fusion feature. The fusion feature is then trained using a preset model to obtain the target model, which is used to identify upper limb movements.

[0077] In this embodiment, an attention mechanism is introduced for weighted operation, fusing temporal and spectral features from multiple sensors to generate unified latent spatiotemporal features. Specifically, this can be expressed as follows: the temporal feature vector output by the TCN-BiGRU network and the spectral feature vector output by the 2D CNN-LSTM network are respectively input into a fully connected layer to generate temporal view descriptors and frequency domain view descriptors with unified dimensions; layer normalization is performed on the temporal view descriptors and frequency domain view descriptors to eliminate the numerical scale differences of different view features. The time-domain view descriptor and frequency-domain view descriptor, after being normalized by the layer, are concatenated and input into a fully connected layer for feature interaction. Then, through the Softmax attention calculation layer of the attention mechanism fusion module, the dynamic attention scores of the time-domain view descriptor and frequency-domain view descriptor are calculated in combination with the real-time monitoring values ​​of sensor noise to dynamically reflect the importance of the two types of features to action classification. Among them, the sensor reliability assessment value is introduced when calculating the attention score. The reliability assessment value is based on the signal-to-noise ratio (SNR) of the time-domain view descriptor and the frequency-domain view descriptor as the core quantification basis. The feature weights are dynamically adjusted according to the SNR of the time-domain view descriptor and the frequency-domain view descriptor to reduce the impact of noise interference on the sensor on the fusion result. Specifically, the sensor reliability assessment value is determined by calculating the signal-to-noise ratio (SNR) between the time-domain view descriptor and the frequency-domain view descriptor. The reliability assessment value is set to 1 when SNR ≥ 30dB, 0.3 when SNR < 20dB, and linear interpolation is used for values ​​in between. This factor is multiplied by the attention score and then weighted and summed to achieve reliability-driven feature fusion. Based on the attention score, the temporal and frequency-domain features are weighted and summed to generate multi-dimensional single-sensor fusion features. These fusion features are input into the Transformer feature fusion model to stitch together the fusion features from multiple sensors. Through its multi-head self-attention module, it captures cross-sensor and cross-time and frequency-domain correlation features and performs global feature fusion training to obtain the target model, thereby calibrating the feature space bias caused by sensors being worn on different parts of the upper limb. In the step of weighted summation of temporal features and frequency domain features based on attention scores, the temporal features and frequency domain features are the feature component values ​​of each dimension of the temporal feature vector and the spectral feature vector mentioned above, respectively. The weighted summation is a dimension-wise numerical summation operation of the feature vector, and the weights are dynamically adjusted based on the sensor reliability assessment value.

[0078] The attention mechanism fusion module includes a feature unification layer, a Softmax attention calculation layer, and a weighted fusion layer. Specifically, the feature unification layer inputs the temporal feature vector and the spectral feature vector into a fully connected layer with 256 hidden neurons, respectively. After activation by the ReLU activation function, it generates a 128-dimensional unified temporal view descriptor and a frequency domain view descriptor. The Softmax attention calculation layer calculates the dynamic attention score of the temporal view descriptor and the frequency domain view descriptor based on the sensor reliability assessment value. The weighted fusion layer is used to weight and sum the two types of features according to the attention score, and outputs a 64-dimensional fused feature. Furthermore, the Transformer feature fusion model includes a feature stitching layer, a multi-head self-attention module, and a feedforward network layer. In this embodiment, the feature stitching layer is used to stitch the fused features of the three sensors into a 384-dimensional feature vector. The multi-head self-attention module has an 8-head attention mechanism, where the key dimension, query dimension, and value dimension are all 64, used to capture the correlation features across sensors and across the time and frequency domains. The feedforward network layer has a dimension of 256, and outputs a 128-dimensional feature vector after activation by the ReLU activation function.

[0079] Specifically, the Transformer feature fusion model employs a two-layer Transformer encoder in a cascaded structure. Each layer contains a multi-head self-attention module and a feedforward network. The multi-head self-attention module calculates the correlation between feature positions through dot products, capturing the complementarity between features from multiple sensors. Secondly, in the global feature fusion training process, the fused features from the three sensors are randomly divided into training and test sets in a 7:3 ratio. The Adam optimizer is used, with the learning rate automatically selected by Bayesian optimization. Classification cross-entropy is chosen as the loss function, and ReLU as the activation function. The training epochs are set to 50, and the batch size to 32. Thirdly, contrastive loss is introduced during training to improve the discriminative power of the fused features by minimizing the distance between similar action features and maximizing the distance between dissimilar action features. Finally, the model training results are evaluated using accuracy, precision, recall, F1 score, network parameters, and training time as performance metrics to assess the upper limb action recognition performance after fusing temporal and frequency domain features, thus obtaining the target model. The target model obtained by the algorithm provided in this application is used for upper limb action recognition, which further improves the accuracy of upper limb action recognition and the recognition effect meets the needs of human-computer interaction applications.

[0080] To verify the recognition performance of the target model obtained through the technical solution of this application, the proposed target model was evaluated on a self-made dataset, and human-computer interaction experiments involving upper limbs and robotic arms were conducted. The proposed algorithm achieved an upper limb movement recognition accuracy of 99.64%, which is more than 4% higher than that of CNN-BiGRU and CNN-Attention algorithms, and the recognition performance meets the requirements of human-computer interaction applications.

[0081] In summary, the deep learning-based upper limb action recognition method in the above embodiments of the present invention obtains local feature relationships in the preprocessed temporal sequence using a temporal convolutional network, and captures long-term dependencies in the sequence from bidirectional time steps using a bidirectional gated recurrent unit. Secondly, based on the attention mechanism and combined with the real-time reliability evaluation value of the sensors, the temporal feature vectors and spectral feature vectors of multiple sensors are fused to obtain dynamically weighted fused features. Finally, a Transformer feature fusion model is introduced to calculate the correlation degree of different positions, and the fused features are used to train the model to obtain the target model. Upper limb action recognition is performed using the obtained target model, which results in accurate feature extraction and greatly improves the accuracy of similar action recognition, solving the technical problems of difficult feature extraction and low accuracy of similar action recognition in the prior art for upper limb action recognition.

[0082] Example 2

[0083] The deep learning-based upper limb motion recognition system in the second embodiment of the present invention includes:

[0084] The data acquisition module is used to construct sequence data based on different upper limb movements collected by multiple sensors, and to preprocess the sequence data to obtain a spectrogram based on the preprocessed data. The preprocessed data includes a preprocessed time-domain sequence.

[0085] The feature extraction module is used to extract features from the preprocessed time-domain sequence and spectrogram using a deep learning network to output corresponding time-series feature vectors and spectrogram feature vectors. The time-series feature vectors and the spectrogram feature vectors are both structured feature vectors that can be trained and fused by the model.

[0086] The model training module is used to combine the attention mechanism with the real-time reliability evaluation value of the sensor, fuse the temporal feature vectors and spectral feature vectors of multiple sensors to obtain dynamically weighted fusion features, and train the fusion features through a preset model to obtain the target model, so as to recognize upper limb movements based on the target model.

[0087] In summary, the deep learning-based upper limb action recognition system in the above embodiments of the present invention obtains local feature relationships in the preprocessed temporal sequence using a temporal convolutional network, and captures long-term dependencies in the sequence from bidirectional time steps using a bidirectional gated recurrent unit. Secondly, based on the attention mechanism and combined with the real-time reliability evaluation value of the sensors, the temporal feature vectors and spectral feature vectors of multiple sensors are fused to obtain dynamically weighted fused features. Finally, a Transformer feature fusion model is introduced to calculate the correlation degree of different positions, and the fused features are used to train the model to obtain the target model. Upper limb action recognition is performed using the obtained target model, which results in accurate feature extraction and greatly improves the accuracy of similar action recognition, solving the technical problems of difficult feature extraction and low accuracy of similar action recognition in the prior art for upper limb action recognition.

[0088] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described above.

[0089] Furthermore, embodiments of the present invention also propose a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the methods described above.

[0090] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0091] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0092] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0093] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0094] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A deep learning-based upper limb movement recognition method, characterized in that, include: Sequence data is constructed by collecting different upper limb movements from multiple sensors. The sequence data is preprocessed to obtain a spectrogram based on the preprocessed data. The preprocessed data includes a preprocessed time-domain sequence. Deep learning networks are used to extract features from the preprocessed time-domain sequence and spectrogram to output corresponding time-series feature vectors and spectrogram feature vectors. Both the time-series feature vectors and the spectrogram feature vectors are structured feature vectors that can be trained and fused by the model. Based on the attention mechanism and the real-time reliability evaluation value of the sensor, the temporal feature vector and spectral feature vector of multiple sensors are fused to obtain the dynamic weighted fusion feature. The fusion feature is then trained by a preset model to obtain the target model, so as to identify upper limb movements based on the target model. In the step of using a deep learning network to extract features from the preprocessed time-domain sequence and spectrogram to output the corresponding time-series feature vector and spectrogram feature vector, respectively... Methods for extracting time-series feature vectors include: Preprocessed three-dimensional acceleration data is obtained and input into a temporal feature extraction model composed of a deep learning network that integrates a temporal convolutional network and a bidirectional gated recurrent unit to achieve automatic extraction of temporal feature vectors. The temporal convolutional network, with residual blocks as its core, obtains local feature relationships of the preprocessed temporal sequence through causal convolutions within the residual blocks. The bidirectional gated recurrent unit extracts temporal sequence features from both the forward and reverse directions of the preprocessed temporal sequence and eliminates the phase shift in the bidirectional extraction through a temporal alignment module, thereby extracting long-term dependencies in the temporal sequence. The temporal feature extraction model includes an input adaptation layer, a temporal convolutional network layer, a dimension transformation layer, a bidirectional gated recurrent unit layer, an adaptive average pooling layer, and a fully connected layer. Methods for extracting spectral feature vectors include: The amplitude of the spectrogram is numerically normalized, and the spectrograms of different data segments are unified to a fixed size through bilinear interpolation. Then, the spectrograms of multiple sensors are arranged along the channel dimension to form a multi-channel spectrogram tensor. In the bilinear interpolation process, a multi-sensor channel alignment strategy is introduced to ensure the spatial consistency of the spectrogram data of different parts of the sensor. The local time-frequency texture of the spectrogram is extracted by a two-dimensional convolutional neural network. The feature map after dimensionality compression and feature selection is flattened in pixel order and input into a two-dimensional spatial long short-term memory network to model the global dependency relationship between time and frequency dimensions. Principal component analysis is performed on the multidimensional spectral feature vector output by the two-dimensional spatial long short-term memory network to remove redundant features and select key features, including the proportion of high-frequency components and time-frequency entropy, and output the corresponding spectral feature vector. The principal component analysis process includes: combining a time-frequency entropy weighted screening mechanism and dynamically adjusting a preset variance threshold based on prior knowledge of upper limb movement categories to retain principal components whose cumulative variance meets the variance threshold.

2. The deep learning-based upper limb movement recognition method according to claim 1, characterized in that, The input adaptation layer is used to adjust the preprocessed temporal sequence to fit the input of the temporal convolutional network layer. The temporal convolutional network layer contains two residual blocks. Each residual block is configured with two one-dimensional convolutional layers with kernel sizes of 3 and 5 in parallel. The output of each residual block is fused with the output of the residual branch and then activated by the ReLU activation function to output 32-channel local temporal features. The dimension transformation layer is used to adjust the order of feature dimensions to match the input requirements of the bidirectional gated recurrent unit layer; The bidirectional gated loop unit layer has two layers, which cycle bidirectionally in both the forward and reverse directions of the timing sequence. The number of output channels in the hidden layer is 64 and 32, respectively. The adaptive average pooling layer is a pooling layer with a kernel size of 1. After feature flattening, the input dimension is 32 and the output is a 64-dimensional temporal feature vector.

3. The deep learning-based upper limb movement recognition method according to claim 1, characterized in that, The steps involved in obtaining a target model by fusing temporal and spectral feature vectors from multiple sensors based on the attention mechanism and real-time reliability assessment values ​​of the sensors, and then training the fused features using a pre-defined model, are as follows: The time-series feature vector and the spectral feature vector are respectively input into the fully connected layer to generate a time-domain view descriptor and a frequency-domain view descriptor with a unified dimension; layer normalization is performed on the time-domain view descriptor and the frequency-domain view descriptor to eliminate the numerical scale differences of different view features; The time-domain view descriptor and frequency-domain view descriptor, after being normalized by the layer, are concatenated and input into a fully connected layer for feature interaction. Then, the dynamic attention score of the time-domain view descriptor and frequency-domain view descriptor is calculated by the Softmax attention calculation layer of the attention mechanism fusion module, combined with the real-time monitoring value of sensor noise. Based on the attention score, the time-series features and frequency-domain features are weighted and summed to generate multi-dimensional single-sensor fusion features. This fusion feature is input into the Transformer feature fusion model to concatenate the fusion features of multiple sensors. Through its multi-head self-attention module, the correlation features between sensors and between time and frequency domains are captured and global feature fusion training is performed to obtain the target model. Here, the time-series features are the feature component values ​​of each dimension of the time-series feature vector, and the frequency-domain features are the feature component values ​​of each dimension of the spectral feature vector.

4. The deep learning-based upper limb movement recognition method according to claim 1, characterized in that, The steps of constructing sequence data based on different upper limb movements collected from multiple sensors, and preprocessing the sequence data to obtain a spectrogram from the preprocessed data include: The three-dimensional acceleration data of different upper limb movements are acquired by the sensor and preprocessed. The preprocessed three-dimensional acceleration data is then converted into a two-dimensional spectrum using short-time Fourier transform. The data preprocessing methods include: The magnitude of the three-dimensional acceleration of each sensor is calculated to convert the three-dimensional acceleration data into one-dimensional data, thereby creating a one-dimensional time series view to reduce the complexity of the sequence data; One-dimensional data is input into a multi-order Butterworth low-pass filter for denoising. The multi-order Butterworth low-pass filter adopts an adaptive cutoff frequency adjustment mechanism and dynamically matches the filtering parameters according to the action type, so as to retain the subtle features of the action while denoising. Perform data standardization on the denoised one-dimensional data. The standardized data is segmented into continuous data segments by using an adaptive window size adjustment mechanism and a sliding window technique that dynamically adjusts the sliding window based on the upper limb movement rate.

5. The deep learning-based upper limb movement recognition method according to claim 4, characterized in that, The steps for converting preprocessed three-dimensional acceleration data into a two-dimensional spectrogram using short-time Fourier transform include: Each data segment is divided into consecutive overlapping frames based on the adaptive overlap rate of the action cycle, and a Hanning window with a window width dynamically adjusted according to the frequency distribution of the signal within the frame is applied to each frame to suppress spectral leakage caused by frame truncation. Perform a short-time Fourier transform on each windowed signal frame, retain positive frequency components, extract the single-sided spectrum, double the amplitude compensation for non-DC and quister frequency points, and eliminate conjugate symmetric amplitude distortion. The frequency and amplitude information of each frame are arranged in chronological order to form a time-frequency matrix. The time-frequency matrix includes time, frequency and amplitude, and the matrix element values ​​correspond to the signal amplitude at a specific time and frequency point. A two-dimensional spectrogram is generated by mapping the image based on the time-frequency matrix.

6. A deep learning-based upper limb movement recognition system, characterized in that, include: The data acquisition module is used to construct sequence data based on different upper limb movements collected by multiple sensors, and to preprocess the sequence data to obtain a spectrogram based on the preprocessed data. The preprocessed data includes a preprocessed time-domain sequence. The feature extraction module is used to extract features from the preprocessed time-domain sequence and spectrogram using a deep learning network to output corresponding time-series feature vectors and spectrogram feature vectors. The time-series feature vectors and the spectrogram feature vectors are both structured feature vectors that can be trained and fused by the model. The model training module is used to combine the attention mechanism with the real-time reliability evaluation value of the sensor, fuse the temporal feature vectors and spectral feature vectors of multiple sensors to obtain dynamic weighted fusion features, and train the fusion features through a preset model to obtain the target model, so as to recognize upper limb movements based on the target model. In the step of using a deep learning network to extract features from the preprocessed time-domain sequence and spectrogram to output the corresponding time-series feature vector and spectrogram feature vector, respectively... Methods for extracting time-series feature vectors include: Preprocessed three-dimensional acceleration data is obtained and input into a temporal feature extraction model composed of a deep learning network that integrates a temporal convolutional network and a bidirectional gated recurrent unit to achieve automatic extraction of temporal feature vectors. The temporal convolutional network, with residual blocks as its core, obtains local feature relationships of the preprocessed temporal sequence through causal convolutions within the residual blocks. The bidirectional gated recurrent unit extracts temporal sequence features from both the forward and reverse directions of the preprocessed temporal sequence and eliminates the phase shift in the bidirectional extraction through a temporal alignment module, thereby extracting long-term dependencies in the temporal sequence. The temporal feature extraction model includes an input adaptation layer, a temporal convolutional network layer, a dimension transformation layer, a bidirectional gated recurrent unit layer, an adaptive average pooling layer, and a fully connected layer. Methods for extracting spectral feature vectors include: The amplitude of the spectrogram is numerically normalized, and the spectrograms of different data segments are unified to a fixed size through bilinear interpolation. Then, the spectrograms of multiple sensors are arranged along the channel dimension to form a multi-channel spectrogram tensor. In the bilinear interpolation process, a multi-sensor channel alignment strategy is introduced to ensure the spatial consistency of the spectrogram data of different parts of the sensor. The local time-frequency texture of the spectrogram is extracted by a two-dimensional convolutional neural network. The feature map after dimensionality compression and feature selection is flattened in pixel order and input into a two-dimensional spatial long short-term memory network to model the global dependency relationship between time and frequency dimensions. Principal component analysis is performed on the multidimensional spectral feature vector output by the two-dimensional spatial long short-term memory network to remove redundant features and select key features, including the proportion of high-frequency components and time-frequency entropy, and output the corresponding spectral feature vector. The principal component analysis process includes: combining a time-frequency entropy weighted screening mechanism and dynamically adjusting a preset variance threshold based on prior knowledge of upper limb movement categories to retain principal components whose cumulative variance meets the variance threshold.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the deep learning-based upper limb motion recognition method as described in any one of claims 1-5.

8. A data processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the deep learning-based upper limb movement recognition method as described in any one of claims 1-5.

Images