A multi-modal progressive hierarchical fusion method for natural gesture recognition

Abstract

The application discloses a kind of multi-modal progressive hierarchical fusion methods for natural gesture recognition, comprising:1 obtains multi-modal data and carries out pre-processing and feature extraction;2 build multi-modal progressive hierarchical fusion deep learning network, including parallel for extracting single mode feature CNN subnetwork and the Transform subnetwork of fusing different modal level features;3 build modal missing detection module and decision layer post-processing mechanism;4 respectively in the decision layer between different CNN subnetwork and Transform subnetwork, introduce KL divergence loss function, and the joint cross-entropy loss function of each subnetwork trains model, obtains the final natural gesture recognition model.The application improves the robustness of gesture recognition, while ensuring the recognition performance of complete and incomplete samples of multi-modal information, so as to promote the popularization and application of human-computer interaction system based on gesture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of biological signal processing, and particularly relates to a multi-modal progressive hierarchical fusion method for natural gesture recognition. BACKGROUND

[0002] The human-computer interface based on gesture recognition is increasingly favored by the academic and business circles, and is an intuitive and feasible human-computer interaction scheme. The multi-modal gesture recognition based on surface electromyography (sEMG) and inertial measurement unit (IMU) signal has the advantages of portability and wearability, and overcomes the deficiencies of the visual-based gesture recognition scheme which usually needs a fixed place. On the other hand, the multi-modal recognition scheme provides more robust gesture recognition performance by obtaining complementary information of different modal signals. The sEMG accompanied by human muscle contraction reflects the motion joint activity intensity and flexion state, as well as the hand movement, shape, position and orientation information in the gesture completion process, and has unique advantages in identifying fine-grained gesture recognition. The IMU signal has better robustness and stability, and is often used in the motion recognition field. At the same time, the multi-modal scheme effectively overcomes the limitations of the single-modal gesture recognition scheme. For example, the electromyography has natural non-stationarity and weakness, and the electromyography signal quality will decrease due to interference such as sweating, electrode offset and motion noise. In particular, the electromyography signal of the amputated part of the amputated patient is more weak and low signal-to-noise ratio, thereby causing the gesture recognition performance based on electromyography to be seriously reduced, affecting the experience of gesture interaction, and even causing safety problems such as misoperation. The physical signal such as IMU is susceptible to external environment interference, and the sensor needs to be calibrated, which also causes inaccurate recognition.

[0003] It is a reliable scheme to improve the accuracy and robustness of gesture recognition by using multi-modal signal fusion scheme. Generally, there are two traditional multi-modal fusion strategies: early fusion or late fusion. Early fusion refers to fusing two signals according to certain rules first, and then sending them into a classifier to get the gesture classification result. This strategy can better utilize the interaction information between the two signals, but it does not fully utilize the single modal information. Late fusion refers to training two classifiers using two signals respectively, and then fusing the classification results of the two classifiers according to certain rules to get the final classification result. This strategy can fully utilize the single modal signal information, but ignores the interaction information between modalities. How to more efficiently fuse multi-modal information is a challenging task. In addition, the existing multi-modal gesture recognition methods are all carried out under a strong hypothesis condition, that is, it is assumed that multi-modal information exists during testing. The research on multi-modal gesture recognition based on sEMG and IMU under the condition of missing modalities in the test stage is still shallow. The performance of the multi-modal gesture recognition human-computer interface will inevitably decline due to data missing in actual use. It is of important application significance to improve the recognition performance of the model for modal missing samples. SUMMARY

[0004] The present application is to solve the above-mentioned deficiencies in the prior art, and proposes a progressive hierarchical fusion method for natural gesture recognition, in order to improve the robustness of gesture recognition, and at the same time to ensure the recognition performance of complete and incomplete samples of multi-modal information, so as to promote the popularization and application of gesture-based human-computer interaction system

[0005] In order to achieve the above-mentioned application purposes, the present application adopts the following technical solutions:

[0006] The multi-modal hybrid fusion method for natural gesture recognition has the characteristics that it includes the following steps:

[0007] Step 1, obtain a multi-modal gesture dataset X={sEMG, ACC} and its gesture class label set Y, and pre-process and extract features from the multi-modal gesture dataset X, so as to construct a training set containing signal feature map samples; wherein sEMG represents a sparse channel surface electromyography signal, and ACC represents an accelerometer signal;

[0008] Step 1.1, data preprocessing:

[0009] The surface electromyography signal sEMG and the accelerometer signal ACC are respectively filtered, normalized and active segment sliding window segmented to obtain an electromyography signal sample set S containing N1 original electromyography signal samples raw And an accelerometer signal sample set A containing N1 accelerometer signal samples raw ;

[0010] Step 1.2, Feature Extraction:

[0011] The electromyographic signal sample set S was extracted using time-frequency domain methods. raw The time-frequency domain feature map of each original electromyography (EMG) signal sample is used to obtain an EMG signal training set S = {S1} containing N1 EMG signal feature map samples. i |i=1,2,…,N1};where S i ∈R d1×ch1 Let S represent any i-th electromyography (EMG) signal feature map sample; d1 and ch1 represent the single-channel feature dimension and channel number of the EMG signal feature map sample, respectively; let S i The gesture category label is Y i ;

[0012] Accelerometer signal sample set A is extracted using time-domain methods. raw The time-domain feature map of each accelerometer signal sample is used to obtain the accelerometer signal training set A = {A}, which contains N1 accelerometer signal feature map samples. i |i=1,2,…,N1};where, Let A represent any i-th accelerometer signal feature map sample; d2 and ch2 represent the single-channel feature dimension and number of channels of the accelerometer signal feature map sample, respectively; let A i The gesture category label is also Y. i ;

[0013] Step 1.3: Construct a training set {S,A,Y} containing multimodal signal feature map samples, where Y represents the set of gesture category labels for signal feature map samples in the electromyography signal training set S and the accelerometer signal training set A, and Y contains L gesture categories;

[0014] Step 2: Construct a multimodal progressive hierarchical fusion deep learning network, including three parallel sub-networks: sEMG single-modal CNN sub-network, ACC single-modal CNN sub-network, and sEMG-ACC hybrid-modal Transformer sub-network; each sub-network includes a feature extraction module and a pattern recognition module.

[0015] Step 2.1: Transfer the i-th electromyography feature map sample S i The input is fed into the sEMG unimodal CNN subnetwork, and after processing by the feature extraction module, the dimensionality-reduced sEMG unimodal features S are obtained. i,down ;

[0016] The dimensionality-reduced sEMG single-modal features S i,down After processing in the pattern recognition module of the sEMG unimodal CNN subnetwork, the output is the classification result vector S of the sEMG unimodal CNN subnetwork. i,score ;

[0017] Step 2.2: Transfer the feature map sample A of the i-th accelerometer signal i The input is fed into the ACC unimodal CNN subnetwork, and after processing by the feature extraction module, the dimensionality-reduced ACC unimodal feature A is output. i,down ;

[0018] The dimensionality-reduced ACC single-modal feature A i,down After processing in the pattern recognition module of the ACC unimodal CNN subnetwork, the classification result vector A of the ACC unimodal CNN subnetwork is output. i,score ;

[0019] Step 2.3: Obtain the multimodal hierarchical fusion feature SA from the feature extraction module in the sEMG-ACC hybrid modal Transformer subnetwork. i,K After being processed by the corresponding pattern recognition module, the classification result vector SA of the sEMG-ACC hybrid modal Transformer subnetwork is output. i,score ;

[0020] Step 3: Construct a modality missing detection module and a decision-making layer post-processing mechanism;

[0021] Step 3.1: The modality loss detection module calculates the i-th electromyographic signal feature map sample S. i The statistical value is used to determine the feature map sample S of the i-th electromyography signal. i If the statistical value does not exist, proceed to step 4.2; otherwise, it indicates that the sEMG modality is missing and is replaced with a zero sample of the same size.

[0022] Step 3.2: Determine the feature map sample S of the i-th electromyography signal. i The difference between the statistical value of S and the mean statistical value of S exceeds the set threshold. If it does, it indicates that the difference between the i-th electromyography signal feature map sample S is greater than or equal to the set threshold. i Anomalies are identified and replaced with zero samples of the same size.

[0023] Step 3.3: Following the procedures in Steps 3.1 and 3.2, process the feature map sample A of the i-th accelerometer signal. i Process it;

[0024] Step 3.4: Post-processing mechanism at the decision-making level:

[0025] When the modality missing detection module determines that both modalities are present, the final classification result vector is R. i =A i,score +S i,score +SA i,score ;

[0026] When the modal missing detection module determines that the ACC modal is missing and only the sEMG modal exists, the final classification result vector is R i = S i,score ;

[0027] When the modal missing detection module determines that the sEMG modal is missing and only the ACC modal exists, the final classification result vector is R i = A i,score ;

[0028] Step 4, network training:

[0029] Step 4.1, normalize the classification result vector of any sub-network using the sofmax function to obtain a normalized classification result vector, which is used to construct the cross-entropy loss of the corresponding sub-network, and obtain the cross-entropy loss L CE_S of the sEMG single modal CNN sub-network, the cross-entropy loss L CE_A of the ACC single modal CNN sub-network, and the cross-entropy loss L CE_F of the Transformer sub-network of fused hierarchical features.

[0030] Step 4.2, using the classification result vector S i,score of the sEMG single modal CNN sub-network and the classification result vector SA i,score of the Transformer sub-network of fused hierarchical features, construct the KL divergence loss function L KL_S between the sEMG single modal CNN sub-network and the sEMG-ACC hybrid modal Transformer sub-network.

[0031] Using the classification result vector A i,score of the ACC single modal CNN sub-network and the classification result vector SA i,score of the Transformer sub-network of fused hierarchical features, construct the KL divergence loss function L KL_A between the ACC single modal CNN sub-network and the sEMG-ACC hybrid modal Transformer sub-network, so that the distribution of the classification result of the single modal signal is aligned with the distribution of the multi-modal classification result.

[0032] Step 4.3, based on the training set {S, A, Y}, the multi-modal hybrid fusion model is trained using the Adam method, and the total loss function L total is calculated. CE_S + L CE_A + L CE_F + L KL_S + L KL_A , until the total loss function L totalConverge to stop, thereby obtaining a trained natural gesture classification model for gesture recognition on any kind of multi-modal gesture data.

[0033] The multi-modal progressive hierarchical fusion method for natural gesture recognition has the characteristics that,

[0034] The feature extraction module in the sEMG single-modal CNN subnetwork and the ACC single-modal CNN subnetwork comprises a dimension increasing unit, a single-modal hierarchical feature fusion unit and a dimension decreasing unit.

[0035] The dimension increasing unit comprises l1 convolutional layers with a 1x1 convolution kernel.

[0036] The single-modal hierarchical feature fusion unit comprises a K-level feature extraction network and a single-modal hierarchical feature fusion layer; each level of the K-level feature extraction network comprises, in sequence, m convolutional layers with a kxk convolution kernel, a batch normalization layer and a ReLu activation function layer; the single-modal hierarchical feature fusion layer is configured to splice and fuse the single-modal hierarchical features output by each level of the K-level feature extraction network in the channel dimension.

[0037] The dimension decreasing unit comprises, in sequence, l2 convolutional layers with a 1x1 convolution kernel, a batch normalization layer, a ReLu activation function layer and a dropout layer.

[0038] The feature extraction module of the sEMG-ACC hybrid-modal Transformer subnetwork comprises a K-level fused feature Transformer network; the Transformer network is composed of an input embedding layer and a feature encoder; the input embedding layer is provided with the same fully connected layer for the sEMG modal, the ACC modal and the sEMG-ACC hybrid modal; the feature encoder comprises a layer normalization layer, a multi-head attention unit, a layer normalization layer and a feedforward unit; the feedforward unit comprises a fully connected layer, a GELU activation function layer, a fully connected layer and a Dropout layer.

[0039] The mode recognition module of each subnetwork is composed of F fully connected layer submodules; the first fully connected layer submodule comprises, in sequence, a fully connected layer with w1 neurons, a batch normalization layer, a ReLu activation function layer and a dropout layer; the dth fully connected layer submodule comprises, in sequence, a fully connected layer with w d d neurons, a batch normalization layer and a ReLu activation function layer, d e {2,3,…,F-1}; the Fth fully connected layer submodule is a fully connected layer with w F F neurons.

[0040] The upscaling unit of the feature extraction module in the sEMG single-modal CNN subnetwork processes the i-th electromyography feature map sample S. i After processing, the upgraded feature S is output. i,up The data is then input into a single-modal hierarchical feature fusion unit and processed sequentially through a K-level feature extraction network. Each level outputs sEMG single-modal hierarchical features {S}. i,k |k∈{1,2,…,K}};where S i,k S represents i The k-th level feature;

[0041] The sEMG unimodal CNN subnetwork's unimodal hierarchical feature fusion layer will fuse the sEMG unimodal hierarchical features {S} output from each level. i,k |k∈{1,2,…,K}} are concatenated and merged according to channels to obtain S i Hierarchical fusion feature S i,fus ;

[0042] The dimensionality reduction unit in the sEMG single-modal CNN subnetwork will hierarchically fuse features S i,fus Dimensionality reduction is performed to obtain the dimensionality-reduced sEMG single-modal features S. i,down .

[0043] The upscaling unit of the feature extraction module in the ACC single-modal CNN subnetwork processes the i-th electromyography feature map sample S. i After processing, the upgraded feature A is output. i,up The data is then input into a single-modal hierarchical feature fusion unit and processed sequentially through a K-level feature extraction network. Each level outputs ACC single-modal hierarchical features {A}. i,k |k∈{1,2,…,K}};where A i,k A represents i The k-th level feature;

[0044] The unimodal hierarchical feature fusion layer in the ACC unimodal CNN subnetwork will fuse the ACC unimodal hierarchical features {A} output from each level. i,k |k∈{1,2,…,K}} are concatenated and merged according to channels to obtain A i Hierarchical fusion feature A i,fus ;

[0045] The dimensionality reduction unit in the ACC single-modal CNN subnetwork will hierarchically fuse feature A. i,fus Dimensionality reduction is performed to obtain the dimensionality-reduced ACC single-modal feature A. i,down .

[0046] The i-th electromyography feature sample S i Feature S after dimensionality upgrade i,upand the i th accelerometer signal feature map sample A i the dimensionality-increased feature A i,up obtained after addition i,up ;

[0047] When k = 1, the initial fused feature SA i,up , the k th level hierarchical feature S i of S i,k and the k th level hierarchical feature A i of A i,k are input into the sEMG-ACC hybrid modal Transformer subnetwork, and after processing by the k th level fused hierarchical feature Transformer network, the k th level fused multi-modal hierarchical fusion feature SA i,k is obtained.

[0048] When k = 2, 3, …, K, the (k-1) th level fused hierarchical fusion feature SA i,k-1 , the k th level hierarchical feature S i of S i,k and the k th level hierarchical feature A i of A i,k are input into the k th level fused hierarchical feature Transformer network for processing, and the k th level fused multi-modal hierarchical fusion feature SA i,k is obtained, and finally the K th level fused multi-modal hierarchical fusion feature SA i,K is output by the K th level fused hierarchical feature Transformer network.

[0049] The multi-modal hierarchical fusion feature SA i,K is input into the pattern recognition module of the sEMG-ACC hybrid modal Transformer subnetwork, and after processing by the F full connection layer modules, the classification result vector SA i,score of the sEMG-ACC hybrid modal Transformer subnetwork is output.

[0050] An electronic device according to the present application comprises a memory and a processor, and is characterized in that the memory is configured to store a program supporting the processor to execute any of the multi-modal progressive hierarchical fusion methods, and the processor is configured to execute the program stored in the memory.

[0051] A computer readable storage medium according to the present application, wherein the computer readable storage medium stores a computer program, and the computer program is characterized in that when executed by a processor, the computer program performs the steps of any of the multi-modal progressive hierarchical fusion methods.

[0052] Compared with the prior art, the present application has the beneficial effects that:

[0053] 1、The present application designs a multi-modal progressive hierarchical fusion model framework, learns the deep features of sEMG and IMU through a deep neural network, and uses a Transformer network to progressively fuse single-modal features and mixed-modal features of different levels, so that the proposed model not only obtains specific information of single-modal signals, but also fully excavates the interaction information between multi-modal signals, thereby improving the accuracy of gesture classification.

[0054] 2、The present application introduces a soft alignment operation based on KL divergence between the single-modal and mixed-modal of the designed multi-modal progressive hierarchical fusion model, so that the feature distribution of each single-modal is aligned to the feature distribution of the mixed-modal, thereby improving the gesture recognition ability of the single-modal feature by using the mixed-modal feature; finally, through a modal missing detection module and a decision layer post-processing mechanism, the recognition performance of the model for test samples with incomplete modal information is improved. This method not only optimizes the test performance of test samples with complete multi-modal information, but also ensures that the recognition rate is still higher than that of the recognition method based on single-modal signals when the test modal is incomplete. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The data preprocessing flowchart of the present application;

[0056] Figure 2 The multi-modal progressive hierarchical fusion neural network model of the present application;

[0057] Figure 3 The single Transformer network diagram of the present application;

[0058] Figure 4a The multi-head attention unit diagram of the Transformer network of the present application;

[0059] Figure 4b The self-attention unit diagram of the multi-head attention unit of the Transformer network of the present application. DETAILED DESCRIPTION

[0060] In this embodiment, a multi-modal progressive hierarchical fusion method for natural gesture recognition is a natural gesture recognition method based on Transformer modal fusion and KL divergence soft alignment. The method learns the deep features of sEMG and IMU through a convolutional neural network (CNN), and uses a Transformer network to progressively fuse single-modal features and mixed-modal features at different levels, so that the proposed model can fully learn the specific information of single-modal signals and the interaction information between multi-modal signals. In particular, the soft alignment of the KL divergence loss function is introduced between single-modal and mixed-modal, so that the feature distribution of each single-modal is aligned to the feature distribution of the mixed-modal, thereby improving the gesture recognition capability of single-modal features. Finally, through the modal missing detection module and the decision layer post-processing mechanism, the recognition performance of the model for incomplete modal test samples is improved. The fusion method can improve the robustness of the natural gesture recognition system, especially increase the recognition accuracy of the model for modal missing samples in the test phase, thereby promoting the popularization and application of the gesture-based human-computer interaction system. Specifically, the multi-modal mixed fusion method includes the following steps:

[0061] Step 1, obtain a multi-modal gesture dataset {X, Y} = {sEMG, ACC, Y}, where sEMG represents sparse channel surface electromyography signals, ACC represents accelerometer signals, and Y represents a gesture class label set for multi-modal data {sEMG, ACC}; pre-process and extract features from the multi-modal data {sEMG, ACC}; and finally construct a training set containing signal feature map samples;

[0062] Step 1.1, data preprocessing:

[0063] Filter, normalize, and activity segment sliding window segmentation are performed on the surface electromyography signals sEMG and the accelerometer signals ACC, respectively, to obtain an electromyography signal sample set S containing N1 original electromyography signal samples raw and an accelerometer signal sample set A containing N1 accelerometer signal samples raw Each sample obtains a corresponding class label.

[0064] In a specific implementation, the surface electromyography signals are 12-channel sparse channel electromyography, and the accelerometer signals are 36 channels, i.e., each channel of the 12-channel electromyography channel has corresponding X, Y, and Z three-axis accelerometer signals.

[0065] The data preprocessing flowchart is shown in Figure 1 A fourth-order Butterworth filter is used to perform 10-500Hz band-pass filtering and 50Hz power frequency notch filtering on the surface electromyography signals; a fourth-order Butterworth filter is used to perform 100Hz low-pass filtering on the surface electromyography signals.

[0066] Normalization is to first calculate the mean and variance of each channel of the training set signal, and then normalize the training set and test set data using the mean and variance of each channel of the training set, so that the mean of each channel data is 0 and the variance is 1;

[0067] The active segment sliding window segmentation is to use the sliding window method to respectively cut the surface electromyography signal and the accelerometer signal, and in specific implementation, the sliding window size is 200ms and the step length is 10ms;

[0068] Step 1.2, feature extraction:

[0069] Using the time-frequency domain method to extract the electromyography signal sample set S raw The time-frequency domain feature map of each original electromyography signal sample, obtaining an electromyography signal training set S containing N1 electromyography feature map samples S i |i=1,2,…,N1}; wherein, represents the i-th electromyography feature map sample; d1 and ch1 respectively represent the single-channel feature dimension and the channel number of the electromyography feature map sample; let S i The gesture category label of the hand is Y i ;

[0070] In specific implementation, the electromyography features include: Marginal of Discrete Wavelet Transform (mDWT), Root Mean Square (RMS), Slope Sign Change (SSC), Waveform Length (WL), Mean Absolute Value (MAV), Willison Amplitude (WAMP), Autoregressive Coefficients (ARC), Mean Frequency (MNF), and Power Spectrum Ratio (PSR). For each surface electromyography signal sample, the features are calculated according to the channel, and the obtained features are spliced according to the column to form an electromyography signal sample containing electromyography feature maps;

[0071] Using the time domain method to extract the accelerometer signal sample set A raw The time domain feature map of each accelerometer signal sample, obtaining an accelerometer signal training set A containing N1 accelerometer signal feature map samples A i |i=1,2,…,N1}; wherein, A i ∈R d2×ch2represents an arbitrary i-th accelerometer signal feature map sample; d2 and ch2 represent the single channel feature dimension and the number of channels of the accelerometer signal feature map sample, respectively; let A i the gesture category label is Y i ;

[0072] In specific implementation, the specific accelerometer signal features include: mean value (Mean), variance (Variance, VAR), root mean square (Root Mean Square, RMS), waveform length value (Waveform Length, WL), mean absolute value (Mean Absolute Value, MAV), mean absolute value slope (Mean Absolute Value Slope, MAVS). For each accelerometer signal sample, the features are calculated by channel, and the obtained features are spliced by column to form an accelerometer signal sample containing an accelerometer signal feature map;

[0073] Step 1.3, constructing a training set {S, A, Y} containing multi-modal signal feature map samples, wherein S represents an electromyography signal training set containing electromyography feature map samples, A represents an accelerometer signal training set containing accelerometer signal feature map samples, and Y represents a label set of the samples;

[0074] In specific implementation, each surface electromyography signal sample has corresponding accelerometer signal sample and label, and the three are strictly corresponding;

[0075] Step 2, constructing a multi-modal progressive hierarchical fusion deep learning network, the overall structure is as shown in Figure 2 , including: three parallel sub-networks, which are sEMG single modal CNN sub-network, ACC single modal CNN sub-network and sEMG-ACC mixed modal Transformer sub-network: each sub-network includes a feature extraction module and a pattern recognition module;

[0076] The feature extraction module in the sEMG single modal CNN sub-network and the ACC single modal CNN sub-network includes: an up-sampling unit, a single modal hierarchical feature fusion unit and a down-sampling unit;

[0077] The up-sampling unit includes λ1 convolution layers with a convolution kernel size of 1×1; for example, the filter number of the convolution layer of the up-sampling unit is set to λ1=64;

[0078] The single-modal hierarchical feature fusion unit comprises a K-level feature extraction network and a single-modal hierarchical feature fusion layer. Each level of the K-level feature extraction network comprises, in sequence, a convolutional layer with μ convolutional kernels of size k x k, a batch normalization layer, and a ReLu activation function layer; the single-modal hierarchical feature fusion layer fuses the outputs of the K-level feature extraction network, i.e., the single-modal features of different levels, by concatenation in the channel dimension; by way of example, the number of levels of the feature extraction network is set to 3, μ is set to 64, and the convolutional kernel sizes of the k x k convolutional layers in the 3-level feature extraction network are set to 3 x 3, 5 x 5, and 3 x 3, respectively;

[0079] The dimension reduction unit comprises, in sequence, λ2 convolutional layers with 1 x 1 convolutional kernels, a batch normalization layer, a ReLu activation function layer, and a dropout layer; by way of example, λ2 is set to 64, and the dropout ratio is set to 0.65;

[0080] The feature extraction module of the Transformer subnetwork comprises K-level fused Transformer networks, each of which is as shown in Figure 3 The Transformer network is composed of an input embedding layer and a feature encoder. The input embedding layer is a fully connected layer, and by way of example, the number of neurons of the fully connected layer for linearly mapping each type of modal hierarchical feature is set to 216, and a same-dimension learnable parameter clc_token and modal_token is added to the mapped features, representing gesture class encoding information and modal encoding information, respectively. Then, the modal information is concatenated and fused by channel and sent to the feature encoder. The feature encoder comprises a layer normalization layer, a multi-head attention unit, a layer normalization layer, and a feedforward unit. The multi-head attention unit is as shown in Figure 4a and Figure 4b By way of example, the number of attention heads is set to 8, and the attention encoding feature dimension is set to 64. The feedforward unit comprises a fully connected layer, a GELU activation function layer, and a fully connected layer. By way of example, in the feedforward unit, the number of hidden neurons between the two fully connected layers of the feedforward unit is set to 128, and the feedforward unit does not change the dimension of the input features;

[0081] The pattern recognition module of each subnetwork is composed of F fully connected layer submodules. The first fully connected layer submodule comprises, in sequence, a fully connected layer with w1 neurons, a batch normalization layer, a ReLu activation function layer, and a dropout layer; the dth fully connected layer submodule comprises, in sequence, a fully connected layer with w d neurons, a batch normalization layer, and a ReLu activation function layer, d e {2, 3, …, F-1}; and the Fth fully connected layer submodule is a fully connected layer with w Ffull connection layer of neurons; exemplary, F is set to 3, w1 = w2 is set to 512, the dropout ratio is set to 0.65, and w F L is the number of gesture categories in the data set, which is determined according to the sample condition of the data set;

[0082] Step 2.1: input the i th sEMG feature map sample S i into the sEMG monomodal CNN subnetwork, and sequentially pass through the processing of the dimension increasing units in the feature extraction module to output the dimension-increased features S i,up , and then input into the monomodal hierarchical feature fusion unit, and sequentially pass through the processing of the K-level feature extraction network, and output the features {A i,k |k∈{1,2,…,K}} from each level; wherein, A i,k represents the k-level hierarchical features of S i ; exemplary, the sEMG monomodal CNN subnetwork will obtain 3 hierarchical features, and the number of channels of each hierarchical feature is 64;

[0083] The monomodal hierarchical feature fusion layer splices and fuses the K hierarchical features {A i,k |k∈{1,2,…,K}} by channels to obtain the hierarchical fusion features S i of S i,fus ; exemplary, the number of channels of the sEMG monomodal hierarchical fusion features S i,fus is 192;

[0084] The dimension reduction unit in the sEMG monomodal CNN subnetwork performs dimension reduction processing on the hierarchical fusion features S i,fus to obtain the dimension-reduced sEMG monomodal features S i,down ; exemplary, the number of channels of the dimension-reduced sEMG monomodal features S i,down is 64;

[0085] The dimension-reduced sEMG monomodal features S i,down are input into the pattern recognition module in the sEMG monomodal CNN subnetwork, and after the processing of the F full connection layer modules, the classification result vector S i,score of the sEMG monomodal CNN subnetwork is output.

[0086] Step 2.2: input the i th accelerometer signal feature map sample A i into the ACC monomodal CNN subnetwork, and after the processing of the dimension increasing units in the feature extraction module, output the dimension-increased features A i,up , and then input into the monomodal hierarchical feature fusion unit, and sequentially pass through the processing of the K-level feature extraction network, and output the ACC monomodal hierarchical features {A i,k |k∈{1,2,…,K}} from each level; wherein, Ai,k represents the kth level feature of A i ; for example, the ACC single-modal CNN subnetwork will obtain 3 level features, and the number of channels of each level feature is 64;

[0087] The single-modal level feature fusion layer in the ACC single-modal CNN subnetwork fuses the ACC single-modal level features {A i,k | k e {1, 2, …, K}} output by each level by channel concatenation, to obtain the level fusion feature A i of A i,fus ; for example, the number of channels of the ACC single-modal level fusion feature A i,fus is 192;

[0088] The dimension reduction unit in the ACC single-modal CNN subnetwork performs dimension reduction processing on the level fusion feature A i,f2 to obtain the ACC single-modal feature A i,down after dimension reduction; i,down ; for example, the number of channels of the ACC single-modal feature A i,down after dimension reduction is 64;

[0089] The ACC single-modal feature A i,down after dimension reduction is input into the mode recognition module of the ACC single-modal CNN subnetwork, and after processing by F fully connected layer modules, the classification result vector A i,score of the ACC single-modal CNN subnetwork is output.

[0090] Step 2.3: After the i-th sEMG feature map sample S i is dimensionally increased, and the i-th accelerometer signal feature map sample A i,up is dimensionally increased, the dimensionally increased feature S i and the dimensionally increased feature A i,up are added to obtain the initial fused feature SA i,up .

[0091] When k = 1, the initial fused feature SA i,up , the kth level feature S i of A i,k , and the kth level feature A i of A i,k are input into the sEMG-ACC hybrid modal Transformer subnetwork, and after processing by the Transformer network of the kth level fusion level feature, the level fusion feature SA i,k after k-level feature fusion is obtained.

[0092] When k = 2, 3, …, K, the level fusion feature SA i,k-1 after k-1-level feature fusion, S ithe kth hierarchical feature S i,k and A i the kth hierarchical feature A i,k After being input into the Transformer network of the kth hierarchical feature and processed, the kth hierarchical feature fusion layer feature SA i,k is obtained, and the kth hierarchical feature fusion layer feature SA i,K is output by the Transformer network after the kth hierarchical feature fusion.

[0093] the hierarchical fusion feature SA i,K is input into the mode recognition module of the sEMG-ACC hybrid modal Transformer subnetwork and processed by F full connection layer modules, and the classification result vector SA i,score of the sEMG-ACC hybrid modal Transformer subnetwork is output.

[0094] Step 3, a modal missing detection module and a decision layer post-processing mechanism are constructed.

[0095] Step 3.1, the statistical value of the ith electromyographic signal feature map sample S i is calculated, and it is judged whether the statistical value of the ith electromyographic signal feature map sample S i exists or not. If it exists, step 4.2 is executed; otherwise, it is indicated that the sEMG modal is missing, and a zero sample of the same size is used to replace it.

[0096] Step 3.2, it is judged whether the difference between the statistical value of the ith electromyographic signal feature map sample S i and the average value of the statistical value of S exceeds the set threshold value. If it exceeds, it is indicated that the ith electromyographic signal feature map sample S i is abnormal, and a zero sample of the same size is used to replace it. Exemplarily, the feature map sample statistical value takes the mean and variance of all elements of the current feature map sample. If the mean and variance of all elements of the current feature map sample do not exist, or are more than 10 times or less than 0.1 times the average value of the statistical value of all feature map samples in the training set, it is determined that the current modal information is missing.

[0097] Step 3.3, the ith accelerometer signal feature map sample A i is processed according to the processes of step 3.1 and step 4.2.

[0098] Step 3.4: the decision layer post-processing mechanism, as shown in Figure 2 .

[0099] When the modal missing detection module determines that both modalities exist, let Figure 2 g sEMG , g ACC , g Fusion be 1, 1 and 1 respectively, that is, the final classification result vector is R i = A i,score + S i,score + SA i,score ;

[0100] When the modal missing detection module determines that the ACC modality is missing and only the sEMG modality exists, let Figure 2 g sEMG , g ACC , g Fusion be 1, 0 and 0 respectively, that is, the final classification result vector is R i = S i,score ;

[0101] When the modal missing detection module determines that the sEMG modality is missing and only the ACC modality exists, let Figure 2 g sEMG , g ACC , g Fusion be 0, 1 and 0 respectively, that is, the final classification result vector is R i = A i,score ;

[0102] Step 4, network training:

[0103] Step 4.1, normalize the classification result vector of any sub-network using the sofmax function to obtain the normalized classification result vector, which is used to construct the cross-entropy loss of the corresponding sub-network, and obtain the cross-entropy loss L CE_S of the sEMG single modal CNN sub-network, the cross-entropy loss L CE_A of the ACC single modal CNN sub-network and the cross-entropy loss L CE_F of the Transformer sub-network of fused hierarchical features.

[0104] Step 4.2, construct the KL divergence loss function L KL_S between the classification result vector S i,score of the sEMG single modal CNN sub-network and the classification result vector SA i,score of the sEMG-ACC hybrid modal Transformer sub-network.

[0105] Use the classification result vector A i,score of the ACC single modal CNN sub-network and the classification result vector SAi,score The KL divergence loss function L between the ACC single-modal CNN subnetwork and the sEMG-ACC hybrid-modal Transformer subnetwork is constructed KL_A ; so that the distribution of the classification results of the single-modal signals aligns with the distribution of the multi-modal classification results; the KL divergence loss can be expressed as:

[0106]

[0107]

[0108] In formula (1) and formula (2), x i,j is the jth probability value in the classification result vector S i,score or A i,score or SA i,score ; L represents the number of gesture categories;

[0109] Step 4.3, based on the training set {S, A, Y}, the multi-modal hybrid fusion model is trained by using the Adam method, and the total loss function L total is calculated CE_S +L CE_A +L CE_F +L KL_S +L KL_A , until the total loss function L total converges, thereby obtaining the trained natural gesture classification model for gesture recognition on any kind of multi-modal gesture data.

[0110] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above-mentioned multi-modal progressive hierarchical fusion method, and the processor is configured to execute the program stored in the memory.

[0111] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above-mentioned multi-modal progressive hierarchical fusion method.

[0112] In specific implementation, the proposed neural network is implemented using the pytorch deep learning framework, the network is optimized and trained using the Adam optimizer, the number of samples per batch is set to 512, the neural network learning rate is set to [0.001, 10 -4 , 10 -5 , 10 -6 ], and 3 rounds, 6 rounds, 3 rounds and 1 round are learned in turn at each learning rate.

[0113] The feasibility and effectiveness of the proposed model in the case of complete information of test samples and the case of missing modalities of test samples are verified by experiments on three public datasets, namely, Ninapro DB2, DB3 and B7. Gesture recognition is performed on the public datasets using the multi-modal progressive hierarchical fusion model proposed in the application, and the gesture recognition accuracy is used as an evaluation index. The gesture recognition accuracy of each subject is defined as the number of correctly classified samples / total sample number x 100%. The number of subjects participating in the experiment in the three public datasets is 40, 6 and 20 respectively. The experimental results are the average gesture recognition accuracy on a single dataset, and the average gesture recognition rate is defined as the sum of the gesture recognition accuracy of the subjects participating in the experiment in the dataset / the number of subjects participating in the experiment in the dataset x 100%. The test results show that in the case of complete modal information of test samples, the experimental results of the method proposed in the application and the experimental results of the most advanced HyFusion method in the literature are subjected to paired t-test. The performance of the method proposed in the application is significantly higher than that of the most advanced multi-modal gesture recognition algorithm in the existing literature. In the case of incomplete modal information of test samples, the sEMG modality feature map or the ACC modality feature map in all test samples is set to zero, and the zero sample and the existing modality information are input into the model to obtain the test accuracy of the entire dataset. For fair comparison, the single modal model uses the single modal CNN subnetwork in the application, and the single modal test performance of sEMG and ACC in the single modal training condition is tested. Finally, the experimental results of the method proposed in the application and the experimental results of the single modal CNN subnetwork are subjected to paired t-test. The test results show that the test performance of the model is significantly better than that of the single modal CNN subnetwork.

[0114] Table 1 Comparison of experimental results of the method of the application and other methods in the case of complete information of test samples

[0115]

[0116] *P < 0.05; **P < 0.01.

[0117] Table 2 Comparison of experimental results of the method of the application and the single modal CNN model in the case of incomplete information of test samples

[0118]

[0119] *P < 0.05; **P < 0.01.

[0120] The experimental results in Table 1 show that the multi-modal progressive hierarchical fusion model proposed in the application obtains the highest performance under the condition that the information of the test sample is complete. The experimental results in Table 2 show that the recognition rate of the method proposed in the application is still higher than that of the recognition method based on a single modal signal when the test modal sample is incomplete. In the experimental results, the comparative method is the HyFusion method proposed by Duan et al. in the literature "AHybrid Multimodal Fusion Framework for sEMG-ACC-Based Hand Gesture Recognition" and the multi-view CNN (MV-CNN) proposed by Wei et al. in the literature "Surface-electromyography-based gesture recognition by multi-view deep learning". It should be noted that the experimental data division method of the application is consistent with the division method used by HyFusion and MV-CNN when performing multi-modal gesture recognition on Ninapro DB2, DB3 and DB7.

Claims

1. A multimodal progressive hierarchical fusion method for natural gesture recognition, characterized in that, Includes the following steps: Step 1: Obtain the multimodal gesture dataset and its gesture category label set and on a multimodal gesture dataset Preprocessing and feature extraction are performed to construct a training set containing signal feature map samples; among which, This represents surface electromyography (EMG) signals in sparse channels. This indicates the accelerometer signal; Step 1.1, Data Preprocessing: surface electromyography signals and accelerometer signal Filtering, normalization, and active segment sliding window segmentation are performed separately to obtain the result containing A collection of original electromyography (EMG) signal samples. and contain A set of accelerometer signal samples ; Step 1.2, Feature Extraction: Extracting electromyographic signal sample sets using time-frequency domain methods. The time-frequency domain feature map of each original electromyography signal sample is obtained, containing Electromyography (EMG) signal training set of 100 EMG signal feature map samples ;in, Represents any number of i One electromyography signal feature map sample; and Let represent the single-channel feature dimension and the number of channels of the electromyographic signal feature map sample, respectively; let The gesture category label is ; Extracting accelerometer signal sample sets using time-domain methods The time-domain feature map of each accelerometer signal sample is obtained, containing Accelerometer signal training set of accelerometer signal feature map samples ;in, Represents any number of i One accelerometer signal feature map sample; and Let represent the single-channel feature dimension and the number of channels of the accelerometer signal feature map sample, respectively; let The gesture category label is also for ; Step 1.3: Construct a training set containing multimodal signal feature map samples. ,in, representing the training set of electromyographic signals and accelerometer signal training set The set of gesture category labels for the signal feature map samples, and Include Types of gestures; Step 2: Construct a multimodal progressive hierarchical fusion deep learning network, including three parallel sub-networks, namely... Single-modal CNN subnetwork, Single-modal CNN subnetworks and Hybrid-modal Transformer sub-networks; each sub-network includes: a feature extraction module and a pattern recognition module; Step 2.1: Place the first i Sample of electromyography features Input to In the single-modal CNN subnetwork, after processing by the feature extraction module, the dimensionality-reduced result is obtained. Single-modal features ; The dimensionality reduction Single-modal features enter After processing in the pattern recognition module of the single-modal CNN subnetwork, the output is... Classification result vector of a single-modal CNN subnetwork ; Step 2.2: Place the first i A sample of accelerometer signal features enter In the single-modal CNN subnetwork, after processing by the feature extraction module, the output is dimensionality-reduced. Single-modal features ; The dimensionality reduction Single-modal features enter After processing in the pattern recognition module of the single-modal CNN subnetwork, the output is... Classification result vector of a single-modal CNN subnetwork ; Step 2.3: From The feature extraction module in the hybrid modal Transformer subnetwork obtains multimodal hierarchical fusion features. After being processed by the corresponding pattern recognition module, the output is... Classification result vector of mixed-modal Transformer subnetwork ; Step 3: Construct a modality missing detection module and a decision-making layer post-processing mechanism; Step 3.1: The modal missing detection module calculates the first... i Sample of electromyographic signal features The statistical value, and determine the first i Sample of electromyographic signal features If the statistical value does not exist, proceed to step 4.2; otherwise, it indicates that... Modalities were missing and replaced with zero samples of the same size; Step 3.2, determine the first i Sample of electromyographic signal features Statistical values ​​and Whether the difference between the statistical mean values ​​exceeds the set threshold; if it does, it indicates that the first... i Sample of electromyographic signal features Anomalies are identified and replaced with zero samples of the same size. Step 3.3: Follow the procedures in steps 3.1 and 3.2 to process the first... i A sample of accelerometer signal features Process it; Step 3.4: Post-processing mechanism at the decision-making level: When the modality missing detection module determines that both modalities are present, the final classification result vector is: ; When the modal missing detection module determines that the ACC mode is missing, only the existing mode is missing. In modal analysis, the final classification result vector is: ; When the modality missing detection module determines When the modality is missing and only the ACC modality exists, the final classification result vector is: ; Step 4, Network Training: Step 4.1: Normalize the classification result vector of any sub-network using the sofmax function to obtain a normalized classification result vector, which is used to construct the cross-entropy loss of the corresponding sub-network. Cross-entropy loss of a single-modal CNN subnetwork , Cross-entropy loss of a single-modal CNN subnetwork Cross-entropy loss of Transformer subnetworks that integrate hierarchical features ; Step 4.2, using Classification result vector of a single-modal CNN subnetwork The classification result vector of the Transformer subnetwork with fused hierarchical features Construct the Single-modal CNN subnetwork and KL divergence loss function between mixed-modal Transformer subnetworks ; use Classification result vector of a single-modal CNN subnetwork The classification result vector of the Transformer sub-network with fused hierarchical features Build Single-modal CNN subnetwork and KL divergence loss function between mixed-modal Transformer subnetworks This aligns the distribution of classification results for single-modal signals with the distribution of classification results for multimodal signals. Step 4.3, based on the training set The Adam method is used to train a multimodal hybrid fusion model, and the total loss function is calculated. = until the total loss function The process continues until convergence, resulting in a well-trained natural gesture classification model for gesture recognition of any multimodal gesture data.

2. The multimodal progressive hierarchical fusion method for natural gesture recognition according to claim 1, characterized in that, The Single-modal CNN subnetwork, The feature extraction module in a single-modal CNN subnetwork includes: a dimensionality-upgrading unit, a single-modal hierarchical feature fusion unit, and a dimensionality-reduction unit; The dimensional upgrading unit includes A convolutional layer with a kernel size of 1×1; The single-modal hierarchical feature fusion unit includes: Level-based feature extraction network and single-modality level-based feature fusion layer; Each level in the multi-level feature extraction network includes, in sequence: The size of each convolutional kernel is The convolutional layer, a batch normalization layer, and a ReLU activation function layer; the single-modality hierarchical feature fusion layer is used to... The single-modal hierarchical features output from each level in the hierarchical feature extraction network are spliced ​​and fused according to the channel dimension; The dimensionality reduction unit includes, in sequence: The system consists of a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function layer, and a dropout layer.

3. The multimodal progressive hierarchical fusion method for natural gesture recognition according to claim 2, characterized in that, The The feature extraction module of the hybrid modal Transformer subnetwork includes: A Transformer network that fuses hierarchical features; the Transformer network consists of an input embedding layer and a feature encoder; the input embedding layer is... Modality Modal and - The hybrid modalities are each equipped with the same fully connected layer; the feature encoder includes a layer normalization layer, a multi-head attention unit, a layer normalization layer, and a feedforward unit; the feedforward unit includes a fully connected layer, a GELU activation function layer, a fully connected layer, and a Dropout layer.

4. The multimodal progressive hierarchical fusion method for natural gesture recognition according to claim 3, characterized in that: The pattern recognition module of each sub-network is composed of It consists of several fully connected layer sub-modules; wherein, the first fully connected layer sub-module includes, in sequence: A fully connected layer with 1 neuron, a batch normalization layer, a ReLU activation function layer, and a dropout layer; The fully connected layer modules sequentially include: A fully connected layer with 10 neurons, a batch normalization layer, and a ReLU activation function layer. ;No. Each fully connected layer module contains A fully connected layer of neurons.

5. The multimodal progressive hierarchical fusion method for natural gesture recognition according to claim 4, characterized in that: The The dimension-upgrading unit of the feature extraction module in a single-modal CNN subnetwork for the 1st generation i Sample of electromyography features After processing, the upgraded features are output. Then it is input into the single-modality hierarchical feature fusion unit, and sequentially passes through... The processing of the multi-stage feature extraction network is based on the output of each stage. Single-modal hierarchical features ;in, express The Hierarchical features; The The unimodal hierarchical feature fusion layer in the unimodal CNN subnetwork will combine the outputs of each level Single-modal hierarchical features By splicing and merging according to the channels, we obtain Hierarchical fusion features ; The The dimensionality reduction unit in the single-modal CNN subnetwork fuses features hierarchically. Dimensionality reduction is performed to obtain the dimensionality-reduced result. Single-modal features .

6. The multimodal progressive hierarchical fusion method for natural gesture recognition according to claim 5, characterized in that: The The dimension-upgrading unit of the feature extraction module in a single-modal CNN subnetwork for the 1st generation i Sample of electromyography features After processing, the upgraded features are output. Then it is input into the single-modality hierarchical feature fusion unit, and sequentially passes through... The processing of the multi-stage feature extraction network is based on the output of each stage. Single-modal hierarchical features ;in, express The Hierarchical features; The The unimodal hierarchical feature fusion layer in the unimodal CNN subnetwork will combine the outputs of each level Single-modal hierarchical features By splicing and merging according to the channels, we obtain Hierarchical fusion features ; The The dimensionality reduction unit in the single-modal CNN subnetwork fuses features hierarchically. Dimensionality reduction is performed to obtain the dimensionality-reduced result. Single-modal features .

7. The multimodal progressive hierarchical fusion method for natural gesture recognition according to claim 6, characterized in that: The first i Sample of electromyography features Features after dimensionality enhancement and the i A sample of accelerometer signal features Features after dimensionality enhancement The initial fused multimodal features are obtained after addition. ; When k=1, the initial fused features , k-th level features and k-th level features Input to In the hybrid modal Transformer subnetwork, and after processing by the Transformer network that fuses features at the k-th level, the multimodal hierarchical fused features after fusing features at the k-th level are obtained. ; When k=2,3,…,K, the level fusion feature is obtained by fusing the features of the (k-1)th level. , The Hierarchical features and k-th level features Enter the number After processing by a Transformer network that fuses the hierarchical features at each level, the multimodal hierarchical fused features after fusing the k-th level features are obtained. Finally, by the first The Transformer network outputs the multimodal hierarchical fused feature after fusing the features of the Kth level. ; The multimodal hierarchical fusion features enter In the pattern recognition module of the hybrid modal Transformer subnetwork, and after... After processing by each fully connected layer module, the output is... Classification result vector of mixed-modal Transformer subnetwork .

8. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing any of the multimodal progressive hierarchical fusion methods of claims 1-7, and the processor is configured to execute the programs stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the multimodal progressive hierarchical fusion method according to any one of claims 1-7.

Images