A multi-modal hybrid fusion method for natural gesture recognition and application thereof

Abstract

The application discloses a multi-modal mixed fusion method for natural gesture recognition and application thereof, and the method comprises the following steps: 1, acquiring multi-modal data, pre-processing and feature extraction of multi-modal signals; 2, constructing a multi-modal signal mixed fusion model, combining pre-fusion, multi-scale hierarchical feature fusion and post-fusion to enable the model to simultaneously learn specific information of a single mode and interaction information between multi-modes; 3, designing a multi-scale space attention module, fully mining key information of input signals by fusing multi-scale hierarchical features; 4, designing a metric learning loss function, enhancing the discrimination ability of the model to similar gestures; and 5, training the multi-modal mixed fusion model to obtain an optimal natural gesture recognition model. The application can improve the robustness of natural gesture recognition, especially increase the recognition accuracy of the model to a large number of gestures and similar gestures, thereby promoting the popularization and application of a gesture-based human-computer interaction system.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of biological signal processing, and particularly relates to a multi-modal mixed fusion method for natural gesture recognition. BACKGROUND

[0002] Gesture recognition using electromyography (EMG) is an intuitive and feasible human-computer interaction scheme. Surface electromyography (sEMG) accompanied by muscle contraction of the human body reflects the movement joint activity intensity and flexion state, as well as the hand movement, shape, position and orientation during the completion of the gesture, and has a unique advantage in gesture recognition, and therefore can be used to represent the motion state or motion intention of the user.

[0003] Electromyographic control is a technology for using surface electromyography to convert movement or intention into a command for manipulating a prosthesis or other external robotic device. Multi-degree-of-freedom electromyographic pattern recognition is considered a milestone technology. Due to the dexterity of the upper limb or hand, this capability is necessary to achieve the control required for the upper limb or hand. However, electromyographic signals have natural non-stationarity and weakness, and the signal quality will decrease due to interference such as sweating, electrode offset and motion noise; in particular, the electromyographic signals of amputated patients at the amputated site are even weaker and have low signal-to-noise ratio, and similar gestures are difficult to distinguish in natural gesture recognition. Thus, the performance of gesture recognition based on electromyography is severely degraded, affecting the experience of gesture interaction, and even causing safety problems such as misoperation.

[0004] Using multi-modal signal fusion to solve the problem of performance degradation of electromyographic pattern recognition is a reliable solution. There are generally two traditional multi-modal fusion strategies: early fusion or late fusion. Early fusion refers to first fusing two signals according to certain rules, and then inputting the signals into a classifier to obtain gesture classification results. This strategy can fully utilize the interaction information between the two signals, but does not fully utilize the single-modal information. Late fusion refers to first training two classifiers using two signals respectively, and then fusing the classification results of the two classifiers according to certain rules to obtain the final classification result. This strategy can fully utilize the single-modal signal information, but ignores the interaction information between the modalities. In addition, existing gesture recognition methods lack special processing for extremely similar gestures, such as natural gestures when grabbing different shaped objects, which are extremely similar, resulting in poor classification effect of similar gestures by the model. SUMMARY

[0005] This invention addresses the shortcomings of existing technologies by proposing a multimodal hybrid fusion method for natural gesture recognition and its application. The method combines early-stage fusion, multi-scale hierarchical feature fusion, and late-stage fusion to enable the model to simultaneously learn specific information of a single modality and interactive information between multiple modalities. This allows for the acquisition of more comprehensive and useful information from multimodal signals for gesture classification. Furthermore, the introduction of metric learning enhances the model's ability to distinguish similar gestures, thereby improving the model's recognition accuracy for a large number of gestures and similar gestures, and strengthening the robustness of the natural gesture recognition system.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The multimodal hybrid fusion method for natural gesture recognition of the present invention is characterized by comprising the following steps:

[0008] Step 1: Obtain the multimodal gesture dataset X = {sEMG, ACC} and its gesture category label set Y0, and preprocess and extract features from the multimodal gesture dataset X to construct a training set containing signal feature map samples; where sEMG represents the surface electromyography signal with sparse channels and ACC represents the accelerometer signal.

[0009] Step 1.1, Data Preprocessing:

[0010] The surface electromyography (sEMG) signal and the accelerometer signal (ACC) are filtered, normalized, and subjected to active segment sliding window segmentation to obtain an EMG signal sample set S containing N1 original EMG signal samples. raw and an accelerometer signal sample set A containing N1 accelerometer signal samples. raw ;

[0011] Step 1.2, Feature Extraction:

[0012] The electromyographic signal sample set S was extracted using time-frequency domain methods. raw The time-frequency domain feature map of each original electromyography (EMG) signal sample is used to obtain an EMG signal training set S = {S1} containing N1 EMG signal feature map samples. i |i=1,2,…,N1};where, Let S represent any i-th electromyography (EMG) signal feature map sample; d1 and ch1 represent the single-channel feature dimension and channel number of the EMG signal feature map sample, respectively; let S i The gesture category label is Y i ;

[0013] Accelerometer signal sample set A is extracted using time-domain methods. raw The time-domain feature map of each accelerometer signal sample is used to obtain the accelerometer signal training set A = {A}, which contains N1 accelerometer signal feature map samples.i i = 1, 2, …, N1}; wherein, represents an arbitrary i-th accelerometer signal feature map sample; d2 and ch2 represent the single channel feature dimension and the channel number of the accelerometer signal feature map sample respectively; let A i gesture class label of the gesture recognition model is Y i ;

[0014] Step 1.3, constructing a training set {S, A, Y} containing multi-modal signal feature map samples, wherein Y represents the label set of the signal feature map samples in the electromyographic signal training set S and the accelerometer signal training set A, and Y contains L gesture classes;

[0015] Step 2, constructing a multi-modal hybrid fusion model, including three parallel network branches and a decision layer fusion module; each network branch includes a feature extraction subnetwork and a pattern recognition subnetwork;

[0016] The feature extraction subnetwork includes a dimension increasing unit, a hierarchical feature fusion unit and a dimension reducing unit;

[0017] The hierarchical feature fusion unit includes a K-level residual network and a multi-scale attention module;

[0018] The dimension increasing unit successively includes a batch normalization layer, λ1 convolution layers with a convolution kernel size of 1×1, a batch normalization layer and a ReLu activation function layer;

[0019] Each level in the K-level residual network successively includes a batch normalization layer, λ'2 convolution layers with a convolution kernel size of 1×1, a batch normalization layer, μ convolution layers with a convolution kernel size of k×k, a batch normalization layer, λ2 convolution layers with a convolution kernel size of 1×1, a batch normalization layer and a ReLu activation function layer;

[0020] The multi-scale attention module includes K scales of spatial attention units and a fusion unit;

[0021] Each level in the K-level residual network corresponds to a spatial attention unit of each scale; each spatial attention unit of each scale successively includes a max-pooling layer, an average-pooling layer, a convolution layer and a sigmod activation function layer;

[0022] The dimension reducing unit successively includes λ3 convolution layers with a convolution kernel size of 1×1, a batch normalization layer, a ReLu activation function layer and a dropout layer;

[0023] The mode recognition sub-network is composed of F full connection layer modules; wherein, the first full connection layer module sequentially comprises: a full connection layer containing w1 neurons, a batch normalization layer, a ReLu activation function layer and a dropout layer; the dth full connection layer module sequentially comprises: a full connection layer containing w d neurons, a batch normalization layer and a ReLu activation function layer, d∈{2,3,…,F-1}; the Fth full connection layer module is a full connection layer containing w F neurons;

[0024] Step 2.1: input the i th electromyographic feature map sample S i into the first parallel network branch, and after processing by the dimension increasing unit in the feature extraction sub-network, output the dimension-increased feature , and then input into the hierarchical feature fusion unit, sequentially pass through the K-level residual network processing, and output the residual feature from each level. , wherein, represents the output residual feature of the kth level residual network.

[0025] The residual feature output by each level in the K-level residual network corresponds to input each kind of scale spatial attention unit respectively, so as to output the spatial attention weight , wherein, represents the spatial attention weight output by the kth kind of scale spatial attention unit; the spatial attention unit carries out Hadamard product on and to obtain K hierarchical weighted hierarchical features , wherein, represents the kth level weighted hierarchical feature obtained by the kth level residual feature and the kth kind of scale spatial attention weight .

[0026] The fusion unit splices and fuses the K levels of weighted features to obtain the hierarchical fusion feature

[0027] The dimension reduction unit carries out dimension reduction processing on the hierarchical fusion feature to obtain the dimension-reduced feature

[0028] The dimension-reduced feature is input into the mode recognition sub-network, and after processing by the first full connection layer module, the first full connection layer feature is output. The F-1th fully connected layer feature is outputted after the F-1th fully connected layer module and the Fth fully connected layer module are sequentially processed The F-1th fully connected layer feature is outputted after the F-1th fully connected layer module and the Fth fully connected layer module are sequentially processed The classification result of the first branch of the parallel network is outputted after the Fth fully connected layer module is processed

[0029] Step 2.2: the i th accelerometer signal feature map sample A i The F-1th fully connected layer feature is outputted after the F-1th fully connected layer module and the Fth fully connected layer module are sequentially processed and the classification result of the second branch of the parallel network

[0030] Step 2.3: the i th electromyography feature map sample S i and the i th accelerometer signal feature map sample A i The spliced graph SA i The F-1th fully connected layer feature is outputted after the F-1th fully connected layer module and the Fth fully connected layer module are sequentially processed and the classification result of the third branch of the parallel network

[0031] Step 2.4, the decision layer fusion module is to fuse the branch classification results outputted by the three parallel network branches After the elements are added, the classification result R after decision fusion is obtained i ;

[0032] Step 3, network training:

[0033] Step 3.1, define and randomly initialize the center matrix of the gesture category wherein, w F-1 represents the dimension of the F-1th fully connected layer feature; the center loss of any one parallel network branch is constructed by using formula (1)

[0034]

[0035] In formula (1), a represents the proportion of the center loss in the total loss, x i represents the F-1th fully connected layer feature outputted in any one parallel network branch, represents the label Y iThe class center vector of the corresponding gesture class in the center matrix Z of the gesture class;

[0036] The center cosine similarity loss of any one parallel network branch is constructed by formula (2)

[0037]

[0038] In formula (2), beta represents the weight of the center cosine similarity loss in the total loss, c p and c q respectively represent the class center vector of the pth gesture class and the qth gesture class in the center matrix Z of the gesture class;

[0039] Step 3.2, the center loss of any one parallel network branch And the corresponding center cosine similarity loss Are added to obtain the metric learning loss of any one parallel network branch

[0040] The classification results of any one parallel network branch are normalized by using the sofmax function to obtain the normalized classification results, which are used to construct the cross entropy loss of the corresponding parallel network branch

[0041] Step 3.3, the classification results R i After decision fusion are normalized by using the sofmax function to obtain the normalized classification results, which are used to construct the cross entropy loss of the decision layer fusion module

[0042] Step 3.4, the metric learning loss, the cross entropy loss of the three parallel network branches and the cross entropy loss of the decision layer fusion module Jointly constitute the total loss function

[0043] Step 3.5, based on the training set {S, A, Y}, the multi-modal hybrid fusion model is trained by using the Adam method, and the total loss function Until the total loss function Converges, thereby obtaining the trained natural gesture classification model for gesture recognition of any one multi-modal gesture data.

[0044] The electronic device of the present application comprises a memory and a processor, characterized in that the memory is used to store a program supporting the processor to execute the multi-modal hybrid fusion method, and the processor is configured to execute the program stored in the memory.

[0045] The application is a computer readable storage medium, and a computer program is stored on the computer readable storage medium, wherein the computer program performs the steps of the multi-modal hybrid fusion method when executed by a processor.

[0046] Compared with the prior art, the application has the following beneficial effects:

[0047] 1. The application designs a multi-modal hybrid fusion model framework, which combines pre-fusion, multi-scale hierarchical feature fusion and post-fusion, so that the model can learn specific information of a single mode and interaction information between multi-modal, ensure that multi-modal signals can play a complementary function in the model without mutual inhibition, thereby improving the accuracy of gesture classification.

[0048] 2. The application embeds a multi-scale spatial attention fusion module in the designed multi-modal hybrid fusion model, which can weight and fuse multi-scale features of different levels of the neural network, thereby fully mining the key information of the input signal and enhancing the feature expression ability of the neural network, increasing the accuracy of gesture classification.

[0049] 3. The application introduces a metric learning loss function in the designed multi-modal hybrid fusion model, which can optimize the neural network parameters through the loss function, increase the aggregation of similar gestures and the distinction of different gestures of the neural network, and thereby make the model more accurately complete similar gesture classification. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The data preprocessing flowchart of the application;

[0051] Figure 2 The multi-modal hybrid fusion neural network model diagram of the application;

[0052] Figure 3 The multi-scale spatial attention module schematic diagram of the application;

[0053] Figure 4 The neural network model training and testing scheme schematic diagram of the application. DETAILED DESCRIPTION

[0054] In this embodiment, a multi-modal hybrid fusion method for natural gesture recognition is a multi-modal hybrid fusion method for natural gesture recognition based on multi-scale spatial attention and metric learning. The method combines pre-fusion, multi-scale hierarchical feature fusion and post-fusion to enable the model to simultaneously learn specific information of a single modality and interaction information between multiple modalities, so as to obtain more comprehensive useful information from multi-modal signals for gesture classification. The particularly introduced metric learning can increase the aggregation of similar gestures and the discrimination of different gestures by the neural network, thereby enabling the model to more accurately complete similar gesture classification. The related invention content can improve the robustness of the natural gesture recognition system, especially increase the recognition accuracy of the model for a large number of gestures and similar gestures, thereby promoting the popularization and application of the gesture-based human-computer interaction system. Specifically, the multi-modal hybrid fusion method comprises the following steps:

[0055] Step 1, obtain a multi-modal gesture dataset {X, Y0} = {sEMG, ACC, Y0}, wherein sEMG represents a surface electromyography signal of a sparse channel, ACC represents an accelerometer signal, and Y0 represents a gesture category label set of the multi-modal data {sEMG, ACC}; and pre-process and extract features from the multi-modal data {sEMG, ACC}; finally, construct a training set containing signal feature map samples;

[0056] Step 1.1, data preprocessing:

[0057] Filter, normalize and active segment sliding window segmentation are performed on the surface electromyography signal sEMG and the accelerometer signal ACC, respectively, to obtain an electromyography signal sample set S containing N1 original electromyography signal samples raw and an accelerometer signal sample set A containing N1 accelerometer signal samples raw Each sample obtains a corresponding category label;

[0058] In a specific implementation, the surface electromyography signal is a 12-channel sparse channel electromyography, and the accelerometer signal is a 36-channel signal, that is, each channel of the 12-channel electromyography channel has corresponding X, Y and Z three-axis accelerometer signals;

[0059] The data preprocessing flowchart is shown in Figure 1 The filter is a 4th order Butterworth filter used for 10-500Hz band-pass filtering and 50Hz power frequency notch filtering of the surface electromyography signal; and a 4th order Butterworth filter is used for 100Hz low-pass filtering of the surface electromyography signal;

[0060] The normalization is to first calculate the mean and variance of each channel of the training set signal, and then normalize the training set and test set data using the mean and variance of each channel of the training set, so that the mean of each channel data is 0 and the variance is 1;

[0061] The active segment sliding window segmentation uses the sliding window method to segment the surface electromyography signal and accelerometer signal separately. In the specific implementation, the sliding window size is 200ms and the step size is 10ms.

[0062] Step 1.2, Feature Extraction:

[0063] The electromyographic signal sample set S was extracted using time-frequency domain methods. raw The time-frequency domain feature map of each original electromyography (EMG) signal sample is used to obtain an EMG signal training set S = {S1} containing N1 EMG feature map samples. i |i=1,2,…,N1};where, Let S represent any i-th electromyography (EMG) feature map sample; d1 and ch1 represent the single-channel feature dimension and channel number of the EMG feature map sample, respectively; let S i The gesture category label is Y i ;

[0064] In specific implementation, the electromyographic features include: Marginal of Discrete Wavelet Transform (mDWT), Histogram (HIST), Root Mean Square (RMS), Zero Crossing (ZC), Slope Sign Change (SSC), Waveform Length (WL), Mean Absolute Value (MAV), Mean Absolute Value Slope (MAVS), Willison Amplitude (WAMP), Autoregressive Coefficients (ARC), Mean Frequency (MNF), and Power Spectrum Ratio (PSR). Features are extracted for each surface electromyographic signal sample by channel, and the obtained features are concatenated column by column to form an electromyographic signal sample containing electromyographic feature maps.

[0065] Accelerometer signal sample set A is extracted using time-domain methods. raw The time-domain feature map of each accelerometer signal sample is used to obtain the accelerometer signal training set A = {A}, which contains N1 accelerometer signal feature map samples. i |i=1,2,…,N1};where, represents an arbitrary i-th accelerometer signal feature map sample; d2 and ch2 represent the single channel feature dimension and the number of channels of the accelerometer signal feature map sample, respectively; let A i the gesture category label is Y i ;

[0066] In specific implementation, the specific accelerometer signal features include: mean value (Mean), variance (Variance, VAR), root mean square (Root Mean Square, RMS), waveform length value (Waveform Length, WL), mean absolute value (Mean Absolute Value, MAV), mean absolute value slope (Mean Absolute Value Slope, MAVS). For each accelerometer signal sample, the features are calculated by channel, and the obtained features are spliced by column to form an accelerometer signal sample containing an accelerometer signal feature map;

[0067] Step 1.3, constructing a training set {S, A, Y} containing multi-modal signal feature map samples, wherein S represents an electromyography signal training set containing electromyography feature map samples, A represents an accelerometer signal training set containing accelerometer signal feature map samples, and Y represents a label set of the samples;

[0068] In specific implementation, each surface electromyography signal sample has a corresponding accelerometer signal sample and label, and the three are strictly corresponding;

[0069] Step 2, constructing a multi-modal hybrid fusion model, the overall structure is as shown in Figure 2 , including: three parallel network branches and a decision layer fusion module; each network branch includes: a feature extraction sub-network and a pattern recognition sub-network;

[0070] The feature extraction sub-network includes: a dimension increasing unit, a hierarchical feature fusion unit and a dimension reducing unit;

[0071] The hierarchical feature fusion unit includes: a K-level residual network and a multi-scale attention module;

[0072] For example, the number of levels of the residual network is set to 3;

[0073] The dimension increasing unit includes: a batch normalization layer, λ1 convolution layers with a convolution kernel size of 1×1, a batch normalization layer and a ReLu activation function layer in sequence; for example, the number of filters of the convolution layer of the dimension increasing unit is set to λ1=64;

[0074] Each level in the K-level residual network comprises, in sequence, a batch normalization layer, λ'2 convolution layers with a 1x1 convolution kernel, a batch normalization layer, μ convolution layers with a kxk convolution kernel, a batch normalization layer, λ2 convolution layers with a 1x1 convolution kernel, a batch normalization layer, and a ReLu activation function layer; for example, λ'2, μ, and λ2 are each set to 64; the convolution kernel sizes of the kxk convolution layers in the 3-level residual network are set to 3x3, 5x5, and 3x3, respectively;

[0075] The multi-scale attention module comprises K scales of spatial attention units and a fusion unit; a schematic diagram of the multi-scale spatial attention module is as shown in Figure 3

[0076] For example, the multi-scale attention module comprises K=3 scales of spatial attention units, and the scale sizes represent the sizes of the spatial convolution kernels, which are 7x7, 5x5, and 3x3 in sequence;

[0077] Each level in the K-level residual network corresponds to a spatial attention unit of each scale; each spatial attention unit of each scale comprises, in sequence, a max-pooling layer, an average-pooling layer, a convolution layer, and a sigmod activation function layer;

[0078] The dimension reduction unit comprises, in sequence, λ3 convolution layers with a 1x1 convolution kernel, a batch normalization layer, a ReLu activation function layer, and a dropout layer;

[0079] For example, the number of convolution filters of the dimension reduction unit is λ3=64;

[0080] The pattern recognition subnetwork comprises F fully connected layer modules; the first fully connected layer module comprises, in sequence, a fully connected layer with w1 neurons, a batch normalization layer, a ReLu activation function layer, and a dropout layer; the dth fully connected layer module comprises, in sequence, a fully connected layer with w d neurons, a batch normalization layer, and a ReLu activation function layer, where d∈{2,3,…,F-1}; the Fth fully connected layer module is a fully connected layer with w F neurons;

[0081] For example, the number of fully connected layer modules of the pattern recognition subnetwork is F=3; the number of neurons of the first fully connected layer and the second fully connected layer are each set to 512; the number of neurons of the last fully connected layer is set to 50 or 41, which is determined according to the number of gesture categories in a specific multi-modal data set;

[0082] Step 2.1: input the ith electromyographic feature map sample S i into the first parallel network branch, and output the features after dimension increase after processing by the dimension increase unit in the feature extraction subnetwork​ In the post-input hierarchical feature fusion unit, the K-level residual network is processed in turn, and the residual feature output by each level is output wherein, represents the residual feature output by the kth level residual network;

[0083] For example, the first branch will obtain the 3-level residual feature;

[0084] The residual feature output by each level in the K-level residual network In the spatial attention unit inputting each scale respectively, the spatial attention weight is output wherein, represents the spatial attention weight output by the kth scale spatial attention unit; the spatial attention unit will and The K-level weighted hierarchical features are obtained by performing Hadamard product on the corresponding wherein, represents the kth level weighted hierarchical feature obtained by the kth level residual feature and the spatial attention weight of the kth scale; For example, the first branch will obtain the 3-level weighted hierarchical feature, and the channel dimension of each level feature is 64;

[0085] The K-level weighted hierarchical features are fused by channel splicing to obtain the hierarchical fusion feature For example, the first branch will obtain a hierarchical fusion feature with a channel dimension of 192

[0086] The hierarchical fusion feature is processed by the dimension reduction unit to obtain the dimension-reduced feature For example, the dimension-reduced feature has a channel dimension of 64;

[0087] The dimension-reduced feature is input into the pattern recognition sub-network and processed by the first fully connected layer module to output the first fully connected layer feature The first fully connected layer feature is input into the second fully connected layer module and processed by F-2 fully connected layer modules in turn to output the F-1 fully connected layer feature The F-1 fully connected layer feature is input into the Fth fully connected layer module to output the classification result of the first branch of the parallel network

[0088] For example, the F-1th fully connected layer feature of the first branch The dimension is 1×512, and the classification result of the first branch is... The dimension is 1×50;

[0089] Step 2.2: Transfer the feature map sample A of the i-th accelerometer signal i The input is fed into the second parallel network branch, and after being processed by the feature extraction subnetwork and the pattern recognition subnetwork in sequence, the F-1th fully connected layer module and the Fth fully connected layer module respectively output the F-1th fully connected layer features. Classification results of the second branch of the parallel network

[0090] For example, the F-1th fully connected layer feature of the second branch The dimension is 1×512, and the classification result of the second branch is... The dimension is 1×50;

[0091] Step 2.3: Transfer the i-th electromyography feature map sample S i and the feature map sample A of the i-th accelerometer signal i SA splicing image i The input is fed into the third parallel network branch, and after being processed by the feature extraction subnetwork and the pattern recognition subnetwork in sequence, the F-1th fully connected layer module and the Fth fully connected layer module respectively output the F-1th fully connected layer features. Classification results of the third branch of the parallel network

[0092] For example, the F-1th fully connected layer feature of the third branch The dimension is 1×512, and the classification result of the third branch is... The dimension is 1×50;

[0093] Step 2.4, the decision-level fusion module, integrates the classification results of each branch output from the three parallel network branches. After adding the elements, we obtain the classification result R after decision fusion. i For example, the classification result R of the third branch i The dimension is 1×50;

[0094] Step 3, Network Training:

[0095] Step 3.1: Set the parameters of the trainable gesture category center matrix. Where L represents the number of gesture categories in the multimodal gesture dataset {X,Y}, w F-1dimension of the F-1 full connection layer feature, each row of the center matrix Z can be regarded as a center vector of a gesture; the center loss of any parallel network branch is constructed by using formula (1)

[0096]

[0097] In formula (1), a represents the weight of the center loss in the total loss, N represents the number of samples in a batch during network training, x i represents the F-1 full connection layer feature output in any parallel network branch, Y i is the label of the input sample S i represents the class Y i in the gesture class center matrix Z.

[0098] For example, the gesture class center matrix parameter Z ∈ R 50×512 , a is set to 0.001, and the number of samples N in a batch during network training is 512.

[0099] The center cosine similarity loss of any parallel network branch is constructed by using formula (2)

[0100]

[0101] In formula (2), represents the center cosine similarity loss, represents the cosine distance between each center vector, the purpose is to punish the similarity of the class center and increase the distance between each class, in order to avoid the positive and negative offset of the cosine distance, so 1 is added on the basis of the cosine distance; β represents the weight of the center cosine similarity loss in the total loss, represents the label set of each batch of samples in the training data, c p and c q respectively represent the class center vectors of the classes p and q in the gesture class center matrix Z.

[0102] According to formula (1) and formula (2), the center loss of the first branch is and the center cosine similarity loss of the first branch is After being added, the metric learning loss of the first branch is obtained

[0103] The sofmax function is used to normalize the classification result of the first branch After the normalized classification result is obtained, it is sent to the cross entropy loss function, and the cross entropy loss of the first branch is obtained

[0104] ​Step 3.2, constructing the center loss of the second branch according to formula (1) constructing the center cosine similarity loss of the second branch according to formula (2) adding the two to obtain the metric learning loss of the second branch

[0105] using the sofmax function to obtain the classification result of the second branch normalizing to obtain the normalized classification result, and then inputting the normalized classification result into a cross-entropy loss function to obtain the cross-entropy loss of the second branch

[0106] constructing the center loss of the third branch according to formula (1) constructing the center cosine similarity loss of the second branch according to formula (2) adding the two to obtain the metric learning loss of the second branch

[0107] using the sofmax function to obtain the classification result of the second branch normalizing to obtain the normalized classification result, and then inputting the normalized classification result into a cross-entropy loss function to obtain the cross-entropy loss of the second branch

[0108] Step 3.3, using the sofmax function to obtain the classification result R after decision fusion i normalizing to obtain the normalized classification result, and then inputting the normalized classification result into a cross-entropy loss function to obtain the cross-entropy loss of the decision fusion classification result

[0109] Step 3.4, the metric learning loss of the three parallel network branches, the cross-entropy loss, and the cross-entropy loss of the strategy fusion module jointly constitute a total loss function

[0110] Step 3.5, training the multi-modal hybrid fusion model with the training set {S, A, Y} containing the multi-modal signal feature map sample prepared in step 1 to obtain the trained natural gesture classification model. The network model training and testing scheme is shown in Figure 4 .

[0111] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the multi-modal hybrid fusion method, and the processor is configured to execute the program stored in the memory.

[0112] In this embodiment, a computer readable storage medium stores a computer program on the computer readable storage medium, and the computer program is executed by a processor to perform the steps of the multi-modal hybrid fusion method.

[0113] In a specific implementation, the proposed neural network is implemented using the pytorch deep learning framework, the network is optimized and trained using the Adam optimizer, the number of samples per batch is set to 512, the neural network learning rate is [0.001, 0.0001, 0.00001], and 3 rounds, 4 rounds and 3 rounds of learning are performed at each learning rate.

[0114] The feasibility and effectiveness of the proposed model are verified through comparative experiments on three public datasets. The three public datasets are Ninapro DB2, DB3 and B7. The comparative experiments include gesture recognition on public datasets using the mixed fusion model, early fusion model and late fusion model proposed in the application, using gesture recognition accuracy as an evaluation index, defining the gesture recognition accuracy of each subject = the number of correctly classified samples / total sample number x 100%, the number of subjects participating in the experiment in the three public datasets is 40, 6 and 20 respectively, and the experimental results shown are the average gesture recognition accuracy on a single dataset, defined as average gesture recognition rate = the sum of gesture recognition accuracy of subjects participating in the experiment on the dataset / the number of subjects participating in the experiment on the dataset x 100%. In order to simplify the experiment, the early fusion model adopts the third branch of the mixed fusion model of the application, the late fusion model adopts the first and second branches and the fusion unit of the mixed fusion model of the application, and other experimental settings are consistent.

[0115] Table 1 Comparison of experimental results of the method of the application and two traditional methods on public datasets

[0116]

[0117] Table 2 Comparison of experimental results of the method of the application and the most advanced method on public datasets

[0118]

[0119] The experimental results are compared in Table 1, and the results show that the mixed fusion model proposed in the application obtains the highest performance. In order to further show the advancement of the application, the experimental results obtained by the method of the application are compared with the best experimental results of the currently published literature. Wei [1] et al. proposed multi-view CNN (MV-CNN) in the literature "Surface-electromyography-based gesture recognition by multi-view deep learning", which obtains the best performance of multi-modal gesture recognition on NinaproDB2, DB3 and DB7. It should be noted that the experimental data division method of the application is consistent with the division method adopted by MV-CNN. The experimental results of the comparison of the method of the application are shown in Table 2, and the results show that the method of the application obtains better performance.

Claims

1. A multimodal hybrid fusion method for natural gesture recognition, characterized in that, Includes the following steps: Step 1: Obtain the multimodal gesture dataset and its gesture category label set and on a multimodal gesture dataset Preprocessing and feature extraction are performed to construct a training set containing signal feature map samples; among which, This represents surface electromyography (EMG) signals in sparse channels. This indicates the accelerometer signal; Step 1.1, Data Preprocessing: surface electromyography signals and accelerometer signal Filtering, normalization, and active segment sliding window segmentation are performed separately to obtain the result containing A collection of original electromyography (EMG) signal samples. and contain A set of accelerometer signal samples ; Step 1.2, Feature Extraction: Extracting electromyographic signal sample sets using time-frequency domain methods. The time-frequency domain feature map of each original electromyography signal sample is obtained, containing Electromyography (EMG) signal training set of 100 EMG signal feature map samples ;in, This represents any i-th electromyographic signal feature map sample; and Let represent the single-channel feature dimension and the number of channels of the electromyographic signal feature map sample, respectively; let The gesture category label is ; Extracting accelerometer signal sample sets using time-domain methods The time-domain feature map of each accelerometer signal sample is obtained, containing Accelerometer signal training set of accelerometer signal feature map samples ;in, This represents any i-th accelerometer signal feature map sample; and Let represent the single-channel feature dimension and the number of channels of the accelerometer signal feature map sample, respectively; let The gesture category label is ; Step 1.3: Construct a training set containing multimodal signal feature map samples. ,in, representing the training set of electromyographic signals and accelerometer signal training set The label set of the signal feature map samples, and Include Types of gestures; Step 2: Construct a multimodal hybrid fusion model, including: three parallel network branches and a decision layer fusion module; each network branch includes: a feature extraction subnetwork and a pattern recognition subnetwork; The feature extraction subnetwork includes: a dimensionality-upgrading unit, a hierarchical feature fusion unit, and a dimensionality-reducing unit; The hierarchical feature fusion unit includes: Level residual networks and multi-scale attention modules; The dimensionality-upgrading unit includes, in sequence: a batch normalization layer, A convolutional layer with a kernel size of 1×1, a batch normalization layer, and a ReLU activation function layer; Each level in a hierarchical residual network consists of: a batch normalization layer, One convolutional layer with a kernel size of 1×1, and one batch normalization layer. The size of each convolutional kernel is A convolutional layer, a batch normalization layer, A convolutional layer with a kernel size of 1×1, a batch normalization layer, and a ReLU activation function layer; The multi-scale attention module includes: Spatial attention units and fusion units at various scales; Each level in the hierarchical residual network corresponds to a spatial attention unit of each scale; each spatial attention unit of each scale consists of a max pooling layer, an average pooling layer, a convolutional layer, and a sigmoid activation function layer in sequence. The dimensionality reduction unit includes, in sequence: A convolutional layer with a kernel size of 1×1, a batch normalization layer, a ReLU activation function layer, and a dropout layer; The pattern recognition subnetwork is composed of It consists of several fully connected layer modules; wherein, the first fully connected layer module sequentially includes: A fully connected layer with 1 neuron, a batch normalization layer, a ReLU activation function layer, and a dropout layer; The fully connected layer modules sequentially include: A fully connected layer with 10 neurons, a batch normalization layer, and a ReLU activation function layer. ;No. Each fully connected layer module contains A fully connected layer of 10 neurons; Step 2.1: Sample the i-th electromyography feature map The input is fed into the first parallel network branch and processed by the upscaling unit in the feature extraction subnetwork, outputting the upscaled features. Then it is input into the hierarchical feature fusion unit, and passes through sequentially. The residual network is processed in stages, and the residual features are output from each stage. ;in, Indicates the first The residual characteristics of the output of a multi-level residual network; Residual characteristics of each stage output in a multi-stage residual network The spatial attention weights are then output by being input into spatial attention units of each scale. ;in, This represents the spatial attention weights output by the spatial attention unit at the k-th scale; the spatial attention unit will... and Performing the corresponding Hadamard product, we get Hierarchical features after weighting at each level ;in, Indicates by the first Level residual characteristics Spatial attention weights at the k-th scale The weighted hierarchical features obtained at the k-th layer; The fusion unit will Weighted features at each level By splicing and blending according to channels, a hierarchical fusion feature is obtained. ; The dimensionality reduction unit will fuse features at different levels. Dimensionality reduction is performed to obtain the reduced features. ; The dimensionality-reduced features The input is processed by the first fully connected layer module in the pattern recognition subnetwork, and then the first fully connected layer features are output. Features of the first fully connected layer The input is entered into the second fully connected layer module, and then sequentially passes through... After processing by the first fully connected layer module, the output is the first... Fully connected layer features The first Fully connected layer features Enter the first After processing by each fully connected layer module, the classification result of the first branch of the parallel network is output. ; Step 2.2: Sample the feature map of the i-th accelerometer signal. The input is fed into the second parallel network branch, and after being processed sequentially by the feature extraction subnetwork and the pattern recognition subnetwork, it is then processed by the first... The first fully connected layer module and the first The fully connected layer modules output the first... Fully connected layer features Classification results of the second branch of the parallel network ; Step 2.3: Sample the i-th electromyography feature map and the feature map sample of the i-th accelerometer signal splicing image The input is fed into the third parallel network branch, and after being processed sequentially by the feature extraction subnetwork and the pattern recognition subnetwork, it is then processed by the... The first fully connected layer module and the first The fully connected layer modules output the first... Fully connected layer features Classification results of the third branch of the parallel network ; Step 2.4: The decision-level fusion module integrates the classification results of each branch output from the three parallel network branches. , , After adding the elements together, we obtain the classification result after decision fusion. ; Step 3, Network Training: Step 3.1: Define and randomly initialize the center matrix of gesture categories. ,in, Indicates the first The dimension of the fully connected layer features; the central loss of any parallel network branch is constructed using equation (1). : (1) In equation (1), This indicates the weight of the central loss relative to the total loss. This represents the output of any parallel network branch. Features of fully connected layers Indicates label The corresponding gesture category in the center matrix of gesture categories The category center vector in; Construct the center cosine similarity loss for any parallel network branch using equation (2). : (2) In equation (2), The weight representing the proportion of the center cosine similarity loss to the total loss. and They represent the first Types of gestures and the first The gesture categories are located in the center matrix of the gesture categories. The category center vector in; Step 3.2: Central loss of any parallel network branch and its corresponding central cosine similarity loss Add them together to obtain the metric learning loss for any parallel network branch. ; The classification results of any parallel network branch are normalized using the sofmax function to obtain normalized classification results, which are then used to construct the cross-entropy loss of the corresponding parallel network branch. ; Step 3.3: Use the sofmax function to evaluate the classification results after decision fusion. Normalization is performed to obtain normalized classification results, which are then used to construct the cross-entropy loss of the decision-level fusion module. ; Step 3.4: Measure the learning loss, cross-entropy loss of the three parallel network branches, and the cross-entropy loss of the policy layer fusion module. Together they constitute the total loss function ; Step 3.5, based on the training set The multimodal hybrid fusion model is trained using the Adam method, and the total loss function is calculated. until the total loss function The process continues until convergence, resulting in a well-trained natural gesture classification model for gesture recognition of any multimodal gesture data.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the multimodal hybrid fusion method of claim 1, wherein the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the multimodal hybrid fusion method of claim 1.

Images