Action-free gesture recognition method, device and system based on surface electromyogram signals

Abstract

The invention relates to an action-free gesture recognition method, device and system based on surface electromyogram signals. The method comprises the following steps: acquiring preprocessed electromyographic signal samples in different motion states; performing zero filling and preliminary feature extraction on the preprocessed electromyographic signal sample to obtain a zero filling feature map and a first feature map, performing stacking distillation on the zero filling feature map to obtain a second feature map, splicing the first feature map and the second feature map, and performing multi-scale convolution to obtain a multi-scale time sequence feature map; performing multi-sub-attention-based multi-channel relation extraction on the time domain feature of each channel in the time sequence feature map on each scale, and performing deep convolution on a multi-channel relation extraction result to obtain a corresponding spatial-temporal feature map; and fusing the spatial-temporal feature maps of all the scales to output a fusion result, and performing non-action gesture recognition according to the fusion result to obtain a recognition result. By adopting the method, the recognition accuracy of the non-action gesture electromyographic signals in different motion states can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer human-computer interaction technology, and in particular to a method, device and system for motionless gesture recognition based on surface electromyography signals. Background Art

[0002] In meetings, negotiations or team collaboration, traditional means of information transmission such as verbal communication and gestures may lead to the exposure of intentions, thus affecting the final outcome.

[0003] In recent years, with the continuous emergence of emerging technologies, new information interaction technologies have also been continuously iterated. These interaction technologies have great potential in covert implementation, such as gesture recognition based on surface electromyography signals, silent voice interaction based on lip reading recognition, interaction based on line of sight detection, and mind interaction based on brain-computer interface technology.

[0004] However, these covert interaction methods all have some shortcomings: lip reading, line of sight detection, and brain-computer interface methods all require personnel to carry auxiliary equipment to complete information transmission, and the carrying of auxiliary equipment will hinder the movement of personnel, thereby affecting the completion of the task. The shortcomings of the above-mentioned covert interaction methods are mainly due to the lack of hardware, and the covert information interaction cannot be fully realized. In addition, the traditional gesture recognition based on electromyographic signals still requires hand movements as information basis. Obvious hand movements may cause intentions to be recognized during negotiations, weakening the negotiation advantage. Traditional covert interaction methods are difficult to achieve a balance between concealment and practicality. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device and system for non-action gesture recognition based on surface electromyography signals to address the above technical problems.

[0006] A method for non-action gesture recognition based on surface electromyography signals, the method comprising: Acquire preprocessed electromyographic signal samples under different motion states; the electromyographic signal samples are represented according to the number of channels and the number of time points; the electromyographic signal samples are marked by labels; Constructing a non-action gesture recognition network; the non-action gesture recognition network includes a time domain convolution module, a spatial domain convolution module and a time-space fusion classification module; The preprocessed electromyographic signal samples are zero-filled and initially feature extracted by the time domain convolution module to obtain a zero-filled feature map and a first feature map, the zero-filled feature map is stacked and distilled to obtain a second feature map, the first feature map and the second feature map are concatenated, and multi-scale convolution is performed to obtain time series feature maps of multiple scales; Through the spatial domain convolution module, the time domain features of each channel in the time series feature map are respectively subjected to multi-channel relationship extraction based on multi-head sub-attention at each scale to obtain the corresponding multi-channel relationship extraction results, and the multi-channel relationship extraction results are subjected to deep convolution to obtain the corresponding spatiotemporal feature map; The spatiotemporal feature maps of multiple scales are subjected to separable convolution by the spatiotemporal fusion classification module, a fusion result is output, and no-action gesture recognition is performed according to the fusion result to obtain a recognition result; Training the no-motion gesture recognition network according to the labels of the electromyographic signal samples and the prediction results obtained by inputting the electromyographic signal samples into the no-motion gesture recognition network to obtain a trained no-motion gesture recognition network; The preprocessed electromyographic signal to be detected is input into the trained non-action gesture recognition network to obtain a non-action gesture recognition result of the electromyographic signal to be detected.

[0007] A non-action gesture recognition device based on surface electromyography signals, the device comprising: A sample acquisition module is used to acquire preprocessed electromyographic signal samples under different motion states; the electromyographic signal samples are represented according to the number of channels and the number of time points; the electromyographic signal samples are marked by labels; A model building module, used to build a non-action gesture recognition network; the non-action gesture recognition network includes a time domain convolution module, a spatial domain convolution module and a time-space fusion classification module; A time domain feature extraction module is used to perform zero padding and preliminary feature extraction on the preprocessed electromyographic signal samples to obtain a zero-filled feature map and a first feature map, stack and distill the zero-filled feature map to obtain a second feature map, concatenate the first feature map and the second feature map, and then perform multi-scale convolution to obtain time series feature maps of multiple scales; The spatiotemporal feature extraction module is used to perform multi-channel relationship extraction based on multi-head sub-attention on the time domain features of each channel in the time series feature map at each scale, obtain the corresponding multi-channel relationship extraction results, perform deep convolution on the multi-channel relationship extraction results, and obtain the corresponding spatiotemporal feature map; The result output module is used to perform separable convolution on the spatiotemporal feature maps of multiple scales, output the fusion results, perform non-action gesture recognition based on the fusion results, and obtain the recognition results; A model training module, used for training the no-motion gesture recognition network according to the labels of the electromyographic signal samples and the prediction results obtained by inputting the electromyographic signal samples into the no-motion gesture recognition network to obtain a trained no-motion gesture recognition network; The gesture recognition module is used to input the preprocessed electromyographic signal to be detected into the trained non-action gesture recognition network to obtain the non-action gesture recognition result of the electromyographic signal to be detected.

[0008] A non-action gesture intelligent recognition system based on surface electromyography signals, the system comprising: A hand signal acquisition terminal, a motion state control unit, a data processing unit and a model training unit; the data processing unit includes a visualization module and an experiment management module; The hand signal acquisition terminal includes an electrode sheet, a gyroscope and a three-axis accelerometer, which is used to collect electromyographic signal samples of non-action gestures under different motion states and corresponding three-dimensional acceleration signals and transmit them to a data processing unit; The motion state control unit is used to control the motion speed of the subject; The visualization module is used to pre-process the electromyographic signal samples and the three-dimensional acceleration signal before observation; The experiment management module is used to manage the subject information, experiment information configuration and gesture management; The model training unit is used to perform model training on the non-motion gesture recognition network by using the non-motion gesture recognition method based on surface electromyography signals as described in the above embodiment to obtain a trained non-motion gesture recognition network.

[0009] The above-mentioned method, device and system for motionless gesture recognition based on surface electromyographic signal have the following beneficial effects: by collecting pre-processed electromyographic signal samples, using the samples to train the pre-built motionless gesture recognition network, and then using the trained motionless gesture recognition network to recognize the electromyographic signal to be detected, wherein the pre-processed electromyographic signal samples are first zero-filled and preliminarily feature extracted by the time domain convolution module, and multi-scale time series feature maps are obtained by stacking distillation and other operations, which can effectively mine local to global features from the time dimension to avoid information loss, and the spatial domain convolution module extracts multi-channel relationships based on multi-head sub-attention at each scale and obtains the spatiotemporal feature map by deep convolution, which fully considers the characteristics and mutual relationships of different channels, and the spatiotemporal fusion classification module uses separable convolution to fuse and output the spatiotemporal feature maps of multiple scales, integrating the key information of the spatiotemporal dimension. The modules work together to comprehensively and deeply extract and fuse the electromyographic signal features, so that the network can accurately capture the characteristics of the electromyographic signal of motionless gestures under different motion states, thereby improving its recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 1 is a flow chart of a method for non-action gesture recognition based on surface electromyography signals in one embodiment; Figure 2: is a schematic diagram of the structure of EMG Transformer in one embodiment, wherein: Figure 2 A represents the temporal embedding from local to global, B represents the multi-channel relationship extraction based on the attention mechanism, C represents the channel contribution evaluation, and D represents the spatiotemporal feature fusion; Figure 3 This is a token comparison diagram in an embodiment, where: Figure 3 (a) is a schematic diagram of the token in the traditional Transformer structure. Figure 3 (b) is a schematic diagram of the token in the traditional EMG Transformer structure; Figure 4 A schematic diagram of a data processing flow of stacked distillation layers in one embodiment; Figure 5 A schematic diagram of time embedding from local to global in one embodiment; Figure 6 A schematic diagram of an experiment management interface in an embodiment; Figure 7 A schematic diagram of a no-action gesture prompt message in an embodiment; Figure 8 A schematic diagram of a sample collection process in one embodiment; Fig. 9 4 is a structural block diagram of a non-action gesture recognition device based on surface electromyography signals in an embodiment. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0012] In one embodiment, Figure 1 As shown, a method for non-action gesture recognition based on surface electromyography signals is provided, comprising the following steps: S102, obtaining pre-processed electromyographic signal samples under different motion states.

[0013] The EMG signal samples are represented according to the number of channels and the number of time points; the EMG signal samples are labeled by labels; and the labels are used to mark the sample categories.

[0014] The preprocessing of the EMG signal samples includes designing power frequency notch, 1Hz high-pass filter and 1-100Hz band-pass filter preprocessing functions according to the characteristics of the EMG signals to effectively remove power frequency noise and low-frequency noise.

[0015] S104, constructing a non-action gesture recognition network.

[0016] The non-motion gesture recognition network includes a temporal convolution module, a spatial convolution module, and a spatiotemporal fusion classification module. Specifically, the non-motion gesture recognition network learns the spatiotemporal physiological characteristics of surface electromyography (sEMG) signals through EMG (Electromyography) Transformer, and classifies sEMG signals based on these characteristics, such as Figure 2 The structural diagram of the EMG Transformer shown in the figure first extracts multi-granular temporal dynamic information from local to global based on the stacked distillation module, then takes the temporal dynamics of each channel as a token and inputs it into the Transformer structure with only an encoder, so as to model the relationship between each channel, and measure the co-absorption of each channel and fuse the spatiotemporal features through the deep convolution module and the separable convolution module. Finally, the flattening layer and the fully connected layer are used to map the feature representation to the classification space.

[0017] S106, performing zero padding and preliminary feature extraction on the preprocessed electromyographic signal samples through the time domain convolution module to obtain a zero-padded feature map and a first feature map, stacking and distilling the zero-padded feature map to obtain a second feature map, concatenating the first feature map and the second feature map, and then performing multi-scale convolution to obtain time series feature maps of multiple scales.

[0018] The time domain convolution module is used to learn the timing information of electromyographic signals in different frequency bands. When processing electromyographic signals, signals in different time periods may represent different muscle activity states. The time domain convolution module can distinguish subtle differences in movements or gestures by capturing temporal dynamic representations from local to global.

[0019] For sEMG signals with high sampling rates, extracting global temporal correlations requires a larger receptive field. The present invention uses stacked convolution to achieve multi-granular time embedding of time series from local to global extraction. The stacked convolution method uses a small convolution kernel to process data, and its receptive field is relatively small, which can focus more finely on local features, and can better capture the subtle changes in muscle electrical activity in a short period of time, such as the signal changes caused by instantaneous muscle contraction or relaxation. These local information is crucial to understanding the details of muscle activity. The present invention uses a shallower network structure to effectively prevent overfitting, and the shallower network layers and small convolution kernels can process signals faster and provide timely feedback on relevant information about muscle activity without consuming a large amount of computing resources and time for complex deep network calculations.

[0020] S108, performing multi-channel relationship extraction based on multi-head sub-attention on the time domain features of each channel in the time series feature map at each scale through the spatial domain convolution module, obtaining the corresponding multi-channel relationship extraction results, performing deep convolution on the multi-channel relationship extraction results, and obtaining the corresponding spatiotemporal feature map.

[0021] The spatial convolution module can capture the intrinsic structural connections between channels and measure the contribution of each channel to the gesture recognition task. The electromyographic signal usually contains multiple channels, each of which reflects the activity of different aspects of the muscle. The multi-head self-attention mechanism can efficiently model the dependency between local and long-term representations, while the deep convolution layer can analyze each channel independently, helping the model to better understand the differences and connections between channels.

[0022] It should be noted that if Figure 3 As shown in (a), the traditional Transformer structure usually treats the multi-channel signals of the same sampling point as a token, thereby establishing the temporal dependency between these tokens. This strategy is beneficial to extract more comprehensive time domain features to a certain extent. However, for sEMG signals, points at the same time step, including signals from different channels, have completely different physical meanings. If they are embedded in a token, the correlation between multiple channel signals may be eliminated; and the receptive field of the token formed by a single sampling point is too local, and it may be difficult to reveal meaningful information, such as Figure 3 As shown in (b), EMGTransformer uses the time domain features of each channel as a token and inputs it into the multi-head self-attention network to extract the relationship between each channel. It can fully explore the intrinsic connection between the time domain features of different channels and avoid information confusion caused by mixing signals with different physical meanings. At the same time, it expands the receptive field in a more reasonable way, so that the model can examine the interaction between each channel from a more macro perspective, and then accurately capture the coordinated changes in muscle electrical activity reflected by different channels during muscle activity. It provides more discriminative and representative feature information for subsequent tasks such as more accurate analysis of muscle movement patterns and judgment of cognitive activity status, effectively improving the performance of the entire model in processing sEMG signal-related tasks.

[0023] S110, performing separable convolution on the spatiotemporal feature maps of multiple scales through the spatiotemporal fusion classification module, outputting the fusion result, and performing non-action gesture recognition based on the fusion result to obtain a recognition result.

[0024] The separable convolution layer can fuse the spatiotemporal information on each frequency band. The separable convolution first processes each channel of the input through depthwise convolution, and then integrates the outputs of these channels using point-by-point convolution (1x1 convolution). This step-by-step processing greatly improves the efficiency of parameter utilization and the computational efficiency of the model. This structure enables the network to effectively integrate information from different channels of the EMG signal while remaining lightweight, improving the model's ability to classify complex gestures. In particular, the separable convolution layer performs well when processing EMG signals that contain rich spatiotemporal features.

[0025] S112, training the non-motion gesture recognition network according to the label of the electromyographic signal sample and the prediction result obtained by inputting the electromyographic signal sample into the non-motion gesture recognition network to obtain a trained non-motion gesture recognition network.

[0026] S114, inputting the preprocessed electromyographic signal to be detected into a trained non-action gesture recognition network to obtain a non-action gesture recognition result of the electromyographic signal to be detected.

[0027] In one embodiment, Figure 4 As shown, a schematic diagram of the data processing flow of a stacked distillation layer is provided, and stacked distillation is performed on the zero-filled feature map to obtain a second feature map, including: in each stacked convolution layer, an exponentially expanding expansion factor is used as the step size of the convolution kernel, and features are extracted from the input feature map according to the step size to obtain the output of the next stacked convolution layer, and the second feature map is obtained after stacking convolutions in sequence.

[0028] In this embodiment, if Figure 2 As shown in Part A, a local-to-global time-domain feature extraction module is designed to extract the temporal representations of different channels and embed them into tokens independently.

[0029] Specifically, Figure 5 The schematic diagram of temporal embedding from local to global is shown in the figure. The convolution kernel acts on the input sequence by skipping elements at a specific step size, which is called the dilation factor. In order to achieve a larger receptive field, the dilation factor is exponentially expanded and stacked to extract the temporal correlation of data in a longer time window while reducing the temporal convolution layer. The sample is recorded as ,in C and T represents the number of channels and time points. Indicates i -th channel timing signal. The dilated time convolution operation is performed by Apply convolution kernel on f To get the output of the next layer , after stacking the dilated temporal convolutional layers, the output is .

[0030] In one embodiment, using the exponentially expanded dilation factor as the step length of the convolution kernel, performing feature extraction on the input feature map according to the step length, and obtaining the output of the next stacked convolution layer includes: using the exponentially expanded dilation factor as the step length of the convolution kernel, performing feature extraction on the input feature map according to the step length, and obtaining the output of the next stacked convolution layer as: ; in, is the output of the next stacked convolutional layer, j ={-1, 0, 1} is a convolutional layer, represents the corresponding element of the convolution kernel, Represents input data, represents the expansion factor. Specifically, the kernel size of the convolutional layer is 1x3. It is worth noting that when j = -1, The subscript of is negative, indicating the past state of the current timestamp in the input data corresponding to the output data. In addition, when d = 1, the dilated convolution degenerates into a traditional convolution operation.

[0031] In one embodiment, multi-channel relationship extraction based on multi-head sub-attention is performed on the time domain features of each channel in the time series feature graph, and the corresponding multi-channel relationship extraction results include: linearly transforming the time domain features of each channel to construct a triple; the triplet includes a query vector, a key vector and a value vector; the correlation scores between each two channels are calculated according to the query vector and the key vector, the correlation scores between the channels are normalized to obtain a multi-channel relationship graph, and matrix multiplication is performed on the value vector and the multi-channel relationship graph to obtain a multi-channel relationship extraction result.

[0032] In this embodiment, if Figure 2 As shown in Part B, a multi-head attention mechanism is used to reveal the correlation between multiple channels. Specifically, represents the output of the temporal embedding module, where K Indicates the number of convolution kernels in time embedding. First, each token Linear transformation into triplets of the same shape .in, is the dimension of the projection head. The Dot product is used for and The transpose of is used to evaluate the correlation between different channels. Subsequently, the scale and softmax operations are used to calculate the correlation scores between two channels. This process can be expressed by the following formula: ; The entire score graph shows the correlation between the channel pairs. For the next The mechanism proposed by the method of the present invention is more natural and can be used for multi-channel sEMG signal modeling.

[0033] In one embodiment, performing deep convolution on the multi-channel relationship extraction results to obtain the corresponding spatiotemporal feature map includes: performing deep convolution on the multi-channel relationship extraction results using multiple convolution kernels whose lengths are equal to the number of channels to obtain the corresponding spatiotemporal feature map.

[0034] In this embodiment, if Figure 2 As shown in Figure 1, part C applies deep convolution to measure the contribution of each channel. Specifically, after temporal embedding and channel relationship modeling, spatial convolution operations are applied on all signal channels to further extract the spatial representation of sEMG signals. Considering that cognitive activities are reflected in the EMG signals of multiple channels, EMGTransformer adopts K The length is equal to the number of channels C The convolution kernel is used to evaluate the contribution of each channel to the spatial representation. The calculation process can be expressed as follows: ; Among them, f represents the spatial convolution kernel, and Represents the kernel size in channel and time dimensions k , c and t Represents the indices of kernel, channel and time point respectively.

[0035] Subsequently, the batch normalization (BN) layer speeds up the training process and enhances the generalization ability of the model. We use the ELU function as the nonlinear activation function after the BN layer. ELU has a soft saturation property when processing small input values ​​and is more robust to noise and input variations. Finally, an average pooling layer of size (1, 4) and a Dropout layer are attached in sequence to improve the generalization ability while maintaining the nonlinear mapping.

[0036] In one embodiment, separable convolution is performed on spatiotemporal feature maps of multiple scales, and outputting a fusion result includes: performing depth convolution on the spatiotemporal feature maps at different scales, then performing point-by-point convolution, and performing weighted combination in the depth direction to obtain a corresponding fusion result.

[0037] In this embodiment, if Figure 2As shown, the separable convolution in part D is used to fuse spatiotemporal representations from different kernels. First, each spatiotemporal feature map is optimized using depthwise convolution, and then the relationship between feature maps is learned using point-by-point convolution. PhyTransformer is very useful in learning how to combine kernels in the temporal dimension and how to optimize the combination separately, because different feature maps represent information of data at different time granularities.

[0038] Specifically, input It is first input into a depthwise convolution with a kernel size of , whose formula is shown below. Then, use K Point-wise convolution with kernel size (1, 1) to capture the relationship between feature maps; ; Similar to the operation in channel contribution evaluation, the BN layer, ELU activation function, average pooling and Dropout layer are connected in sequence to improve the generalization ability of the model. The pooling layer kernel size is 1X 8.

[0039] In one embodiment, no-motion gesture recognition is performed based on the fusion result to obtain the recognition result, including: converting the fusion results at each scale into a no-motion gesture feature vector through a fully connected layer, predicting the corresponding no-motion gesture category based on the no-motion gesture feature vector, and obtaining the recognition result.

[0040] In this embodiment, a fully connected layer and a softmax function are used to map the spatiotemporal representation to the classification space. The calculation process is shown in the following formula: ; Where Q is the transformation matrix, which maps the representation to the label space for final classification, b is the bias, which can increase the flexibility of the function and improve the fitting ability of the neuron, M Indicates the number of classes.

[0041] During training, the cross entropy loss function is used to evaluate the difference between the predicted value and the label, and its formula is as follows: ; in N represents the number of samples, is the label of the input signal.

[0042] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0043] In one embodiment, a non-action gesture intelligent recognition system based on surface electromyography signals is characterized in that the system includes a hand signal acquisition terminal, a motion state control unit, a data processing unit and a model training unit; the data processing unit includes a visualization module and an experiment management module; The hand signal acquisition terminal includes an electrode sheet, a gyroscope and a three-axis accelerometer, which is used to collect myoelectric signal samples of non-action gestures under different motion states and the corresponding three-dimensional acceleration signals and transmit them to the data processing unit; The motion state control unit is used to control the motion speed of the subject; The visualization module is used to pre-process the electromyographic signal samples and three-dimensional acceleration signals before observation; The experiment management module is used to manage the subject information, experiment information configuration and gesture management; The model training unit is used to perform model training on the non-motion gesture recognition network by using the non-motion gesture recognition method based on surface electromyography signals as in the above embodiment, so as to obtain a trained non-motion gesture recognition network.

[0044] Specifically, the hand signal acquisition terminal is in the shape of a wristwatch. The main body is made of soft silicone. The inner cavity made by the silicone molding process conceals the wires. In addition, the shaft adopts a hollow design, and the wires in the silicone belt are connected to the PCB motherboard of the host shell through the hollow shaft. Eight pairs of 4.0mm medical standard female buckles are arranged longitudinally on the inside of the wristband, which are compatible with the electrode sheet of the 3.9mm standard interface and can collect 8-channel 500Hz sampling rate electromyographic signals. At the same time, the terminal also has a built-in gyroscope and a three-axis accelerometer, which can measure x, y and z three-dimensional acceleration signals. That is, the data collected by the terminal includes: (1) eight-channel wrist electromyographic signals, which are used to monitor the tension / relaxation state or movement intention of the hand muscles; (2) three-axis acceleration signals, which are used to monitor the movement state of the wearer.

[0045] Considering that personnel are often in motion when wearing terminal equipment to perform tasks. In practical applications, it is necessary not only to be able to decode the intention of electromyographic signals in a stable state, but also to ensure normal operation in a moving state. Therefore, the collected experimental data should also cover various scenes of a stable state and a moving state, so as to analyze the characteristics of the data in the moving state and design the corresponding algorithm. The main body of the motion state control unit is implemented on a treadmill, which has a speed adjustment range of 0~12km / h. Different motion scenes such as slow walking, fast walking, and jogging are simulated by setting different motion speeds.

[0046] The visualization module is used to display the preprocessed EMG signal samples and three-dimensional acceleration signals on the visualization interface. The main drawing area of ​​the visualization interface draws 8-channel EMG signals and hand movement posture signals (three-way acceleration signals) in real time. At the same time, according to the characteristics of EMG signals, power frequency notch, 1Hz high-pass filter and 1-100Hz band-pass filter preprocessing functions are designed to effectively remove power frequency noise and low-frequency noise. Through this interface, the acquisition quality of EMG signals can be directly observed during the experiment, which is convenient for experimental process control and ensures the experimental effect.

[0047] The purpose of the experiment management module is to effectively manage the information of different subjects, experiment information, and configure experiment parameters. Figure 5 As shown in the figure, the prompt information of the data collection process is added in the middle of the display interface. During the experiment, the subjects collect data by taking corresponding actions according to the prompt information of this interface. The right part of the display interface shows the basic information of the subjects and the setting of experimental parameters. Figure 5 In the upper right area of ​​the interface, you can enter the basic information of the subject, including name, gender and age. You can also select whether the subject is in a static or moving state for data collection, and set the number of blocks in the collection process and the continuous collection time of a single gesture.

[0048] The gesture management part is used to manage the gestures used in the experiment. You can select the gestures you need to collect data from the gesture library as needed. The gesture library of this experimental system currently contains 5 gestures with action and 12 gestures without action, such as Figure 7As shown in the figure, 12 types of no-action gesture prompt information are displayed. During the experiment, the picture prompts the subject to perform the corresponding action. The fingers marked with different colors represent that the gesture requires the finger to maintain a force state, and the fingers marked with the same color as the palm remain relaxed. During the no-action experiment, the subject is required to hold a cylindrical object in his hand to simulate the state of holding a weapon. During the force process, the fingers always hold the cylindrical object tightly to ensure that the action is produced in a no-action state. The experimental software also adds a voice broadcast function to remind the subject when the prompt information changes, helping the subject to complete the gesture switching more accurately. The sample collection paradigm of the system of the present invention is designed as follows: (1) The paradigm is based on the experimental platform design, and the terminal collects 8 channels of no-action electromyographic data with a sampling rate of 500hz; (2) The paradigm combines the application requirements of no-action intention decoding in sports scenes, and sets the movement speeds of 4km / h, 6km / h, and 8km / h to simulate the three movement modes of slow walking, fast walking, and jogging. The paradigm process is as follows Figure 8 As shown in the figure, it is divided into two stages: preparation and data collection. The data collection stage is described in detail as follows: After wearing the terminal, the subjects grasped a hard water bottle (simulating the state of holding a gun) and stood on the motion platform to collect data. Each block included two modes: static and motion. The motion mode was divided into three sub-modes according to the motion speed: 4km / h, 6km / h and 8km / h. In the static mode, the subjects stood on the treadmill and made corresponding gestures according to the voice broadcast and the prompt interface signal. The duration of each trial was set to 10s. In the motion mode and the static mode, the subjects' tasks were consistent, but the treadmill would run at a preset speed, and the subjects needed to move synchronously with the movement of the treadmill. A single subject will collect 12 types of gesture data without action in one experiment. Each type of gesture is collected for 10s. The 12 types of gestures are traversed as one group, which takes 10×12=120s. The groups rested for one minute for the subjects to adjust. There were 4 groups of static, 4km / h, 6km / h, and 8km / h different motion states to form a block. The number of blocks was set to 3. The estimated single data collection time is about 36 minutes.

[0049] In one embodiment, Fig. 9 As shown, a non-action gesture recognition device based on surface electromyography signal is provided, comprising: The sample acquisition module is used to obtain the pre-processed electromyographic signal samples under different motion states; the electromyographic signal samples are represented according to the number of channels and the number of time points; the electromyographic signal samples are marked by labels; A model building module is used to build a non-motion gesture recognition network; the non-motion gesture recognition network includes a time domain convolution module, a spatial domain convolution module and a time-space fusion classification module; A time domain feature extraction module is used to perform zero padding and preliminary feature extraction on the preprocessed electromyographic signal samples to obtain a zero-padded feature map and a first feature map, stack and distill the zero-padded feature map to obtain a second feature map, concatenate the first feature map and the second feature map, and then perform multi-scale convolution to obtain time series feature maps of multiple scales; The spatiotemporal feature extraction module is used to perform multi-channel relationship extraction based on multi-head sub-attention on the time domain features of each channel in the time series feature map at each scale, obtain the corresponding multi-channel relationship extraction results, perform deep convolution on the multi-channel relationship extraction results, and obtain the corresponding spatiotemporal feature map; The result output module is used to perform separable convolution on the spatiotemporal feature maps of multiple scales, output the fusion results, perform non-action gesture recognition based on the fusion results, and obtain the recognition results; A model training module is used to train the no-motion gesture recognition network according to the labels of the electromyographic signal samples and the prediction results obtained by inputting the electromyographic signal samples into the no-motion gesture recognition network to obtain a trained no-motion gesture recognition network; The gesture recognition module is used to input the preprocessed electromyographic signal to be detected into the trained non-action gesture recognition network to obtain the non-action gesture recognition result of the electromyographic signal to be detected.

[0050] For the specific definition of the non-action gesture recognition device based on surface electromyography signals, please refer to the definition of the non-action gesture recognition method based on surface electromyography signals above, which will not be repeated here. Each module in the above-mentioned non-action gesture recognition device based on surface electromyography signals can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0051] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for non-action gesture recognition based on surface electromyography signals, characterized in that: The method comprises: Acquire preprocessed electromyographic signal samples under different motion states; the electromyographic signal samples are represented according to the number of channels and the number of time points; the electromyographic signal samples are marked by labels; Constructing a non-action gesture recognition network; the non-action gesture recognition network includes a time domain convolution module, a spatial domain convolution module and a time-space fusion classification module; The preprocessed electromyographic signal samples are zero-filled and initially feature extracted by the time domain convolution module to obtain a zero-filled feature map and a first feature map, the zero-filled feature map is stacked and distilled to obtain a second feature map, the first feature map and the second feature map are concatenated, and multi-scale convolution is performed to obtain time series feature maps of multiple scales; Through the spatial domain convolution module, the time domain features of each channel in the time series feature map are respectively subjected to multi-channel relationship extraction based on multi-head sub-attention at each scale to obtain the corresponding multi-channel relationship extraction results, and the multi-channel relationship extraction results are subjected to deep convolution to obtain the corresponding spatiotemporal feature map; The spatiotemporal feature maps of multiple scales are subjected to separable convolution by the spatiotemporal fusion classification module, a fusion result is output, and no-action gesture recognition is performed according to the fusion result to obtain a recognition result; Training the no-motion gesture recognition network according to the labels of the electromyographic signal samples and the prediction results obtained by inputting the electromyographic signal samples into the no-motion gesture recognition network to obtain a trained no-motion gesture recognition network; The preprocessed electromyographic signal to be detected is input into the trained non-action gesture recognition network to obtain a non-action gesture recognition result of the electromyographic signal to be detected.

2. The method for non-action gesture recognition based on surface electromyography signal according to claim 1, characterized in that: The stacking and distilling of the zero-filled feature map to obtain the second feature map includes: In each stacked convolution layer, the exponentially expanding dilation factor is used as the step size of the convolution kernel. The input feature map is extracted according to the step size to obtain the output of the next stacked convolution layer. After stacking the convolutions in sequence, the second feature map is obtained.

3. The method for non-action gesture recognition based on surface electromyography signal according to claim 2, characterized in that: The exponentially expanding dilation factor is used as the step size of the convolution kernel, and the input feature map is extracted according to the step size. The output of the next stacked convolution layer includes: The exponentially expanding dilation factor is used as the step size of the convolution kernel. The input feature map is extracted according to the step size, and the output of the next stacked convolution layer is: ； in, is the output of the next stacked convolutional layer, j ={-1, 0, 1} is a convolutional layer, represents the corresponding element of the convolution kernel, Represents input data, Represents the expansion factor.

4. The method for non-action gesture recognition based on surface electromyography signal according to claim 1, characterized in that: The time domain features of each channel in the time series feature graph are subjected to multi-channel relationship extraction based on multi-head sub-attention, and the corresponding multi-channel relationship extraction results include: Performing linear transformation on the time domain features of each channel to construct a triplet; the triplet includes a query vector, a key vector and a value vector; The correlation scores between each pair of channels are calculated based on the query vector and the key vector. The multi-channel relationship graph is obtained after the correlation scores between each channel are normalized. Matrix multiplication is performed based on the value vector and the multi-channel relationship graph to obtain the multi-channel relationship extraction result.

5. The method for non-action gesture recognition based on surface electromyography signal according to claim 1, characterized in that: The multi-channel relationship extraction results are subjected to deep convolution to obtain the corresponding spatiotemporal feature maps including: Multiple convolution kernels with lengths equal to the number of channels are used to perform deep convolution on the multi-channel relationship extraction results to obtain the corresponding spatiotemporal feature maps.

6. The method for non-action gesture recognition based on surface electromyography signal according to claim 1, characterized in that: Separable convolution is performed on spatiotemporal feature maps of multiple scales, and the output fusion results include: The spatiotemporal feature maps are deeply convolved at different scales, then point-by-point convolution is performed, and weighted combination is performed in the depth direction to obtain the corresponding fusion results.

7. The method for non-action gesture recognition based on surface electromyography signal according to claim 1, characterized in that: According to the fusion results, non-action gesture recognition is performed, and the recognition results include: The fusion results at each scale are converted into a no-action gesture feature vector through a fully connected layer, and the corresponding no-action gesture category is predicted according to the no-action gesture feature vector to obtain a recognition result.

8. A non-action gesture recognition device based on surface electromyography signal, characterized in that: The device comprises: A sample acquisition module is used to acquire preprocessed electromyographic signal samples under different motion states; the electromyographic signal samples are represented according to the number of channels and the number of time points; the electromyographic signal samples are marked by labels; A model building module, used to build a non-action gesture recognition network; the non-action gesture recognition network includes a time domain convolution module, a spatial domain convolution module and a time-space fusion classification module; A time domain feature extraction module is used to perform zero padding and preliminary feature extraction on the preprocessed electromyographic signal samples to obtain a zero-filled feature map and a first feature map, stack and distill the zero-filled feature map to obtain a second feature map, concatenate the first feature map and the second feature map, and then perform multi-scale convolution to obtain time series feature maps of multiple scales; The spatiotemporal feature extraction module is used to perform multi-channel relationship extraction based on multi-head sub-attention on the time domain features of each channel in the time series feature map at each scale, obtain the corresponding multi-channel relationship extraction results, perform deep convolution on the multi-channel relationship extraction results, and obtain the corresponding spatiotemporal feature map; The result output module is used to perform separable convolution on the spatiotemporal feature maps of multiple scales, output the fusion results, perform non-action gesture recognition based on the fusion results, and obtain the recognition results; A model training module, used for training the no-motion gesture recognition network according to the labels of the electromyographic signal samples and the prediction results obtained by inputting the electromyographic signal samples into the no-motion gesture recognition network to obtain a trained no-motion gesture recognition network; The gesture recognition module is used to input the preprocessed electromyographic signal to be detected into the trained non-action gesture recognition network to obtain the non-action gesture recognition result of the electromyographic signal to be detected.

9. An intelligent recognition system for gestures without motion based on surface electromyography signals, characterized in that: The system includes a hand signal acquisition terminal, a motion state control unit, a data processing unit and a model training unit; the data processing unit includes a visualization module and an experiment management module; The hand signal acquisition terminal includes an electrode sheet, a gyroscope and a three-axis accelerometer, which is used to collect electromyographic signal samples of non-action gestures under different motion states and corresponding three-dimensional acceleration signals and transmit them to a data processing unit; The motion state control unit is used to control the motion speed of the subject; The visualization module is used to pre-process the electromyographic signal samples and the three-dimensional acceleration signal before observation; The experiment management module is used to manage the subject information, experiment information configuration and gesture management; The model training unit is used to perform model training on the non-motion gesture recognition network using the non-motion gesture recognition method based on surface electromyography signals as described in any one of claims 1 to 7 to obtain a trained non-motion gesture recognition network.

Images