Gesture Recognition Method, Device and Electronic Device

Abstract

The present invention discloses a gesture recognition method, apparatus and electronic device. The method includes: acquiring electroencephalogram (EEG) signals, electromyogram (EMG) signals and inertial navigation signals; determining a gesture action according to the EMG signals and the inertial navigation signals; judging whether the gesture action is a valid action according to the EEG signals; if the gesture action is a valid action, acquiring an operation instruction corresponding to the gesture action, and performing a corresponding action according to the operation instruction. By adding EEG signals in the gesture recognition process and judging whether the gesture action is a valid action according to the EEG signals, the present invention can eliminate unconscious gesture actions and reduce misoperations.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control, and more specifically, to a gesture recognition method, device and electronic device. Background Art

[0002] Electromyography (EMG) is a weak non-linear high internal resistance signal formed by the final spatio-temporal superposition at the detection electrode after the action potential sequence formed by multiple motor units participating in muscle activity passes through the filtering effect of the volume conductor composed of skin, subcutaneous tissue and muscle.

[0003] Since users may make unconscious actions, in the existing gesture recognition methods, it is not possible to accurately determine whether a gesture action is effective only through EMG signals, resulting in a reduction in the accuracy of gesture recognition and an increase in misoperations. Summary of the Invention

[0004] An object of the present invention is to provide a new technical solution for gesture recognition.

[0005] According to a first aspect of the present invention, there is provided a gesture recognition method, the method comprising:

[0006] Obtaining electroencephalogram signals, electromyogram signals and inertial navigation signals;

[0007] Determining a gesture action according to the electromyogram signal and the inertial navigation signal;

[0008] Judging whether the gesture action is a valid action according to the electroencephalogram signal;

[0009] If the gesture action is a valid action, obtaining an operation instruction corresponding to the gesture action and performing a corresponding action according to the operation instruction.

[0010] Optionally, the gesture action includes a hand action and an arm action, and the judging the type of the gesture action according to the electromyogram signal and the inertial navigation signal includes:

[0011] Determining a hand action according to the electromyogram signal;

[0012] Determining an arm action according to the inertial navigation signal;

[0013] Determining the gesture action according to the hand action and the arm action.

[0014] Optionally, the determining a hand action according to the electromyogram signal includes:

[0015] Calculate a first eigenvalue, a second eigenvalue, and a third eigenvalue of the electromyogram signal, where the first eigenvalue is the average value of the time-domain features of the electromyogram signal on a first channel, the second eigenvalue is the average value of the time-domain features of the electromyogram signal on a second channel, and the third eigenvalue is the average value of the time-domain features of the electromyogram signal on a third channel;

[0016] If the first eigenvalue is greater than a first threshold, the first eigenvalue is greater than the second eigenvalue, and the first eigenvalue is greater than the third eigenvalue, determine that the hand movement is an eversion movement;

[0017] If the second eigenvalue is greater than a second threshold, the second eigenvalue is greater than the first eigenvalue, and the second eigenvalue is greater than the third eigenvalue, determine that the hand movement is a fist clenching movement;

[0018] If the third eigenvalue is greater than a third threshold, the third eigenvalue is greater than the first eigenvalue, and the third eigenvalue is greater than the second eigenvalue, determine that the hand movement is an inversion movement.

[0019] Optionally, the inertial navigation signal includes a first inertial navigation signal corresponding to the upper arm and a second inertial navigation signal corresponding to the lower arm. Determining the arm movement according to the inertial navigation signal includes:

[0020] Obtain first acceleration values in three preset directions according to the first inertial navigation signal, and obtain the corresponding upper arm posture according to the first acceleration values;

[0021] Obtain second acceleration values in three preset directions according to the second inertial navigation signal, and obtain the corresponding lower arm posture according to the second acceleration values;

[0022] Determine the arm movement according to the upper arm posture and the lower arm posture.

[0023] Optionally, determining whether the gesture movement is a valid movement according to the electroencephalogram signal includes:

[0024] Obtain a start time and an end time corresponding to the gesture movement in the electromyogram signal;

[0025] Obtain a target signal, where the target signal is the signal of the electroencephalogram signal between the start time and the end time;

[0026] Perform frequency decomposition on the target signal by a fast Fourier transform method to obtain the energy of the target signal in each preset frequency band;

[0027] Calculate an attention value according to the energy of the target signal in each preset frequency band and the total energy of the target signal according to a preset algorithm;

[0028] If the attention value exceeds the threshold, it is determined that the gesture action is a valid action.

[0029] Optionally, the obtaining of the start time and end time corresponding to the gesture action in the myoelectric signal includes:

[0030] Obtain the sub-signals of the myoelectric signal in each channel;

[0031] Calculate the root mean square value of the sub-signals in each sampling window, and calculate the mean square signal of the myoelectric signal in each channel according to the root mean square value;

[0032] Superimpose the mean square signals of the myoelectric signal in each channel to obtain a superimposed signal;

[0033] Calculate the average value of the superimposed signal;

[0034] Calculate the active segment segmentation threshold according to the average value of the superimposed signal and a preset coefficient;

[0035] Set the time period in the superimposed signal that is greater than the active segment segmentation threshold as the active segment;

[0036] Obtain the corresponding start time and end time according to the active segment.

[0037] Optionally, before determining the type of the gesture action according to the myoelectric signal and the inertial navigation signal, the method further includes:

[0038] Filter the myoelectric signal through an eighth-order Butterworth filter;

[0039] Filter the electroencephalogram signal through a low-pass filter with a cut-off frequency of 30 Hz and a high-pass filter with a cut-off frequency of 0.1 Hz.

[0040] Optionally, the obtaining of the operation instruction corresponding to the gesture action includes:

[0041] Obtain the decoding data corresponding to the gesture action;

[0042] Query the corresponding operation instruction from a preset gesture coding library according to the decoding data.

[0043] According to the second aspect of the present invention, there is provided a gesture recognition device, and the device includes:

[0044] A first acquisition module, configured to acquire an electroencephalogram signal;

[0045] A second acquisition module, configured to acquire a myoelectric signal;

[0046] A third acquisition module, configured to acquire an inertial navigation signal;

[0047] A first determination module, configured to determine a gesture action according to the electromyogram signal and the inertial navigation signal;

[0048] A first judgment module, configured to judge whether the gesture action is a valid action according to the electroencephalogram signal;

[0049] A fourth acquisition module, configured to acquire an operation instruction corresponding to the gesture action when the gesture action is a valid action.

[0050] According to a third aspect of the present invention, there is provided an electronic device, including a processor and a memory, where the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the gesture recognition method described in the first aspect of the present invention are implemented.

[0051] According to an embodiment of the present disclosure, by adding an electroencephalogram signal to the gesture recognition process and judging whether the gesture action is a valid action according to the electroencephalogram signal, the present invention can eliminate unconscious gesture actions and reduce misoperations.

[0052] Through the following detailed description of the exemplary embodiments of the present invention with reference to the accompanying drawings, other features and advantages of the present invention will become clear. Description of the Drawings

[0053] The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments of the present invention and, together with the description, are used to explain the principles of the present invention.

[0054] Figure 1 is a flowchart of a gesture recognition method of the present invention.

[0055] Figure 2 is an electromyogram feature map when the hand makes a fist in the present invention.

[0056] Figure 3 is an electromyogram feature map when the hand is everted in the present invention.

[0057] Figure 4 is an electromyogram feature map when the hand is everted in the present invention.

[0058] Figure 5 is a schematic diagram of a sliding window in the present invention.

[0059] Figure 6a is a schematic diagram before smoothing the inertial navigation signal in the present invention.

[0060] Figure 6b is a schematic diagram after smoothing the inertial navigation signal in the present invention.

[0061] Figure 7aIt is the waveform diagram of the electromyogram signal in the present invention.

[0062] Figure 7b It is the waveform diagram of the mean square signal in the present invention.

[0063] Figure 8 It is the schematic diagram of the superimposed signal and the active segment threshold in the present invention.

[0064] Figure 9 It is the schematic diagram of the active segment segmentation of the electromyogram signal in the present invention.

[0065] Figure 10 It is the waveform diagram and the spectrogram of the electromyogram signal in the present invention.

[0066] Figure 11 It is the waveform diagram and the spectrogram of the electromyogram signal after filtering processing in the present invention.

[0067] Figure 12 It is the schematic diagram of the gesture recognition device in the present invention.

[0068] Figure 13 It is the schematic diagram of an electronic device in the present invention. Detailed implementation manners

[0069] Now, various exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present invention.

[0070] The following description of at least one exemplary embodiment is merely illustrative in nature and in no way limiting of the present invention or its application or use.

[0071] Technologies, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the said technologies, methods and devices should be regarded as part of the specification.

[0072] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0073] It should be noted that: Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0074] As Figure 1 shown, the present invention introduces a gesture recognition method, and the method includes steps S101 - S104.

[0075] S101: Obtain electroencephalogram (EEG) signals, electromyogram (EMG) signals, and inertial navigation signals.

[0076] S102: Determine a gesture action based on the EMG signal and the inertial navigation signal.

[0077] S103: Determine whether the gesture action is a valid action based on the EEG signal.

[0078] S104: If the gesture action is a valid action, obtain an operation instruction corresponding to the gesture action, and perform a corresponding action according to the operation instruction.

[0079] An EEG sensor is provided on the user's head, and an EMG sensor and an inertial navigation sensor are provided on the user's hand. The EEG signal is obtained through the EEG sensor, the EMG signal of the user's hand is obtained through the EMG sensor, and the inertial navigation signal of the user's hand is obtained through the inertial navigation sensor. Decision-level fusion is performed on the EMG signal and the inertial navigation signal to judge the gesture action, and the gesture action interval is mapped to the EEG signal. The user's current attention value can be obtained according to the EEG signal, and it is judged whether the gesture action is a valid action according to the attention value.

[0080] If the gesture action is a valid action, then obtain an operation instruction according to the gesture action and perform control according to the operation instruction. For example, the operation instruction can be used to control an autonomous vehicle. The autonomous vehicle can be controlled to drive straight, turn, accelerate, decelerate, stop, etc. according to various different gestures. The corresponding relationship between the gesture action and the autonomous vehicle instruction can be established in advance, and the corresponding autonomous vehicle instruction is queried after the gesture action is obtained to perform corresponding control on the autonomous vehicle.

[0081] In the present invention, by adding the EEG signal in the gesture recognition process and judging whether the gesture action is a valid action according to the EEG signal, unconscious gesture actions can be eliminated and misoperations can be reduced.

[0082] In an embodiment of the present invention, the gesture action includes a hand action and an arm action. The method for judging the type of the gesture action based on the EMG signal and the inertial navigation signal includes: determining the hand action according to the EMG signal; determining the arm action according to the inertial navigation signal; and determining the gesture action according to the hand action and the arm action.

[0083] The hand action refers to the action of the part below the wrist, including the palm and fingers, such as "fist clenching", "eversion", and "inversion". The arm includes the upper arm and the lower arm, and the corresponding arm actions include upper arm actions and lower arm actions. The hand action is determined according to the EMG signal, and the arm action is determined according to the inertial navigation signal. The EMG hand action recognition and the inertial navigation arm action recognition are independent of each other, and the final gesture action is determined according to the hand action and the arm action.

[0084] Determining the hand movement according to the myoelectric signal includes:

[0085] Calculating a first eigenvalue, a second eigenvalue, and a third eigenvalue of the myoelectric signal, where the first average value is the average value of the time-domain features of the myoelectric signal on the first channel, the second average value is the average value of the time-domain features of the myoelectric signal on the second channel, and the third average value is the average value of the time-domain features of the myoelectric signal on the third channel;

[0086] If the first eigenvalue is greater than the first threshold, the first eigenvalue is greater than the second eigenvalue, and the first eigenvalue is greater than the third eigenvalue, it is determined that the hand movement is an eversion movement;

[0087] If the second eigenvalue is greater than the second threshold, the second eigenvalue is greater than the first eigenvalue, and the second eigenvalue is greater than the third eigenvalue, it is determined that the hand movement is a fist clenching movement;

[0088] If the third eigenvalue is greater than the third threshold, the third eigenvalue is greater than the first eigenvalue, and the third eigenvalue is greater than the second eigenvalue, it is determined that the hand movement is an inversion movement.

[0089] In the present invention, time-domain features with low algorithm complexity and stable actual use effects are used to reduce system delay. The steps for extracting myoelectric signal features are as follows:

[0090] S201: Calculate the root mean square signal rms_EMG of each channel data. For the time series S k (i) of the myoelectric signal data of each channel, use a window W with a length of n = 90, and calculate the root mean square value within each window according to the following formula as the value of the root mean square signal at time i; slide the window W along the direction of time t, and sequentially calculate the root mean square values after each slide to obtain the time series signal S k_rms .

[0091]

[0092] where k represents the k-th channel, T is the signal length, and n is the window size.

[0093] S202: Slide the window along the time t direction. Calculate the MAV (mean absolute value) feature of each channel data within each window according to the following formula.

[0094]

[0095] where M is the number of sliding windows.

[0096] S203: Horizontally splice the features of all windows together to obtain the feature matrix VMAV (chan * M). Chan is the number of channels, and M is the number of sliding windows.

[0097] S204: Calculate the mean value of the feature matrix to obtain the feature vector V MAV (chan * 1) is used as the time-domain feature of this gesture.

[0098]

[0099] As Figures 2 to 4 shown, the RMS electromyogram feature maps of the three actions of making a fist, eversion, and inversion are respectively shown. It can be seen that the feature results of the three actions on the 3 channels have obvious differences. Among them, the action of making a fist is mainly related to the features of the second and third channels, and is most related to the second channel. The eversion action is mainly related to the first channel. The inversion action is related to the features of the second and third channels, and the features are more obvious on the third channel.

[0100] It can be found through the time-domain feature map that the features of the gesture actions on each channel have significant differences. By setting the thresholds of different channels, a set of logical criterion recognition algorithms based on time-domain features is designed in this project, and the sliding window voting method is used to improve the recognition accuracy of the algorithm. The gesture logic judgment rules are as follows:

[0101] If the first eigenvalue is greater than the first threshold, and at the same time the first eigenvalue is greater than the second eigenvalue and greater than the third eigenvalue, then the hand action is an eversion action.

[0102] If the second eigenvalue is greater than the second threshold, and at the same time the second eigenvalue is greater than the first eigenvalue and the second eigenvalue is greater than the third eigenvalue, it is determined that the hand action is a fist-making action.

[0103] If the third eigenvalue is greater than the third threshold, and at the same time the third eigenvalue is greater than the first eigenvalue and the third eigenvalue is greater than the second eigenvalue, it is determined that the hand action is an inversion action.

[0104] To further reduce the classification error and improve the classification effect, during the gesture recognition process, a sliding window is used to perform multiple discriminations on it, and the voting method is used for the final gesture determination, that is, the gesture action with the most votes is used as the final gesture. During the actual online classification process, the sliding step is 5 ms. The sliding method is as Figure 5 shown.

[0105] In an embodiment of the present invention, the inertial navigation signals include a first inertial navigation signal corresponding to the upper arm and a second inertial navigation signal corresponding to the lower arm. Determining the arm movement according to the inertial navigation signals includes: obtaining first acceleration values in three preset directions according to the first inertial navigation signal, and obtaining the corresponding upper arm posture according to the first acceleration values; obtaining second acceleration values in three preset directions according to the second inertial navigation signal, and obtaining the corresponding lower arm posture according to the second acceleration values; determining the arm movement according to the upper arm posture and the lower arm posture.

[0106] An inertial navigation system (IMU, Inertial Measurement Unit) is an autonomous navigation system that uses inertial sensors to measure the specific force and angular velocity information of a carrier, and combines given initial conditions to calculate parameters such as the carrier speed, azimuth, and posture in real time. In the present invention, two six-axis inertial sensors are used to identify the arm posture. Each sensor includes a three-axis gyroscope and a three-axis accelerometer. The x, y, z-axis gravitational accelerations provided by the three-axis accelerometer can be used to judge the position and posture of the arm. The first inertial navigation signal and the second inertial navigation signal are obtained through two inertial sensors respectively.

[0107] In the actual use process, the execution of the action and the slight vibration of the arm will have a certain impact on the signal waveform, causing it to fluctuate and generate a small number of short-time pulses, increasing the complexity of arm posture recognition and reducing the accuracy of gesture recognition. Therefore, in the present invention, a moving average method is adopted to smooth the 3-axis gravitational acceleration data, that is, the data mean of the past moment and the current moment is used as the gravitational acceleration data at the current moment. The waveform before data smoothing is as Figure 6a shown, and the waveform after data smoothing is shown in Figure 6b.

[0108] The value range of the gravitational acceleration g is between -9.8 and +9.8. When the positive direction of a certain axis points vertically to the ground, the acceleration value of the corresponding axis is +9.8, and vice versa it is -9.8. When the system is designed, only three directions of the x, y, z axes are adopted: the positive direction of the axis points to the ground, the positive direction of the axis is horizontal to the ground, and the negative direction of the axis is perpendicular to the ground. The discretized output gravity(x, y, z) of the gravitational accelerations of the x, y, z axes follows the following rules:

[0109]

[0110] If g is between (-9.8, -8), then g = -1, that is, at this time the negative direction of the axis points vertically to the ground; if g is between (-1, 1), then g = 0, that is, at this time the axis direction is parallel to the ground; if g is between (8, 9.8), then g = 1, that is, at this time the positive direction of the axis points vertically to the ground.

[0111] State encoding is performed based on the discretized output results of two inertial sensors. The corresponding relationships between the encoding states and the arm postures are shown in Table 1. Since the arm cannot be strictly perpendicular or parallel to the ground, after the discretized output of the gravitational acceleration in each axis, it can not only improve the accuracy of arm posture recognition but also significantly reduce the computational complexity.

[0112] Table 1 Corresponding relationships between arm states and inertial sensors

[0113]

[0114] In an embodiment of the present invention, the determining whether the gesture action is a valid action according to the electroencephalogram signal includes: obtaining the start time and the end time corresponding to the gesture action in the electromyogram signal; obtaining a target signal, where the target signal is the electroencephalogram signal between the start time and the end time; performing frequency decomposition on the target signal by means of the fast Fourier transform method to obtain the energy of the target signal in each preset frequency band; calculating an attention value according to the energy of the target signal in each preset frequency band and the total energy of the target signal according to a preset algorithm; and if the attention value exceeds a threshold, determining that the gesture action is a valid action.

[0115] After obtaining the start time and the end time corresponding to the gesture action, the corresponding target signal is intercepted from the electroencephalogram signal. Frequency decomposition is performed on the target signal by means of the fast Fourier transform, the proportion of each frequency component is calculated, and the current attention value is calculated according to the proportion of the energy of each frequency band in the total energy of the electroencephalogram signal, and the attention value is normalized so that the attention value is within the range of 0-100. When the attention value is within the range of 0-20, it indicates that the attention is extremely inattentive and in a state of being upset and distracted; when the attention value is within the range of 20-40, it indicates that the wearer's attention is slightly inattentive and a little distracted; when the attention value is within the range of 40-60, it indicates that the attention is in a general concentrated state; when the attention value is within the range of 60-80, it indicates that the wearer's attention is very concentrated; when the attention value is within the range of 80-100, it indicates that the wearer's attention is highly concentrated; the larger the attention value, the more concentrated the user's attention is. The present invention presets an attention threshold. If the attention calculated according to the target signal is greater than the preset attention threshold, then it is determined that the above gesture action is a valid action.

[0116] Since the human electroencephalogram is affected by various factors, the attention level is highly correlated with factors such as the sensor wearing position, human sweat glands, and the mood of the subject. In order to accurately evaluate the current attention level of the subject, the attention threshold is corrected before each experiment. The specific method is as follows: 1) The subject performs a 10-minute gesture action test; 2) Intentionally perform the gesture 2 times and unintentionally perform the gesture 2 times, alternating; 3) Record the attention value during the execution of the gesture action; 4) Take the average value of the attention values as the current attention threshold.

[0117] In an embodiment of the present invention, obtaining the start time and end time corresponding to the gesture action in the myoelectric signal includes: obtaining the sub-signal of the myoelectric signal in each channel; calculating the root mean square value of the sub-signal in each sampling window, and calculating the mean square signal of the myoelectric signal in each channel according to the root mean square value; superimposing the mean square signals of the myoelectric signal in each channel to obtain a superimposed signal; calculating the average value of the superimposed signal; calculating an active segment segmentation threshold according to the average value of the superimposed signal and a preset coefficient; setting the time period in the superimposed signal that is greater than the active segment segmentation threshold as the active segment; obtaining the corresponding start time and end time according to the active segment.

[0118] The active segment signal is the signal in the actually executed interval of the action in the continuously collected signal, which is an intuitive reflection of the gesture action signal and also a basic unit in pattern recognition. The goal of active segment detection is to determine the start time and end time of the execution of the gesture action signal, and the correctness of its judgment is the key to constructing and recognizing the gesture recognition model.

[0119] The present invention uses an active segment segmentation method based on RMS (Root Mean Square). Its basic idea is to suppress the magnitude of the myoelectric data in the non-active segment while increasing the value of the data in the active segment. Substantially, it is a short-time energy detection method. Its algorithm steps are as follows:

[0120] S301: Calculate the mean square signal rms_EMG of the myoelectric signal in each channel. For the time series S k (i) of the myoelectric data in each channel, use a window W with a length of n = 90, and calculate the root mean square value in each window according to the following formula as the value of the mean square signal at time i; slide the window W along the direction of time t, and calculate the root mean square value after each slide in turn to obtain the time series signal S k_rms of the mean square signal rms_EMG of each channel.

[0121]

[0122] where k is the kth channel, T is the signal length, and n is the window size. The waveform diagram of the myoelectric signal is asFigure 7a As shown, the corresponding mean square signal waveform diagram is as Figure 7b shown.

[0123] S302: Superimpose the mean square signals of the EMG signals in each channel to obtain a superimposed signal. Specifically, superimpose the rms_EMG data of the mean square signals of all channels to obtain the superimposed time series data sum_rms_EMG.

[0124]

[0125] where N is the number of channels, T is the length of the time series, and S k_rms (i) is the rms time series signal of the k-th channel.

[0126] S303: Calculate the average value of the superimposed signal, and calculate the active segment segmentation threshold thr according to the average value of the superimposed signal and the preset coefficient mean_coef.

[0127]

[0128] where mean_coef is a preset coefficient with a value between 0 and 1, used to adjust the size of the active segment segmentation threshold thr. The superimposed signal and the active segment segmentation threshold thr are as Figure 8 shown.

[0129] S304: Set the time period in the superimposed signal that is greater than the active segment segmentation threshold as the active segment.

[0130] S m potential = [S sum_rms > thr], m = 1,..., M

[0131] where M is the number of potential active segments. The schematic diagram of the potential active segments is as Figure 9 shown, and the dotted line in the figure is the active segment segmentation line.

[0132] S305: Obtain the corresponding start time and end time according to the active segment, and each active segment corresponds to a start time and an end time.

[0133] In an embodiment of the present invention, before determining the type of gesture action according to the EMG signal and the inertial navigation signal, the method further includes: filtering the EMG signal through an eighth-order Butterworth filter; filtering the EEG signal through a low-pass filter with a cut-off frequency of 30 Hz and a high-pass filter with a cut-off frequency of 0.1 Hz.

[0134] The surface electromyogram (sEMG) signal is a weak non-linear high internal resistance signal formed by the spatio-temporal superposition at the detection electrodes after the action potential sequences formed by multiple motor units participating in muscle activities are filtered by the volume conductor composed of skin, subcutaneous tissue, muscle, etc. It has the following characteristics:

[0135] Weakness: The amplitude generally fluctuates between 0 - 5 mV and is easily affected by volume conductors such as the skin.

[0136] Low frequency: The frequency spectrum is distributed from 0 - 500 Hz, and the effective signal frequency is mainly within 200 Hz.

[0137] Regularity: When the limb assumes different postures, the time-domain characteristics of the generated myoelectric signals are roughly similar.

[0138] Alternation: The amplitude of the myoelectric signal is closely related to muscle force and the state of neuromuscular function.

[0139] In the present invention, an eighth-order Butterworth filter is used to perform band-pass filtering (10 - 100 Hz) and 50 Hz band-stop filtering (the stop-band bandwidth is 2 Hz) on the myoelectric signal. The original myoelectric signal and the spectrogram are as Figure 10 shown. It can be seen that there is a large amount of low-frequency and high-frequency noise in the original signal, and the useful myoelectric signal is completely submerged in the noise data. After filtering the original myoelectric signal with a 10 - 100 HZ band-pass filter and a 50 HZ band-stop filter, and performing a fast Fourier transform on the filtered signal, the waveform and spectrogram of the filtered myoelectric signal are as Figure 11 shown. By comparing Figure 10 and Figure 11 , it can be seen that the low-frequency noise, high-frequency noise, and 50 HZ power frequency noise of the myoelectric signal are significantly filtered out, and the gesture myoelectric information is well extracted.

[0140] According to different frequency ranges, electroencephalogram (EEG) signals can generally be divided into five bands: the Delta band (frequency less than 4 Hz), the Theta band (frequency in the range of 4 - 8 Hz), the Alpha band (frequency in the range of 8 - 13 Hz), the Beta band (frequency in the range of 13 - 30 Hz), and the Gamma band (frequency greater than 30 Hz). Among them, when a person is in a waking state and has a relatively high level of attention, the main bands in the EEG components are the Beta band with a frequency of 13 - 30 Hz and the Alpha band with a frequency of 8 - 12 Hz; when fatigued and the attention level drops, the main band in the EEG components is the Theta band with a frequency of 4 - 8 Hz. As a special electrophysiological signal, EEG signals are highly sensitive and are extremely likely to be submerged in noise signals during the acquisition process. Common noise sources include electrooculogram, electrocardiogram, electromyogram, acquisition equipment, etc. In this project, a low-pass filter with a cut-off frequency of 30 Hz and a high-pass filter with a cut-off frequency of 0.1 Hz are used to filter the EEG signals.

[0141] Both electromyogram (EMG) signals and EEG signals are weak electrophysiological signals, and there is a lot of interference during the acquisition process. Performing preprocessing operations such as filtering on them can improve the accuracy of gesture recognition and the robustness of the system.

[0142] In an embodiment of the present invention, the obtaining of the operation instruction corresponding to the gesture action includes: obtaining the decoded data corresponding to the gesture action; querying the corresponding operation instruction from a preset gesture coding library according to the decoded data.

[0143] The present invention presets a gesture coding library. After determining the gesture action, the gesture action is decoded, and the corresponding operation instruction is queried from the gesture coding library according to the decoded data. For example, when the arm is stretched out flat, it corresponds to the unmanned vehicle moving forward. The corresponding relationship between the gesture action and the operation instruction can be set according to the functions of the unmanned vehicle.

[0144] As Figure 12 shown, an embodiment of the present invention introduces a gesture recognition device 200, and the device 200 includes:

[0145] A first acquisition module 201, configured to acquire EEG signals.

[0146] A second acquisition module 202, configured to acquire EMG signals.

[0147] A third acquisition module 203, configured to acquire inertial navigation signals;

[0148] A first determination module 204, configured to determine the gesture action according to the EMG signals and the inertial navigation signals;

[0149] A first judgment module 205, configured to judge whether the gesture action is a valid action according to the EEG signals;

[0150] A fourth acquisition module 206, configured to obtain an operation instruction corresponding to the gesture action when the gesture action is a valid action.

[0151] By adding electroencephalogram signals during the gesture recognition process and judging whether the gesture action is a valid action according to the electroencephalogram signals, the present invention can eliminate unconscious gesture actions and reduce misoperations.

[0152] As Figure 13 shown, an embodiment of the present invention introduces an electronic device 300, including a processor 301 and a memory 302. The memory 302 stores a program or instruction that can run on the processor 301. When the program or instruction is executed by the processor 301, the steps of the gesture recognition method described in any embodiment of the present invention are implemented.

[0153] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0154] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium used herein is not construed as being a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0155] The computer-readable program instructions described herein can be downloaded to various computing / processing devices from a computer-readable storage medium or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0156] The computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer-readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present invention.

[0157] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0158] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises a manufacture including instructions that implement various aspects of the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0159] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes of the flowchart and / or block diagram.

[0160] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a segment of code, or a portion of an instruction, and the module, segment of code, or portion of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the boxes may occur out of the order noted in the figures. For example, two consecutive boxes may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each box of the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions. As is well known to those skilled in the art, implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

[0161] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A gesture recognition method, characterized in that, the method includes: Obtaining electroencephalogram (EEG) signals, electromyogram (EMG) signals, and inertial navigation signals; Determining a gesture action according to the EMG signal and the inertial navigation signal; Judging whether the gesture action is a valid action according to the EEG signal; If the gesture action is a valid action, obtaining an operation instruction corresponding to the gesture action and performing a corresponding action according to the operation instruction; wherein, the characteristic value of the EMG signal is obtained through the following steps: For the time series of EMG signal data for each channel , a window W with a length of n = 90 is used, and according to the formula , the root mean square value within each window is calculated as the value of the mean square signal at time i; the window W is slid along the direction of time t, and the root mean square value after each slide is calculated in turn to obtain the time series signal of the mean square signal for each channel ; where k represents the k-th channel, T is the signal length, and n is the window size; Slide the window along the time t direction, and according to the formula calculate the mean absolute value feature MAV(k) of the data of each channel within each window, where M is the number of sliding windows; Horizontally splice the features of all windows together to obtain a feature matrix ; where Chan is the number of channels and M is the number of sliding windows; Calculate the mean value of the feature matrix. According to the formula , the feature vector V MAV ( chan *1) is used as the time-domain feature of the EMG signal.

2. The method according to claim 1, characterized in that, the gesture action includes a hand action and an arm action, and the judging the type of the gesture action according to the EMG signal and the inertial navigation signal includes: Determining a hand action according to the EMG signal; Determining an arm action according to the inertial navigation signal; Determining the gesture action according to the hand action and the arm action.

3. The method according to claim 2, characterized in that, the determining a hand action according to the EMG signal includes: Calculating a first characteristic value, a second characteristic value, and a third characteristic value of the EMG signal, wherein the first characteristic value is the average value of the time-domain characteristics of the EMG signal on the first channel, the second characteristic value is the average value of the time-domain characteristics of the EMG signal on the second channel, and the third characteristic value is the average value of the time-domain characteristics of the EMG signal on the third channel; If the first characteristic value is greater than a first threshold, the first characteristic value is greater than the second characteristic value, and the first characteristic value is greater than the third characteristic value, judging that the hand action is an eversion action; If the second characteristic value is greater than a second threshold, the second characteristic value is greater than the first characteristic value, and the second characteristic value is greater than the third characteristic value, judging that the hand action is a fist-clenching action; If the third characteristic value is greater than a third threshold, the third characteristic value is greater than the first characteristic value, and the third characteristic value is greater than the second characteristic value, judging that the hand action is an inversion action.

4. The method according to claim 2, characterized in that, the inertial navigation signal includes a first inertial navigation signal corresponding to the upper arm and a second inertial navigation signal corresponding to the lower arm, and the determining an arm action according to the inertial navigation signal includes: Obtaining a first acceleration value in three preset directions according to the first inertial navigation signal, and obtaining a corresponding upper arm posture according to the first acceleration value; Obtaining a second acceleration value in three preset directions according to the second inertial navigation signal, and obtaining a corresponding lower arm posture according to the second acceleration value; Determining the arm action according to the upper arm posture and the lower arm posture.

5. The method according to claim 1, characterized in that, the judging whether the gesture action is a valid action according to the EEG signal includes: Obtaining a start time and an end time corresponding to the gesture action in the EMG signal; Obtaining a target signal, where the target signal is the signal of the EEG signal between the start time and the end time; Performing frequency decomposition on the target signal by a fast Fourier transform method to obtain the energy of the target signal in each preset frequency band; Calculate the attention value according to the energy of the target signal in each preset frequency band and the total energy of the target signal according to a preset algorithm; If the attention value exceeds the threshold, determine that the gesture action is a valid action.

6. The method according to claim 5, wherein, The obtaining the start time and end time corresponding to the gesture action in the myoelectric signal includes: Obtain the sub-signals of the myoelectric signal in each channel; Calculate the root mean square value of the sub-signal in each sampling window, and calculate the mean square signal of the myoelectric signal in each channel according to the root mean square value; Superimpose the mean square signals of the myoelectric signal in each channel to obtain a superimposed signal; Calculate the average value of the superimposed signal; Calculate the active segment segmentation threshold according to the average value of the superimposed signal and a preset coefficient; Set the time period in the superimposed signal that is greater than the active segment segmentation threshold as the active segment; Obtain the corresponding start time and end time according to the active segment.

7. The method according to claim 1, wherein, Before determining the type of the gesture action according to the myoelectric signal and the inertial navigation signal, the method further includes: Perform filtering processing on the myoelectric signal through an eighth-order Butterworth filter; Perform filtering processing on the electroencephalogram signal through a low-pass filter with a cut-off frequency of 30 Hz and a high-pass filter with a cut-off frequency of 0.1 Hz.

8. The method according to claim 1, wherein, The obtaining the operation instruction corresponding to the gesture action includes: Obtain the decoding data corresponding to the gesture action; Query the corresponding operation instruction from a preset gesture coding library according to the decoding data.

9. A gesture recognition device, wherein, The device includes: A first acquisition module for acquiring an electroencephalogram signal; A second acquisition module for acquiring a myoelectric signal; A third acquisition module for acquiring an inertial navigation signal; A first determination module for determining a gesture action according to the myoelectric signal and the inertial navigation signal; A first judgment module for judging whether the gesture action is a valid action according to the electroencephalogram signal; A fourth acquisition module for acquiring an operation instruction corresponding to the gesture action when the gesture action is a valid action; wherein, the characteristic value of the myoelectric signal is obtained through the following steps: For the time series of EMG signal data for each channel , a window W with a length of n = 90 is used, and according to the formula , the root mean square value within each window is calculated as the value of the mean square signal at time i; the window W is slid along the direction of time t, and the root mean square value after each slide is calculated in turn to obtain the time series signal of the mean square signal for each channel ; where k represents the k-th channel, T is the signal length, and n is the window size; Slide the window along the time t direction, and according to the formula calculate the mean absolute value feature MAV(k) of the data of each channel within each window, where M is the number of sliding windows; Horizontally splice the features of all windows together to obtain a feature matrix ; where Chan is the number of channels and M is the number of sliding windows; Calculate the mean value of the feature matrix, and according to the formula , calculate the obtained feature vector V MAV ( chan *1) as the time-domain feature of the EMG signal.

10. An electronic device, wherein, It includes a processor and a memory, and the memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the gesture recognition method according to any one of claims 1-8 are implemented.

Images