Separated wearable myoelectric bracelet device for real-time gesture recognition

Abstract

The invention relates to the technical field of wearable equipment, and provides a separated wearable myoelectricity bracelet device for real-time gesture recognition, which comprises a myoelectricity acquisition wrist strap and a main controller, the main controller is detachably connected with the myoelectricity acquisition wrist strap; the myoelectricity acquisition wrist strap comprises six data acquisition modules; the main controller comprises a power management module, a man-machine interaction module, a minimum system module and a wireless link module; the minimum system module comprises a program guide unit, a data acquisition unit, a preprocessing unit and a local real-time reasoning unit; and the local real-time reasoning unit is used for performing gesture recognition on the preprocessed frame of data through a trained convolutional neural network model. According to the method, the spatial resolution of data is enhanced by adopting multi-channel surface electromyogram signal acquisition, and the accuracy and reliability of gesture recognition are improved by combining an efficient signal processing algorithm and a deep learning model.

Description

Technical Field

[0001] The present invention relates to the technical field of wearable devices, and in particular to a detachable wearable myoelectric wristband device for real-time gesture recognition. Background Art

[0002] In recent years, wearable devices have been widely used in the fields of biomedical engineering, intelligent control and human-computer interaction, especially in gesture recognition scenarios. Surface electromyography (sEMG), as a non-invasive muscle activity electrical signal, can not only reflect the movement characteristics of hand muscles, but also provide high-resolution movement information, thus having important value in limb rehabilitation training, intelligent prosthetic control, virtual reality interaction, etc.

[0003] At present, a variety of gesture recognition devices based on sEMG signals have been studied and developed. These devices usually use multiple electrode arrays to collect signals and realize gesture recognition through feature extraction and pattern classification.

[0004] However, the existing technology and equipment still have the following major defects:

[0005] 1. Lack of integrated, low-cost solutions:

[0006] Most existing devices fail to effectively combine the requirements of low cost, easy wearability, and full real-time. These devices usually rely on independent signal acquisition modules and perform signal preprocessing and gesture recognition through external devices. They lack solutions that integrate signal acquisition, preprocessing, and real-time reasoning functions. Such devices not only increase system complexity, but also reduce the portability of the device, limiting its promotion in daily applications.

[0007] 2. Separation of signal acquisition and reasoning process:

[0008] Many devices usually separate the acquisition of sEMG signals from real-time reasoning. Signal acquisition is usually completed by portable devices, while real-time reasoning needs to rely on large computing devices (such as workstations or high-performance computing platforms). This model not only increases the dependence on external hardware, but also limits the application capabilities of the device in low-power, resource-constrained environments. In addition, due to the delay in the signal transmission process, the real-time requirements are often difficult to meet.

[0009] 3. The inference model is not robust enough and cannot be applied in multiple scenarios:

[0010] Existing gesture recognition models mostly use traditional machine learning or convolutional neural networks (CNN), but the model design is relatively simple and has poor adaptability to different gestures and differences in electromyographic signals between individual users. At the same time, these models often have degraded performance in noisy environments and are difficult to meet the high accuracy requirements in practical applications. Summary of the invention

[0011] The present invention mainly solves the problems of large size, inconvenience in wearing, signal susceptibility to interference, high recognition delay, etc. in the prior art electromyographic signal acquisition equipment, and proposes a separate wearable electromyographic bracelet device for real-time gesture recognition. The device improves the flexibility of the system through modular design, uses multi-channel surface electromyographic signal acquisition to enhance the spatial resolution of the data, combines efficient signal processing algorithms and deep learning models to improve the accuracy and reliability of gesture recognition, and optimizes the hardware circuit and wireless transmission scheme to reduce power consumption and improve real-time performance.

[0012] The present invention provides a detachable wearable myoelectric wristband device for real-time gesture recognition, comprising: an electromyographic acquisition wristband and a main controller; the main controller is detachably connected to the electromyographic acquisition wristband;

[0013] The electromyography acquisition wristband includes six data acquisition modules; the data acquisition modules acquire the original electromyography signals when the user makes different gestures;

[0014] The main controller includes: a power management module, a human-computer interaction module, a minimum system module and a wireless link module;

[0015] The minimum system module includes: a program guiding unit, a data acquisition unit, a preprocessing unit and a local real-time reasoning unit;

[0016] The program guide unit is used to adjust or replace the structure or parameters of the convolutional neural network model running in the minimum system module. When the device is started, the wireless link will be initialized first, and an attempt will be made to establish a connection with the PC. After the connection is successfully established, if the convolutional neural network model needs to be updated, the structure and parameters of the convolutional neural network model to be updated will be received in the form of a data packet through the wireless link module. After the reception is completed, the convolutional neural network model will be automatically unpacked and updated to complete the adjustment of the structure or parameters.

[0017] The data acquisition unit is used to send the original electromyographic signal data collected by the multiple data acquisition modules to the preprocessing unit;

[0018] The preprocessing unit is used to divide the collected original electromyographic signal data into a frame of data through a sliding window, and filter out interference components that do not belong to the electromyographic signal frequency from the frame of data through a filter;

[0019] The local real-time reasoning unit is used to perform gesture recognition on a frame of preprocessed data through a trained convolutional neural network model.

[0020] Preferably, the six data acquisition modules are evenly arranged around the arm, and adjacent data acquisition modules are connected by elastic ropes;

[0021] The six data acquisition modules are a signal aggregation board and five independent acquisition boards; the signal aggregation board and the five independent acquisition boards are both provided with differential sampling electrodes;

[0022] The five independent acquisition boards are connected to the signal summary board via flexible cables and FPC connectors.

[0023] Preferably, the signal aggregation board and the independent acquisition board both include an instrument operational amplifier circuit and a secondary amplifier circuit;

[0024] There are three first magnets and two rows of 1×5 array first metal contacts on the signal aggregation board;

[0025] Three second magnets and two rows of second metal contacts are correspondingly arranged below the main controller.

[0026] Preferably, a filtering module is provided on each data acquisition module;

[0027] The filtering module is used to filter the signals that do not belong to the muscle electrical signal frequency by using the high-pass / low-pass filter formed by the RC circuit during data collection.

[0028] Preferably, the power management module includes a power selection circuit and a lithium battery charge and discharge management circuit, the power selection circuit is responsible for selecting between the lithium battery power source and the USB power source; the lithium battery charge and discharge management circuit is responsible for charging the lithium battery using the USB power source;

[0029] The human-computer interaction module includes a display screen and a three-way dial switch.

[0030] Preferably, the working process of the pre-processing unit includes steps 1.1 to 1.3:

[0031] Step 1.1, perform sliding window segmentation on the original electromyographic signal data;

[0032] Step 1.2: Remove baseline drift, perform bandpass filtering and 50 Hz notch filtering on the electromyographic signal data. The following formula (1) is the mathematical formula for removing baseline drift:

[0033]

[0034] Where x is the current data to remove the baseline drift, x i is the i-th data of a channel in a window, N is the window length, x clean is the processed data point;

[0035] The frequency response mathematical formulas of the bandpass filter and notch filter are shown in the following formulas (2) and (3):

[0036]

[0037] Step 1.3, perform short-time Fourier transform on the electromyographic signal data;

[0038] The mathematical formula of short-time Fourier transform is shown in the following formula (4):

[0039]

[0040] Among them, x(τ) is the original signal, ω(τ-t) represents the time window function, and e -j2πfτ is the complex exponential term in the Fourier transform.

[0041] Preferably, the training process of the convolutional neural network model includes steps 2.1 to 2.3:

[0042] Step 2.1, using an electromyographic wristband to collect electromyographic data of gestures commonly used by different participants to form an electromyographic data set;

[0043] Step 2.2: Preprocess the data on the PC and build a convolutional neural network model;

[0044] Step 2.3: Use the electromyography dataset to train the constructed convolutional neural network model, burn the trained convolutional neural network model to the main controller via wired mode or convert it into a data format that can be unpacked by the main controller, and then transmit it wirelessly via OTA mode;

[0045] In the process from step 2.1 to step 2.3, the construction and training process of the entire convolutional neural network model structure are implemented under the Pytorch deep learning framework. The loss function uses the cross entropy loss function, the optimizer is the Adam optimizer, and the validation loss is used as the monitoring indicator. The scheduler improves the learning rate at a ratio of 0.2. The mathematical representation of the cross entropy loss function is shown in the following formula (5):

[0046]

[0047] Among them, y k ∈{0, 1} is the true label of the kth class, is the predicted probability of the kth class, which satisfies

[0048] Preferably, the minimum system module in the main controller is transplanted by transplanting the trained convolutional neural network model in a wired or wireless manner; for the wired transplantation method, the program is directly burned into the minimum system module; during wireless transplantation, the main controller can start or shut down the device through the three-way toggle switch of the human-computer interaction part, and the transplantation specifically includes steps 3.1 to 3.3:

[0049] Step 3.1, start the main controller device;

[0050] Step 3.2, the main controller initializes the relevant peripherals and wireless link module, and then the program boot unit attempts to connect to the external PC end. If the connection is successful, it attempts to communicate, and the communication is accompanied by model modification related information;

[0051] Step 3.3: If the convolutional neural network model in the minimum system module needs to be modified or updated, the data packet is received through the wireless link module, and then the convolutional neural network model structure is updated after unpacking. If the convolutional neural network model does not need to be modified or updated, the counter in the minimum system module is incremented by one. If the user does not operate for a long time and the counter exceeds the threshold, the device is automatically shut down.

[0052] Preferably, the local real-time reasoning unit performs gesture recognition, including steps 4.1 to 4.2:

[0053] Step 4.1, the 6-channel ADC of the data acquisition unit works in the DMA loop acquisition mode triggered by the 1ms software timer overflow event. After the timer is started, the 6-channel electromyographic signal data will be continuously transferred from the ADC peripheral to the data buffer at a sampling rate of 1KHz;

[0054] Step 4.2: When the data buffer stores enough data for a window of 140ms, take out 6×140 data from the data buffer to form a frame of raw data. After the raw data is preprocessed by the preprocessing unit, it is sent to the convolutional neural network model of the local real-time inference unit to run inference and obtain the inference result.

[0055] Preferably, the main controller uses a multi-vote voting method to determine the inference result to enhance the accuracy of real-time gesture recognition, including steps 5.1 to 5.3:

[0056] Step 5.1: First, it takes 140ms to collect a frame of data, 5ms to preprocess the data, and 15ms to perform inference to obtain the inference result of a single ticket, and store the inference result in the result buffer.

[0057] Step 5.2: Since the EMG data is collected using the DMA loop collection method, it does not occupy the core computing power. Therefore, during the first data preprocessing and inference, about 20ms of new data has been stored in the data cache. The input frame data of the last single-ticket inference is slid forward 20ms, and then the newly collected data is used to complete the frame.

[0058] Step 5.3: Preprocess the new frame data and input it into the convolutional neural network model to obtain the inference result, which is also stored in the result buffer. Repeat steps 5.1 to 5.2 until the number of results in the buffer is the same as the desired number of votes.

[0059] The present invention provides a separate wearable electromyographic bracelet device for real-time gesture recognition, which has innovative designs in hardware architecture, signal processing, and reasoning algorithms. The specific features include:

[0060] 1. Efficient deep learning inference model improves recognition accuracy and robustness. It adopts a convolutional neural network model with multi-feature fusion, integrates time domain features and time-frequency domain features, makes full use of electromyographic signal information, and improves the recognition ability of the model. The introduction of the attention mechanism enables the model to focus on the key areas of the signal and maintain high performance under conditions of individual differences and noise interference. In the offline state, the model achieved an effective recognition accuracy of 98.62% on 7 common gestures, and was verified on multiple data sets such as Myo and Ninapro DB5, all showing good generalization ability.

[0061] 2. Innovative modular hardware design improves flexibility and scalability. The device consists of an acquisition module and an inference module, which are connected and detachable through magnetic contacts, so that the device can be flexibly split and replaced to adapt to different computing needs. The acquisition module is mainly responsible for the real-time acquisition and preprocessing of sEMG signals, while the inference module is responsible for gesture classification. This design facilitates future upgrades of the inference module to adapt to more complex models without replacing the entire system. The main controller and the electromyography acquisition wristband are connected by magnetic contacts, which can reduce signal transmission delay while maintaining the simplicity and reliability of the device.

[0062] 3. Low power consumption and strong real-time performance, meeting the needs of wearable devices. Using wired transmission instead of wireless communication avoids data delay and instability during wireless high-speed transmission, and improves the real-time performance of data transmission. All power is provided by the main controller, and the myoelectric wristband does not require an independent power supply, which reduces energy consumption and improves portability. The main controller adopts a detachable design. When the power is low, the user can remove the main controller separately for charging without affecting the wearing experience of the myoelectric wristband.

[0063] 4. Efficient data acquisition and integrated signal transmission design. The electromyography wristband integrates six acquisition circuit boards, corresponding to six sEMG channels, which are transmitted to the main controller through FPC flexible connection and magnetic metal contacts to achieve efficient and stable data transmission. The acquisition circuit adopts 1100 times signal amplification, so that the weak sEMG signal can be stably collected by the ADC of the microcontroller unit. The magnetic contacts between the signal aggregation board and the main controller are used for both data transmission and power supply, reducing unnecessary wiring and making the device more compact and beautiful.

[0064] 5. Applicable to a variety of application scenarios, with strong scalability. The device can be widely used in virtual reality, human-computer interaction, intelligent control, limb rehabilitation and other fields to meet the needs of myoelectric gesture recognition in different scenarios. In the future, the computing power of the main controller can be upgraded to adapt to more complex deep learning models without replacing the acquisition module, thereby improving the long-term applicability of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of the module composition of a separate real-time gesture recognition wearable myoelectric wristband device in an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of the electromyography acquisition wristband in an embodiment of the present invention;

[0067] Figure 3 This is a principle block diagram of a separate real-time gesture recognition wearable myoelectric wristband device in an embodiment of the present invention;

[0068] Figure 4 Schematic diagram of signal flow of the electromyography acquisition wristband in an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of a separable design of a main controller and a signal aggregation board in an embodiment of the present invention;

[0070] Figure 6 is a training flow chart of a convolutional neural network model in an embodiment of the present invention;

[0071] Figure 7 A multi-feature fusion convolutional neural network model structure diagram with an attention mechanism added to an embodiment of the present invention;

[0072] Figure 8 7 types of gesture classification result confusion matrix commonly used in the embodiments of the present invention;

[0073] Fig. 9 The present invention is a flowchart of the real-time operation of a separate real-time gesture recognition wearable electromyographic bracelet device in an embodiment of the present invention.

[0074] Fig.10 Schematic diagram of seven common types of gestures in the embodiments of the present invention. DETAILED DESCRIPTION

[0075] In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

[0076] like Figure 1 As shown, an embodiment of the present invention provides a detachable wearable myoelectric wristband device for real-time gesture recognition, comprising: an electromyographic acquisition wristband and a main controller 1; the main controller 1 is detachably connected to the electromyographic acquisition wristband.

[0077] The electromyographic acquisition wristband includes six data acquisition modules; the six data acquisition modules are evenly arranged around the arm, and adjacent data acquisition modules are connected by elastic ropes 9; the data acquisition modules acquire original electromyographic signals when the user makes different gestures.

[0078] Specifically, in this embodiment, if Figure 2 As shown, the six data acquisition modules are a signal aggregation board 2 and five independent acquisition boards 3; the signal aggregation board 2 and the five independent acquisition boards 3 are both provided with differential sampling electrodes 4. One signal aggregation board 2 and five independent acquisition boards 3 correspond to six channels of surface electromyography data respectively.

[0079] like Figure 3 As shown in the upper left part, the signal aggregation board 2 and the independent acquisition board 3 both include an instrument operational amplifier circuit and a secondary amplifier circuit, and both have the ability to independently collect electromyographic signals. The instrument operational amplifier circuit amplifies the electromyographic differential signal by about 334 times, and the secondary amplifier circuit amplifies the already amplified signal by 3.4 times again, so that it reaches the acquisition range of the ADC (analog-to-digital converter) in the main controller 1. However, the difference is that the five independent acquisition boards 3 are connected to the signal aggregation board 2 through flexible cables and FPC connectors, and the signals collected by the independent acquisition boards 3 are aggregated to the signal aggregation board 2. The overall structure of the electromyographic acquisition wristband and the flow of signals are shown in the figure. Figure 4 As shown in the content, each data acquisition module has a reserved interface for signal transmission. The signal aggregation board 2 uses flexible cables and adopts a strategy of direct external connection and internal offset connection of each independent acquisition board 3 to aggregate the signals of the three independent acquisition boards 3 on the left and the two independent acquisition boards 3 on the right into itself.

[0080] Furthermore, the differential sampling electrodes 4 in the above six data acquisition modules all use surface dry electrodes. The gold plating operation ensures that the electrodes still have long-term durability when exposed to skin moisture. Compared with wet electrodes, dry electrodes do not require cleaning and unnecessary patch operations, which further improves the wearability of users. The size of the dry electrode is 6mm×10mm, and the three dry electrodes are evenly spaced, all 3mm, which reduces the error caused by the difference in electrode position. The data acquisition module directly performs a differential amplification of the original electromyographic signal by about 1100 times through the instrument amplifier circuit, and the amplified signal can be directly collected by the main controller 1.

[0081] The signal aggregation board 2 is detachably connected to the main controller 1, that is, the main controller 1 is detachably connected to the myoelectric acquisition wristband. Figure 5 As shown, there are three first magnets 5 above the signal summary board 2 in the above-mentioned electromyography acquisition wristband. The first magnets 5 are 2mm×10mm strong Ru magnets. At the same time, two rows of 1×5 arrays of first metal contacts 6 are designed. In addition, the bottom of the main controller 1 is designed in the same layout as the signal summary board 2. Three second magnets 8 and two rows of second metal contacts 7 are correspondingly arranged below the main controller 1 to facilitate the magnetic connection of the magnets and the contact of the metal contacts. The first metal contact 6 and the second metal contact 7 represent VCC, GND and electromyography signals of three different channels from top to bottom.

[0082] A filtering module is provided on each data acquisition module, and the filtering module is used to filter the signals that do not belong to the muscle electrical signal frequency by using a high-pass / low-pass filter formed by an RC circuit during data acquisition.

[0083] The main controller 1 includes: a power management module, a human-computer interaction module, a minimum system module and a wireless link module.

[0084] The power management module includes a power selection circuit and a lithium battery charge and discharge management circuit. The power selection circuit is responsible for selecting between the lithium battery power supply and the USB power supply, giving priority to using the USB power supply to save lithium battery energy; the lithium battery charge and discharge management circuit is responsible for using the USB power supply to charge the lithium battery, or preventing over-discharge problems caused by the device being powered by a lithium battery, thereby damaging the lithium battery.

[0085] Specifically, Figure 3As shown in the figure, the power selection circuit is constructed using a diode B5817W and a PMOS tube SI2301. Through the switch circuit composed of the diode and the MOS tube, when there is USB power, the subsequent VCC is directly connected to the USB power, and the power supply of the entire device is provided by VCC after LDO voltage stabilization; when there is no USB power, but there is lithium battery power, the subsequent VCC is connected to the positive electrode of the lithium battery, and the subsequent power supply of the entire device is provided by the lithium battery. The purpose of saving lithium battery energy is guaranteed. The charge and discharge management circuit of the lithium battery is managed by the TP4065 chip, which ensures that the USB can not only power the entire device, but also charge the lithium battery, while avoiding the problem of over-discharge of the lithium battery and extending the service life of the lithium battery.

[0086] The human-machine interaction module is electrically connected to the minimum system module, and includes a display screen and a three-way dial switch. The display screen is, for example, a 1.3-inch LCD display screen. The display screen and the dial switch can enhance the wearability and practicality of the device. The dial switch allows the wearer to control the entire device, and the LCD allows the wearer to understand the various states of the current device, data waveforms, and gesture recognition results.

[0087] The wireless link module is electrically connected to the minimum system module. The wireless link module provides the possibility of wireless connection between the myoelectric wristband and other devices, and the gesture recognition results can be easily transmitted to the controlled device through the wireless link, so as to control the controlled device to make different responses.

[0088] The minimum system module is the core and brain of the entire device, running the real-time working logic of the entire device. It mainly includes four parts: program boot unit, data acquisition unit, preprocessing unit and local real-time reasoning unit.

[0089] Specifically, Figure 3 As shown, the minimum system module uses STMicroelectronics' high-performance processor STM32H743, with a core frequency of up to 480MHz and 1MB of RAM, which provides sufficient computing power for the neural network. In addition, considering that the parameters in the convolutional neural network often require a large amount of storage space, the STM32H743 with 2MB of FLASH space can also store the convolutional neural network parameters running in this prototype well. The entire module is mainly composed of a reset circuit, a power circuit and a clock circuit. The core's role in the entire device is to obtain data, process data and perform real-time reasoning. The wireless link module mainly uses NRF24L01, which is interconnected with the STM32H743 through an SPI bus interface, providing the device with a 2.4GHz wireless link capability.

[0090] The program boot unit is used to adjust or replace the structure or parameters of the convolutional neural network model running in the minimum system module. When the device starts, it will first initialize the wireless link and try to establish a connection with the PC (server). After the establishment is successful, if the convolutional neural network model needs to be updated, the device will receive the structure and parameters of the convolutional neural network model to be updated in the form of a data packet through the wireless link module. After the reception is completed, it will automatically unpack and update the convolutional neural network model to complete the adjustment of the structure or parameters.

[0091] The data acquisition unit is used to send the original electromyographic signal data collected by multiple data acquisition modules to the preprocessing unit. The data acquisition unit mainly uses the metal contacts of the main controller 1 and the signal aggregation board 2. The metal contacts provide a 6-channel muscle electromyographic signal data interface. Through this interface, the ADC acquisition function of the minimum system module can be used to obtain the wrist surface electromyographic signal data when the user moves. The ADC acquisition is triggered by a 1ms timer overflow event. Whenever the timer overflows, a 6-channel data acquisition is performed, so the acquisition frequency is 1KHz.

[0092] The preprocessing unit is used to divide the collected raw electromyographic signal data into a frame of data through a sliding window, and filter out the interference components that do not belong to the electromyographic signal frequency from the frame of data through a filter. The working process of the preprocessing unit is as follows:

[0093] Step 1.1, perform sliding window segmentation on the original electromyographic signal data.

[0094] The sliding window segmentation is as follows: since the sampling frequency of the original 6-channel electromyographic signal data is 1KHz and the maximum input delay time allowed by real-time control is 300ms, and considering that data acquisition, preprocessing and reasoning all require a certain amount of time, a 140ms sliding window is selected, and 114ms of time is overlapped between each window.

[0095] Step 1.2: Remove baseline drift, perform bandpass filtering and 50 Hz notch filtering on the electromyographic signal data.

[0096] Due to the influence of factors such as the inherent noise of the acquisition equipment, changes in skin resistance and changes in electrode position during the electromyography acquisition process, there is often a certain degree of baseline drift in the acquired data. Baseline drift is usually a low-frequency (close to zero frequency) signal. Its existence will cause a strong low-frequency component to be generated in the Fourier transform spectrum, resulting in the effective frequency component of the signal being masked. Therefore, removing the baseline drift can better analyze the frequency component of the real signal. The following formula (1) is a mathematical formula for removing the baseline drift, where x is the current data to remove the baseline drift, x iis the i-th data of a channel in a window, N is the window length, x clean are the processed data points.

[0097]

[0098] Since the frequency of surface electromyographic signals is 0 to 500 Hz, and the main component is between 50 and 150 Hz, in addition, during the electromyographic acquisition process, the human body is also a conductor and will be interfered by various environmental noises (electromagnetic interference, power line noise, etc.). The power supply in my country is 50 Hz, so it may be interfered by the 50 Hz power frequency component. Therefore, through the bandpass filter and the 50 Hz notch filter, the signals that do not belong to the muscle electromyographic signal frequency can be removed as much as possible while retaining the original electromyographic signal. The frequency response mathematical formulas of the bandpass filter and the notch filter are shown in the following formulas (2) and (3):

[0099]

[0100] Step 1.3, perform short-time Fourier transform on the electromyographic signal data.

[0101] Since surface electromyographic signals contain rich frequency information, compared with traditional time domain features (such as RMS, zero crossing, etc.) and frequency domain features (such as average frequency, power spectrum density, etc.), short-time Fourier transform retains both time and frequency information and can more comprehensively describe the characteristics of the signal. Therefore, by calculating the short-time Fourier features of the data after the filter, the neural network can better learn the differences between different gestures. The mathematical formula of short-time Fourier transform is shown in the following formula (4):

[0102]

[0103] Among them, x(τ) is the original signal, w(τ-t) represents the time window function, which is used to intercept the signal near time t, and e -j2πfτ is the complex exponential term in Fourier transform, which is used to project the signal onto frequency f.

[0104] The local real-time reasoning unit is used to perform gesture recognition on a frame of preprocessed data through a trained convolutional neural network model.

[0105] The local real-time inference unit stores the trained convolutional neural network model in advance through OTA (over-the-air download) or wired mode. Model training is to build a convolutional neural network model on the PC side, and use the collected electromyography data set to train the parameters of the convolutional neural network model. The trained convolutional neural network model achieves the optimal recognition effect. The entire training process can be referred to Figure 6As shown in the flowchart, the training process of the convolutional neural network model is as follows:

[0106] Step 2.1: Use the electromyography wristband to collect 7 types of common gestures of different participants (such as Fig.10 The electromyographic data is formed as shown in FIG.

[0107] Step 2.2: Preprocess the data on the PC and build a convolutional neural network model;

[0108] The convolutional neural network model is a multi-feature fusion gesture recognition convolutional neural network with a non-local attention mechanism. Figure 7 As shown in the figure, the neural network consists of two separate branches. The first part inputs the time domain features of the preprocessed original data, and the second part inputs the time-frequency domain features after Fourier transform. The features are extracted through two layers of convolution, then summarized after the attention mechanism, and finally the final classification result of the network is obtained after two layers of fully connected layers.

[0109] Step 2.3: Use the electromyography dataset to train the constructed convolutional neural network model, burn the trained convolutional neural network model to the main controller via wired mode or convert it into a data format that can be unpacked by the main controller, and then transmit it wirelessly via OTA mode;

[0110] In the training process of this step, the construction of the entire convolutional neural network model structure and the training process are implemented under the Pytorch deep learning framework. The loss function uses the cross entropy loss function commonly used in multi-classification tasks. The optimizer is the Adam optimizer, and the validation loss is used as the monitoring indicator. The scheduler improves the learning rate at a rate of 0.2. The mathematical representation of the cross entropy loss function is shown in the following formula (5):

[0111]

[0112] where y k ∈{0,1} is the true label of the kth class, is the predicted probability of the kth class, which satisfies The network was verified offline on three datasets: SC-EMG, Myo, and Ninapro DB5, achieving accuracies of 98.62%, 98.24%, and 67.42%, respectively. The confusion matrix of the seven types of gestures when the best classification accuracy was achieved on the SC-EMG dataset is shown in the figure below: Figure 8 shown.

[0113] Furthermore, the minimum system module in the main controller 1 is transplanted by transplanting the trained convolutional neural network model in a wired or wireless manner. For the wired transplantation method, the program can be directly burned into the minimum system module. For wireless transplantation, the main controller 1 can start or shut down the device through the three-way toggle switch of the human-computer interaction part. The specific process of transplantation is as follows:

[0114] Step 3.1, start the main controller 1 device.

[0115] Step 3.2, the main controller 1 initializes the relevant peripherals and wireless link module, and then the program boot unit attempts to connect to the external PC end. If the connection is successful, it attempts to communicate, and the communication is accompanied by model modification related information.

[0116] Step 3.3: If the convolutional neural network model in the minimum system module needs to be modified or updated, the data packet is received through the wireless link module, and then the convolutional neural network model structure is updated after unpacking. If the convolutional neural network model does not need to be modified or updated, the counter in the minimum system module is incremented by one. If the user does not operate for a long time and the counter exceeds the threshold, the device is automatically shut down.

[0117] Furthermore, the main controller 1 can perform real-time gesture recognition reasoning. The data acquisition unit continuously collects 6-channel electromyographic signal data, and then sends the processed data to the convolutional neural network model of the local real-time reasoning unit to obtain the reasoning result. The local real-time reasoning unit performs gesture recognition, and the real-time operation flow chart can be referred to Fig. 9 The specific process is as follows:

[0118] Step 4.1, the 6-channel ADC of the data acquisition unit works in the DMA loop acquisition mode triggered by the 1ms software timer overflow event. After the timer is started, the 6-channel electromyographic signal data will be continuously transferred from the ADC peripheral to the data buffer at a sampling rate of 1KHz;

[0119] Step 4.2: When the data buffer stores enough data for a window of 140ms, take out 6×140 data from the data buffer to form a frame of raw data. After the raw data is preprocessed by the preprocessing unit, it is sent to the convolutional neural network model of the local real-time inference unit to run inference and obtain the inference result.

[0120] Furthermore, in order to enhance the accuracy of real-time gesture recognition, the main controller 1 uses a multi-vote voting method to determine the inference result to increase the probability of hitting. Normally, a frame of data of the convolutional neural network model needs to be collected through a window of 140ms in length, and then through a preprocessing time of about 5ms, and finally through an AI inference time of 15ms (MCU running at 480MHz) to get the result. After actual testing, it was found that for 7 common gestures, although the recognition accuracy of the offline convolutional neural network model is very high, the accuracy of real-time recognition is only 74.45% due to multi-party interference (power frequency interference, sampling electrode displacement, etc.), so multi-vote voting is used, and the steps are as follows:

[0121] Step 5.1: First, it takes 140ms to collect a frame of data, 5ms to preprocess the data, and 15ms to perform inference to obtain the inference result of a single ticket, and store the inference result in the result buffer.

[0122] Step 5.2: Since the EMG data is collected using the DMA loop collection method, it does not occupy the core computing power. Therefore, during the first data preprocessing and inference, about 20ms of new data has been stored in the data cache. The input frame data of the last single-ticket inference is slid forward 20ms, and then the newly collected data is used to complete the frame.

[0123] Step 5.3: Preprocess the new frame data and input it into the convolutional neural network model to obtain the inference result, which is also stored in the result buffer. Repeat the steps described in 5.1 to 5.2 above until the number of results in the buffer is the same as the desired number of votes.

[0124] Through the above-mentioned multiple voting, for the experimental subjects who had undergone 2 hours of training within a day, among the 7 types of common gestures, the classification accuracy under a single vote was 74.45%, and the delay was 160ms; the classification accuracy under 10 votes was 96.59%, and the delay increased to the edge of 300ms, but still within the real-time control constraints.

[0125] When the present invention is applied, the data collection process of using the myoelectric acquisition wristband of the present invention is as follows:

[0126] 1) Use medical alcohol to wipe the skin surface where the subject needs to wear the electromyography wristband.

[0127] 2) Wear the electromyographic wristband correctly and manually calibrate the position of the differential sampling electrode 4 to avoid the differences in the position of the differential sampling electrode 4 between different collection objects as much as possible, so as to better collect muscle electrical signal data.

[0128] 3) The subject should sit upright, stretch his arms naturally, face the computer screen, and make corresponding movements according to the instructions of the collection software on the screen. The 7 common gestures are as follows: Fig.10 As shown in the content, it includes the natural state of the palm, clenching the fist, opening the palm, ulnar deviation, radial deviation, wrist flexion and wrist extension. The collection wristband will transmit data synchronously during the action, and the collection program will automatically record and save the data in the background.

[0129] During the collection process, each collection subject needs to exert force for 5 seconds when making different gestures, and rest for 3 seconds between each gesture. No electromyographic signals will be recorded during the rest period. Every 7 gestures completed is called a cycle, and 4 cycles of electromyographic collection are called 1 period. Each collection subject needs to complete at least 2 cycles of electromyographic data collection, and select one cycle of data for training set and another cycle of data for verification set. The control logic of the entire collection process is implemented by the main controller.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that modifying the technical solutions described in the aforementioned embodiments, or replacing some or all of the technical features therein by equivalents, does not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A detachable wearable myoelectric bracelet device for real-time gesture recognition, characterized in that: include: An electromyography acquisition wristband and a main controller (1); the main controller (1) is detachably connected to the electromyography acquisition wristband; The electromyography acquisition wristband includes six data acquisition modules; The data acquisition module acquires original electromyographic signals when the user makes different gestures; The main controller (1) comprises: a power management module, a human-computer interaction module, a minimum system module and a wireless link module; The minimum system module includes: a program guiding unit, a data acquisition unit, a preprocessing unit and a local real-time reasoning unit; The program guide unit is used to adjust or replace the structure or parameters of the convolutional neural network model running in the minimum system module. When the device is started, the wireless link will be initialized first, and an attempt will be made to establish a connection with the PC. After the connection is successfully established, if the convolutional neural network model needs to be updated, the structure and parameters of the convolutional neural network model to be updated will be received in the form of a data packet through the wireless link module. After the reception is completed, the convolutional neural network model will be automatically unpacked and updated to complete the adjustment of the structure or parameters. The data acquisition unit is used to send the original electromyographic signal data collected by the multiple data acquisition modules to the preprocessing unit; The preprocessing unit is used to divide the collected original electromyographic signal data into a frame of data through a sliding window, and filter out interference components that do not belong to the electromyographic signal frequency from the frame of data through a filter; The local real-time reasoning unit is used to perform gesture recognition on a frame of preprocessed data through a trained convolutional neural network model.

2. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 1, characterized in that: Six data acquisition modules are evenly arranged around the arm, and adjacent data acquisition modules are connected by elastic ropes (9); The six data acquisition modules are a signal aggregation board (2) and five independent acquisition boards (3); the signal aggregation board (2) and the five independent acquisition boards (3) are both provided with differential sampling electrodes (4); The five independent acquisition boards (3) are connected to the signal aggregation board (2) via flexible cables and FPC connectors.

3. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 2, characterized in that: The signal aggregation board (2) and the independent acquisition board (3) both include an instrument operational amplifier circuit and a secondary amplifier circuit; Three first magnets (5) and two rows of 1×5 arrays of first metal contacts (6) are provided above the signal aggregation plate (2); Three second magnets (8) and two rows of second metal contacts (7) are correspondingly arranged below the main controller (1).

4. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 1, characterized in that: A filtering module is provided on each data acquisition module; The filtering module is used to filter the signals that do not belong to the muscle electrical signal frequency by using the high-pass / low-pass filter formed by the RC circuit during data collection.

5. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 1, characterized in that: The power management module includes a power selection circuit and a lithium battery charge and discharge management circuit. The power selection circuit is responsible for selecting between the lithium battery power source and the USB power source; the lithium battery charge and discharge management circuit is responsible for charging the lithium battery using the USB power source; The human-computer interaction module includes a display screen and a three-way dial switch.

6. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 1, characterized in that: The working process of the pre-processing unit includes steps 1.1 to 1.3: Step 1.1, perform sliding window segmentation on the original electromyographic signal data; Step 1.2: Remove baseline drift, perform bandpass filtering and 50 Hz notch filtering on the electromyographic signal data. The following formula (1) is the mathematical formula for removing baseline drift: Where x is the current data to remove the baseline drift, x i is the i-th data of a channel in a window, N is the window length, x clean is the processed data point; The frequency response mathematical formulas of the bandpass filter and notch filter are shown in the following formulas (2) and (3): Step 1.3, perform short-time Fourier transform on the electromyographic signal data; The mathematical formula of short-time Fourier transform is shown in the following formula (4): Among them, x(τ) is the original signal, w(τ-t) represents the time window function, and e -j2πfτ is the complex exponential term in the Fourier transform.

7. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 6, characterized in that: The training process of the convolutional neural network model includes steps 2.1 to 2.3: Step 2.1, using an electromyographic wristband to collect electromyographic data of gestures commonly used by different participants to form an electromyographic data set; Step 2.2: Preprocess the data on the PC and build a convolutional neural network model; Step 2.3: Use the electromyography dataset to train the constructed convolutional neural network model, burn the trained convolutional neural network model to the main controller via wired mode or convert it into a data format that can be unpacked by the main controller, and then transmit it wirelessly via OTA mode; In the process from step 2.1 to step 2.3, the construction and training process of the entire convolutional neural network model structure are implemented under the Pytorch deep learning framework. The loss function uses the cross entropy loss function, the optimizer is the Adam optimizer, and the validation loss is used as the monitoring indicator. The scheduler improves the learning rate at a ratio of 0.

2. The mathematical representation of the cross entropy loss function is shown in the following formula (5): Among them, y k ∈{0, 1} is the true label of the kth class, is the predicted probability of the kth class, which satisfies 8. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 7, characterized in that: The minimum system module in the main controller (1) is transplanted by transplanting the trained convolutional neural network model in a wired or wireless manner; in the case of wired transplantation, the program is directly burned into the minimum system module; in the case of wireless transplantation, the main controller (1) can start or shut down the device through a three-way toggle switch of the human-computer interaction part, and the transplantation specifically includes steps 3.1 to 3.3: Step 3.1, start the main controller (1) device; Step 3.2, the main controller (1) initializes the relevant peripherals and the wireless link module, and then the program boot unit attempts to connect to the external PC. If the connection is successful, it attempts to communicate, and the communication is accompanied by model modification related information; Step 3.3: If the convolutional neural network model in the minimum system module needs to be modified or updated, the data packet is received through the wireless link module, and then the convolutional neural network model structure is updated after unpacking. If the convolutional neural network model does not need to be modified or updated, the counter in the minimum system module is incremented by one. If the user does not operate for a long time and the counter exceeds the threshold, the device is automatically shut down.

9. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 8, characterized in that: The local real-time reasoning unit performs gesture recognition, including steps 4.1 to 4.2: Step 4.1, the 6-channel ADC of the data acquisition unit works in the DMA loop acquisition mode triggered by the 1ms software timer overflow event. After the timer is started, the 6-channel electromyographic signal data will be continuously transferred from the ADC peripheral to the data buffer at a sampling rate of 1KHz; Step 4.2: When the data buffer stores enough data for a window of 140ms, take out 6×140 data from the data buffer to form a frame of raw data. After the raw data is preprocessed by the preprocessing unit, it is sent to the convolutional neural network model of the local real-time inference unit to run inference and obtain the inference result.

10. The detachable wearable myoelectric bracelet device for real-time gesture recognition according to claim 9, characterized in that: The main controller (1) uses a multi-vote voting method to determine the inference result to enhance the accuracy of real-time gesture recognition, including steps 5.1 to 5.3: Step 5.1: First, it takes 140ms to collect a frame of data, 5ms to preprocess the data, and 15ms to perform inference to obtain the inference result of a single ticket, and store the inference result in the result buffer. Step 5.2: Since the EMG data is collected using the DMA loop collection method, it does not occupy the core computing power. Therefore, during the first data preprocessing and inference, about 20ms of new data has been stored in the data cache. The input frame data of the last single-ticket inference is slid forward 20ms, and then the newly collected data is used to complete the frame. Step 5.3: Preprocess the new frame data and input it into the convolutional neural network model to obtain the inference result, which is also stored in the result buffer. Repeat steps 5.1 to 5.2 until the number of results in the buffer is the same as the desired number of votes.

Images