Surface electromyography signal text conversion method and storage medium

Abstract

The present application relates to a kind of surface myoelectric signal text conversion method, belong to robot technical field;Solve the problem that upper limb amputee cannot realize text input by typing in prior art;The method of the present application comprises the following steps: obtaining the surface myoelectric signal corresponding to character writing gesture, and the surface myoelectric signal is preprocessed;The surface myoelectric signal after pre-processing is input into pre-trained myoelectric model, and the character prediction probability corresponding to surface myoelectric signal is obtained;Myoelectric model is obtained based on pre-constructed myoelectric text conversion dataset training;Based on the character prediction probability corresponding to each gesture output by myoelectric model, word search is carried out through language model, and the text after conversion is obtained;Language model is obtained based on string search model construction.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and storage medium for text conversion of surface electromyography signals. Background Technology

[0002] With the continuous development of communication and electronic information technology, people are using electronic products more and more commonly in their daily lives, work and study. Text input is also a basic function of various electronic products and has become an extremely common way for people to communicate.

[0003] However, upper limb amputees lose the ability to input text through typing due to the loss of fingers, causing significant inconvenience to their lives and studies. Current intelligent prostheses for amputees control hand movements by collecting electromyographic (EMG) signals from the residual limb. Signal processing and pattern recognition algorithms decode the amputee's intentions and then control the bionic hand to perform corresponding operations. Current research on bionic hands focuses on improving mechanical structures and control strategies to design bionic hands that better conform to human mechanics and anatomy, achieving more precise hand movements and providing a better user experience for patients. Building on this, converting EMG signals into text through EMG signal recognition to restore the typing ability of upper limb amputees has become a highly significant research area. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a method and storage medium for text conversion of surface electromyography signals; and solves the problem in the prior art that upper limb amputees cannot achieve text input by typing.

[0005] The objective of this invention is mainly achieved through the following technical solutions:

[0006] On one hand, the present invention provides a method for text conversion of surface electromyography signals, comprising the following steps:

[0007] Acquire the surface electromyography (EMG) signals corresponding to character writing gestures, and preprocess the EMG signals.

[0008] The preprocessed surface electromyography (EMG) signal is input into a pre-trained EMG model to obtain the character prediction probability corresponding to the surface EMG signal; the EMG model is trained based on a pre-constructed EMG text conversion dataset.

[0009] Based on the character prediction probability corresponding to each gesture output by the electromyography model, word search is performed through a language model to obtain the converted text; the language model is constructed based on a string search model.

[0010] Furthermore, the electromyography model includes a SepConv2D layer, a pooling layer, a Dense layer, and an output layer;

[0011] The SepConv2D layer is used to extract features from the input electromyography signal data through convolution operations.

[0012] The pooling layer is used to downsample and reduce the dimension of the output of the SepConv2D layer to obtain a dimension-reduced electromyographic signal with the original electromyographic signal characteristics;

[0013] The Dense layer is used to extract the correlation between the features of the dimensionality-reduced electromyographic signals through nonlinear transformation and map them to the output space;

[0014] The output layer uses the softmax activation function to output the character prediction probability corresponding to each surface electromyography signal.

[0015] Furthermore, the training of the electromyographic model includes:

[0016] Multiple electromyographic (EMG) signals of character writing gestures were collected from multiple users to obtain an initial EMG-text conversion dataset. The character writing gesture EMG signals include the corresponding EMG signals obtained by controlling wrist movements to perform writing actions of multiple characters.

[0017] Data augmentation was performed on the electromyographic signals in the initial text conversion dataset to construct an electromyographic text conversion dataset.

[0018] The electromyography (EMG) text conversion dataset is input into a depthwise separable convolutional network model, and the EMG model is obtained by iteratively updating the model through a loss function.

[0019] Furthermore, the characters include English letters and punctuation marks; the initial electromyography text conversion dataset includes a training set, a validation set, and a test set;

[0020] The data for each session in the training set includes electromyographic signals corresponding to 26 lowercase English letters, commas, and periods;

[0021] The verification set includes at least one session containing multiple common words, as well as comma and period gestures; at least one session contains multiple common phrases, as well as electromyographic signals corresponding to commas and periods.

[0022] Each session in the test set includes multiple text-related electromyographic signals.

[0023] Furthermore, data augmentation can be performed using any of the following methods:

[0024] The waveform is altered to amplify the data, including: multiplying the sampled value of the acquired electromyographic signal by a preset ratio to change the amplitude; interpolating and downsampling the acquired electromyographic signal to change the velocity, thus obtaining the amplified electromyographic signal;

[0025] Data augmentation is performed by perturbing the detection results, including: after obtaining the complete electromyographic activity segment of the electromyographic signal through the electromyographic activity segment detection method, obtaining different activity segments of the electromyographic signal by moving the detection window left and right to obtain the amplified electromyographic signal; or by increasing the judgment threshold of the activity segment to reduce the length of the activity segment; or by decreasing the judgment threshold of the activity segment to increase the length of the activity segment to obtain the amplified electromyographic signal.

[0026] Data augmentation is achieved by modifying the spectrum, including altering the spectrum of the electromyographic signal through time warping, frequency masking, and temporal masking to obtain an amplified electromyographic signal.

[0027] Furthermore, the process of predicting the character probability corresponding to each gesture based on the output of the electromyography model, and then performing word search through a language model to obtain the converted text, includes:

[0028] Save the top m characters with the highest predicted probability for each gesture;

[0029] If no electromyographic signal is detected after a preset time, it is considered a space, and all characters between the spaces are taken as a prediction combination;

[0030] Based on the order of predicted probabilities from largest to smallest, the first m predicted characters corresponding to the first gesture in the space-space string are taken as the root node. Based on the order of each gesture and the predicted probabilities of the first m predicted characters corresponding to each gesture, the predicted characters corresponding to each gesture are combined and used as child nodes. A search tree is built in sequence, and each branch of the search tree is a group of prediction combinations. The prediction combinations are matched in the corpus.

[0031] If a predicted combination exists in the corpus, it is identified as the final word recognition result, and the search tree construction is terminated; otherwise, the search tree continues to be built and the search continues until all possible combinations have been searched; if none of the obtained combinations appear in the corpus, the character combination with the highest predicted probability for each character is taken as the final word prediction result.

[0032] Furthermore, if the length of the predicted combination is greater than a preset length l, then only the first l characters of the predicted combination are extracted as a word.

[0033] Words longer than 1 in the corpus are truncated to match the output of the string search tree; if the string match is successful, the original word of the truncated word is taken as the recognition result.

[0034] Furthermore, the preprocessing of the surface electromyography signal includes:

[0035] The electromyographic signal is divided into frames by a preset sliding window and step size, and the framed electromyographic signal is then filtered.

[0036] Active segment detection is performed on the filtered frame signal to divide the frame signal into active frame signal and rest frame signal;

[0037] Using the detected active frame signal as the unit of feature extraction, the features of the electromyography (EMG) signals of all channels are extracted to obtain the feature data of the EMG signals, thus completing the preprocessing of the surface EMG signals.

[0038] Furthermore, the active segment detection of the filtered framed signal includes:

[0039] The feature matrix is ​​obtained by calculating the root mean square of the subbands of the framed signal using the three-frequency subband filtering method.

[0040] The data in the feature matrix are iterated using the expectation-maximization clustering algorithm. Similar signals in the feature matrix are aggregated. The maximum a posteriori probability estimate of the Gaussian mixture model parameters is obtained based on the aggregated data. The framed signal is divided into active frame signal and resting frame signal based on the a posteriori probability.

[0041] On the other hand, a storage medium is also provided, which stores computer instructions for execution by a computer to implement the aforementioned text conversion method for surface electromyography signals.

[0042] The beneficial effects of this technical solution are:

[0043] This invention constructs an electromyography-text conversion model that combines an electromyography model and a language model by collecting and amplifying electromyographic signals from text characters, thus realizing electromyography-to-text sequence recognition. This invention can restore the ability of amputees to input characters and, combined with gesture recognition, enable them to grasp everyday objects, thereby maximizing the restoration of amputees' self-care abilities in daily life and work.

[0044] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0045] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0046] Figure 1 This is a text conversion method for surface electromyography signals according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram illustrating the mode switching of the intelligent prosthetic system according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the electromyographic activation process according to an embodiment of the present invention;

[0049] Figure 4 This is a hardware module diagram of an identity verification system based on surface electromyography signals according to an embodiment of the present invention. Detailed Implementation

[0050] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0051] This embodiment presents a text conversion method for surface electromyography signals, such as... Figure 1 As shown, it includes the following steps:

[0052] Step S1: Obtain the surface electromyography (EMG) signal corresponding to the character writing gesture, and preprocess the EMG signal;

[0053] Specifically, the text conversion method based on surface electromyography (EMG) signals in this embodiment uses the EMG signals of the residual limb as the signal source for recognizing movement intentions, assisting amputees in their ability to perform handwriting input. During EMG signal acquisition, the electrode pads of a multi-channel EMG signal detection sensor are attached to the surface of the residual limb, ensuring that the absolute position of the electrode pads remains as constant as possible during each acquisition, in order to accurately acquire the EMG signals corresponding to the user's character writing gestures.

[0054] The character writing gesture is a dynamic gesture, and the surface electromyography (EMG) signal corresponding to the character writing gesture needs to be obtained by preprocessing the complete EMG signal detected during the character writing process.

[0055] Through preprocessing, the surface electromyography (EMG) signal is segmented, filtered, active segment detected, and feature extracted to obtain the EMG signal data corresponding to the character writing gesture. Specifically, the preprocessing of the surface EMG signal includes:

[0056] Sliding window framing: The surface electromyography signal is framed through a window with a length of 128ms and a movement step of 64ms to obtain multiple framed signals.

[0057] Signal filtering: The framed signal is filtered out by a Butterworth 4th order 30-500Hz bandpass filter to remove baseline drift and motion artifacts, and then filtered out by a 50Hz notch filter to remove power frequency interference, thus obtaining the framed signal after interference removal.

[0058] Active Segment Detection (EAD): In this embodiment, the electromyography (EMG) signal detection sensor is a multi-channel acquisition device. During active segment detection, the EMG signal from one channel is selected, and its filtered framed signal is used for feature extraction. A feature matrix is ​​formed by calculating the root mean square (RMS) of the sub-bands of the framed signal using a three-frequency sub-band filtering method (80Hz-150Hz, 150Hz-300Hz, 300Hz-450Hz). Since the EMG data in the feature matrix is ​​unlabeled, it is impossible to distinguish between EMG signals and noise beforehand. This embodiment is based on the difference between EMG and noise. A clustering approach using the expectation-maximization algorithm is employed, iterating through multiple rounds to aggregate similar data. Then, the maximum a posteriori probability estimate of the Gaussian mixture model parameters is obtained based on the aggregated data. This means that the Gaussian mixture model outputs the probability that the framed signal belongs to an active frame or a resting frame. The result corresponding to the highest probability is selected as the judgment result, thus dividing the framed signal into two categories: active frame signals and resting frame signals. The active frame signal is the signal of the active segment during character writing.

[0059] Feature extraction: Continuous activity frames are stitched together to form the electromyographic (EMG) activity segment signal generated when a complete gesture is executed. A fixed length of the signal is extracted to obtain the preprocessed surface EMG signal. The time spectrum of all channels is extracted as signal features and input into the EMG model. In this embodiment, a 1-second EMG signal activity segment is extracted, and the time spectrum of each channel is extracted as a feature and input into the EMG model for text conversion.

[0060] Step S2: Input the preprocessed surface electromyography (EMG) signal into the pre-trained EMG model to obtain the character prediction probability corresponding to the surface EMG signal; the EMG model is trained based on a pre-built EMG text conversion dataset;

[0061] Specifically, the electromyography model includes a SepConv2D layer, a pooling layer, a Dense layer, and an output layer;

[0062] The SepConv2D layer is used to extract features from the input electromyography (EMG) signal data through convolution operations; the pooling layer is used to downsample and reduce the dimensionality of the output of the SepConv2D layer to obtain the dimensionality-reduced EMG signal with the original EMG signal features; the Dense layer is used to extract the correlation between the dimensionality-reduced EMG signal features through nonlinear transformation and map them to the output space; the output layer uses the softmax activation function to output the character prediction probability corresponding to each surface EMG signal.

[0063] Specifically, EMG-to-Text Recognition (ETR) converts electromyographic signals into text sequences. Due to its unique nature, ETR's smallest recognition unit differs from the phonemes, syllables, words, or characters of Automatic Speech Recognition (ASR), or the words or sentences of Facial Speech EMG-to-Text Recognition. ETR's smallest recognition unit is a discrete, single-character gesture, i.e., lowercase English characters and punctuation marks. These characters can be combined to form any word, phrase, or sentence.

[0064] The electromyography (EM) signal to text conversion method in this embodiment is implemented through a pre-trained Electromyography (EM) model and a language model (LM). The EM model is trained using a pre-constructed EM-to-text conversion dataset and is used to extract features and classify the EM signals corresponding to each gesture to obtain the predicted probability of each character corresponding to each gesture. The language model is used to perform character search based on the character prediction probability of each gesture output by the EM model to obtain the converted text.

[0065] The electromyography (EM) model and language model in this section are used during validation and deployment. EM relies on a data-augmented EMG database and requires relatively complete character-written EMG data, including handwritten characters, words, and sentences. The time-spectrum of discrete character EMG sequences is extracted as features, and a convolutional neural network (CNN) is used as the EM in this embodiment. LM relies on a text library to describe the connection frequencies between characters. It is constructed using a large amount of text and a string search model is used as the LM in this embodiment. In practical applications, features of each character's EMG sequence are extracted, and the EMG signals are decoded and searched using EM and LM, finally transforming them into grammatically correct text sequences with semantic information.

[0066] Specifically, the training of the electromyography model includes:

[0067] Electromyography (EMG) signals of multiple users were collected from multiple character writing gestures using an EMG signal detection sensor. The character writing gesture EMG signals include the corresponding EMG signals obtained by controlling wrist movements to perform writing actions of multiple English letters and punctuation marks.

[0068] The collected electromyographic signals were amplified to construct an electromyographic text conversion dataset;

[0069] The electromyography (EMG) text conversion dataset is input into a deep separable convolutional network model, and the EMG model is obtained by iteratively updating the loss function.

[0070] Specifically, for the character writing gesture acquisition task, it includes at least 26 lowercase English letters and at least two punctuation marks (comma and period), for a total of at least 28 gestures. The writing of the 26 English letters and two punctuation marks is achieved by controlling the movement trajectory of the wrist to achieve a shape similar to the character. In this embodiment, the writing of the comma is changed to a greater than sign ">", and the writing of the period is changed to a tilde "~" to enhance the distinguishability of the electromyographic signals corresponding to each gesture.

[0071] The character writing gesture collection task is divided into no fewer than 9 sessions, with each session including all character writing gestures, and one independent session per day. At least 4 sessions are used as the training set, at least 3 sessions as the validation set, and at least 2 sessions as the test set. In this embodiment, the training set collection task involves collecting dynamic electromyographic signals of 26 lowercase English letters, commas, and periods. Each session is divided into 28 trials, executing 28 gestures, each gesture for 3 seconds followed by a 3-second rest, for a total of 10 trials. Specifically, in this embodiment, at least one session of the validation set collects 52 commonly used words, along with 4 sets each of comma, period, and activation gestures, for a total of 55 trials; at least one session collects 30 commonly used phrases, along with 4 sets each of comma, period, and activation gestures, for a total of 33 trials; at least one session collects 15 sentences with punctuation marks, along with 4 sets of activation gestures, for a total of 16 trials. The test set uses text-based content collected in each session, sourced from social media, news websites, and technology books. Within each trail, each character gesture has a 3-second execution time followed by a 3-second rest period.

[0072] The collected data is constructed into a primary electromyography text conversion dataset using the above method, which includes the aforementioned electromyography signals of character writing gestures and their corresponding character labels.

[0073] Furthermore, deep learning-based electromyography (EMG) models typically require a large amount of data for training, while the data acquisition capabilities for EMG-to-text conversion tasks are very limited. Firstly, this task involves a large number of gesture categories. Moreover, character writing gestures are dynamic, requiring recognition of the complete EMG signal from the written character, which significantly limits the amount of available data. Therefore, data augmentation methods are needed to increase the amount of EMG data for character writing gestures to improve model performance. This embodiment addresses the problem of insufficient EMG data for EMG-to-text conversion, leading to unsatisfactory training results, by employing various data augmentation methods to increase the amount of training data and achieve more accurate EMG-to-text conversion results.

[0074] Specifically, data augmentation methods include:

[0075] Waveform augmentation: When performing dynamic gestures, the speed and force of each execution are different. Therefore, multiple forces can be simulated by scaling the amplitude of the signal, and different execution speeds can be simulated by scaling the time scale of the original signal. Multiplying the sampled value of the original signal by a fixed ratio achieves amplitude variation, and interpolating and downsampling the original signal achieves speed variation, resulting in augmented samples.

[0076] Data augmentation is achieved by perturbing the detection results: After obtaining the complete electromyographic signal activity segment using the Active Segment Detection (EAD) method, multiple different activity segments can be obtained by moving the detection window left and right. Each activity segment has a different start and end point, but all correspond to a complete gesture. Furthermore, changing the EAD parameters affects the detection results. For example, increasing the activity segment judgment threshold reduces the activity segment length, while decreasing the threshold increases it. Therefore, data augmentation can be performed by adjusting the EAD parameters.

[0077] Data augmentation by modifying the spectrogram: The method of data augmentation by modifying the spectrogram through time warping, frequency masking and temporal masking has been shown to greatly improve the performance of automatic speech recognition networks; this method can also be used for data augmentation of electromyographic signals.

[0078] Extensive experiments have demonstrated that data augmentation methods that perturb the detection results are more effective. In this embodiment, by moving the active segment window left and right, the results of the electromyographic active segment detection are expanded to 12 times, thus increasing the size of the entire dataset to 12 times its original size.

[0079] Specifically, this embodiment selects a depthwise separable convolutional layer (SepConv2D) as the core network of the ETR EMG model. The input to the model is the time spectrum of a 1-second EMG signal. A SepConv2D layer is added to the model, with 64 filters, a filter space dimension of 3*3, and 'reLU' as the activation function. The output of the SepConv2D layer is batch normalized, followed by 3*3 pooling. A flattening operation is then performed, followed by a dropout operation at a ratio of 0.5. Subsequently, two Dense layers are added to the model, with 80 and 10 network nodes respectively, both using the 'reLU' activation function. The output of each Dense layer is batch normalized, followed by a dropout operation at a ratio of 0.5. Finally, an output layer with 29 nodes and the 'softmax' activation function is added to the model, outputting the predicted probability of each character gesture. During the model compilation process, the classification cross-entropy was selected as the loss function, and the rmsprop optimizer was selected as the backpropagation algorithm. The convergence point electromyography model was obtained through iterative updates.

[0080] Step S3: Based on the character prediction probability corresponding to each gesture output by the electromyography model, the language model is used to search for words and obtain the converted text; the language model is constructed based on the string search model.

[0081] Preferably, this embodiment proposes a string search model to construct the language model in ETR, which is used to perform character search based on the character prediction probability corresponding to each gesture output by the electromyography model, and obtain the converted text, with the aim of correcting the output results of the electromyography model.

[0082] Specifically, after obtaining the predicted probability of each character corresponding to the electromyographic signal of each character writing gesture through the electromyographic model, the first m characters with the highest predicted probability corresponding to each gesture are saved.

[0083] If no electromyographic signal is detected within a preset time, it is considered a space, and all characters between the spaces are taken as a prediction combination; in this embodiment, the preset time is set to 3 seconds.

[0084] Based on the order of predicted probabilities from largest to smallest, the first m predicted characters corresponding to the first gesture in the space-space string are taken as the root node. Based on the order of each gesture and the predicted probabilities of the first m predicted characters corresponding to each gesture, the predicted characters corresponding to each gesture are combined and used as child nodes. A search tree is built in sequence, and each branch of the search tree is a group of prediction combinations. The prediction combinations are matched in the corpus.

[0085] If a predicted combination exists in the corpus, it is identified as the final word recognition result, and the search tree construction is terminated; otherwise, the search tree continues to be built and the search continues until all possible combinations have been searched; if none of the obtained combinations appear in the corpus, the maximum predicted probability of each character is taken as the final prediction result of the string.

[0086] As a specific implementation, the EAD (Electromyography) is first used to detect the resting period duration. If no electromyography signal is detected for more than 3 seconds, it is considered a space between words, and all characters before and after the space are taken as a predicted combination. Specifically, in this embodiment, if the length of the predicted combination is greater than a preset length l, only the first l characters of the predicted combination are extracted as a word; in this embodiment, l is set to 6, that is, if the length of the predicted combination is greater than 6, only the first 6 characters of the word are extracted as a word. Otherwise, all characters are retained.

[0087] Save the first 5 EM prediction results for each character corresponding to a dynamic electromyographic gesture.

[0088] An ordered string search tree is constructed. For each prediction result of the first character in the string, a search tree is constructed sequentially according to the order of prediction probability from largest to smallest, and the order in which the characters corresponding to each electromyographic gesture appear. A total of five search trees can be generated. One branch of each search tree is a set of predicted word combinations used for matching in a corpus. In this embodiment, the COCA corpus is used for word matching. In practical applications, different corpora can be used depending on the application domain to achieve more accurate search results.

[0089] Each branch of the string search tree is matched against words in the corpus in descending order of predicted probability. If a predicted combination exists in the corpus, it is determined as the final recognition result of the string, and the construction of the string search tree is terminated. Otherwise, the search tree continues to be built and searched until all possible combinations have been searched. If none of these combinations appear in the corpus, the maximum predicted probability of each character is taken as the final prediction result of the string. In this embodiment, words longer than 1 in the corpus are also truncated to match the output of the string search tree. If the string matches successfully and the word lengths are equal, the original word with the truncated word is taken as the recognition result.

[0090] As a specific embodiment, the surface electromyography (EMG) signals of the present invention can be applied to an intelligent prosthetic system. The intelligent prosthetic system may include an EMG signal detection sensor, an EMG signal recognition and control module, a motor drive device, and a bionic hand device, which are connected in sequence.

[0091] The electromyography (EMG) signal detection sensor includes electrode pads for attaching to the surface of the residual limb of a person wearing a smart prosthesis, for detecting EMG signals and sending them to the EMG signal recognition and control module;

[0092] The electromyography (EMG) signal recognition and control module is used to recognize and process the received EMG signals, and to perform pattern conversion based on the EMG signal recognition results, thereby realizing identity verification, daily movement control, and conversion of EMG signals to text.

[0093] The motor drive device is used to control the bionic hand device to perform actions corresponding to electromyographic signals based on the recognition results of the electromyographic signal recognition control module, thereby realizing daily movement control.

[0094] Specifically, the intelligent prosthetic system in this embodiment uses surface electromyography (EMG) signals from the residual limb as the signal source for recognizing movement intentions, assisting amputees in maximizing their ability to grasp objects and input handwriting. In addition to the aforementioned structure, the intelligent prosthetic system also includes a cavity for connecting the patient's residual limb and the bionic hand device. The shape of the bionic hand device in the system is fixed, while the EMG acquisition and recognition module and the cavity can be adaptively designed according to different types of amputees. During EMG signal acquisition, the electrode pads of the EMG signal detection sensor are attached to the surface of the residual limb, ensuring that the absolute position of the electrode pads remains as constant as possible during each acquisition.

[0095] Figure 2 This diagram illustrates the mode switching process of the intelligent prosthetic system in this embodiment. When the system is in sleep mode, it operates at low power. The system wakes up by detecting the electromyographic (EMG) signal corresponding to the activated gesture through the mode switching unit, switching the intelligent prosthetic system to the daily control mode. In daily control mode, this embodiment designs four commonly used gestures to perform most hand tasks in daily life. Activated gestures are also recognized in daily control mode. If an activated gesture is detected in daily control mode, the system switches to EMG-to-text conversion mode. In EMG-to-text conversion mode, the intelligent prosthetic system of this embodiment can recognize the EMG signals corresponding to 26 lowercase English letters and 2 punctuation marks, convert them into text information, and transmit them wirelessly or via wired means to display devices such as mobile phones and computers for display, thus realizing the conversion of EMG signals into text sequences. Furthermore, the system can be switched back to sleep mode by activating a gesture in EMG-to-text conversion mode.

[0096] Specifically, the electromyography signal recognition and control module includes a mode switching unit, an authentication unit, a daily control unit, and an electromyography text conversion unit;

[0097] The mode switching unit is used to switch the working modes of the intelligent prosthetic system. The working modes include authentication mode, daily control mode and electromyography-text conversion mode.

[0098] The authentication unit is used to identify whether the user to be authenticated is an authorized user in authentication mode;

[0099] The daily control unit is used to perform actions corresponding to electromyographic signals in daily control mode;

[0100] The electromyography-text conversion unit is used to convert electromyography signals into text information in electromyography-text conversion mode.

[0101] In this embodiment, the mode switching unit, daily control unit, and electromyography text conversion unit of the intelligent prosthetic system all require data collection and construction of training datasets, which are then used for training.

[0102] Specifically, this embodiment collects data and constructs a database from no fewer than five amputees (with or without wrist or forearm amputation). Each patient's dataset is collected in two parts: static gestures and dynamic gestures. Static gestures include electromyographic activation gestures used by the mode switching unit and daily control gestures used in the daily controllable mode; dynamic gestures are those used by the electromyographic text conversion unit. During electromyographic signal acquisition, electrodes are attached to the surface of the patient's residual limb, ensuring that the absolute position of the electrodes remains as constant as possible during each acquisition.

[0103] For static gesture acquisition tasks, the tasks include the electromyographic activation gesture "OK" (gesture 1) and daily control gestures. Daily control gestures include at least four gestures, which can be a five-finger grip (gesture 2, used for grasping tasks), a thumb and index finger pinch (gesture 3, used for pinching tasks), a five-finger spread (gesture 4, used for greeting), and an index finger point (finger 5, used for indicating direction).

[0104] For dynamic gesture acquisition tasks, the aforementioned method for acquiring electromyographic signals from character writing gestures is used.

[0105] More specifically, the static gesture collection task is divided into no fewer than 9 sessions (each session includes all static gestures), with one independent session per day, and the collected content is completely identical. At least 4 sessions are used as the training set, at least 2 sessions as the validation set, and at least 3 sessions as the test set. Each session is divided into 5 trials, executing gestures 1 through 5 respectively. In each trial, each gesture is executed for 3 seconds, followed by a 3-second rest, for a total of 20 executions. Therefore, each session generates 100 (5*20) sets of data.

[0106] The collected data is divided into three datasets using the above method: activated gesture EMG signal dataset, daily control EMG dataset, and EMG text conversion dataset. The activated gesture EMG signal dataset includes positive and negative pairs and corresponding labels. Positive pairs consist of two activated gesture EMG signals from the same user collected in different sessions, or any two activated gesture EMG signals from different users; in this embodiment, the label for positive pairs is set to 1. Negative pairs are sample pairs consisting of activated and non-activated gesture EMG signals, labeled with 0. The daily control EMG dataset includes four daily control gesture EMG signals and corresponding gesture classification labels. The EMG text conversion dataset includes the aforementioned dynamic gesture EMG signals and their corresponding text labels.

[0107] After the training samples are collected, the electromyographic signal data in the training samples can be preprocessed using the aforementioned electromyographic signal preprocessing method for use in the calculation of each unit.

[0108] After processing the data and extracting features from each dataset, it can be used to train each model.

[0109] First, the mode switching unit is trained on the conjoined network based on the activated gesture EMG signal dataset. It is used to activate the smart prosthesis in low-power sleep mode or switch the working mode of the smart prosthesis system after detecting the activated gesture EMG signal. The working modes include daily control mode and EMG-text conversion mode. EMG activation technology, also known as key gesture detection technology, requires detecting the EMG signal corresponding to a predefined activated gesture or key gesture from a series of EMG signals, without needing to recognize all EMG signals. The EMG activation technology in this embodiment is similar to voice wake-up technology. Voice wake-up technology is the foundation for enabling voice interaction capabilities of embedded devices and can be applied to various devices, such as mobile phones, smart speakers, robots, smart homes, in-vehicle devices, and wearable devices. Typically, the device wake-up word is default or pre-set, usually consisting of 3 or 4 syllables, such as Apple's "Hi Siri," Google Assistant's "Hey Goggle," Alibaba's "Tmall Genie," and Xiaomi's "Xiao Ai." When the device is in sleep mode, it continues to pick up and detect wake words. Once a wake word is detected, the device switches from sleep mode to working mode, awaiting subsequent interaction. Voice wake-up technology requires detecting the complete wake word, while electromyography (EMG) activation technology only needs to detect the signal generated by the activation gesture. When the activation gesture is executed and held, the generated EMG signal is continuous and consistent, thus the activation gesture can be recognized during its execution. This invention uses the "OK" gesture as the activation gesture. Figure 3 This is a flowchart of the electromyography activation technology of the present invention.

[0110] Specifically, a Siamese Network is a symmetric network where two symmetrical parts share the same weights and have identical structures. It can employ Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs). The goal is to learn whether the inputs to the two symmetrical parts are identical. The output layer of the Siamese Network uses a loss function to fuse the two symmetrical parts. This loss function represents the similarity or distance between the EMG signals of the test samples input to the two input terminals of the Siamese Network and the EMG signals of the activated gestures. Binary crossentropy is typically used, and its expression is:

[0111]

[0112] Among them, y i The predicted distribution of the test samples, To determine the true distribution of the activation gesture template samples.

[0113] The frame signals of activated gestures are designated as positive samples, while those of other gestures are designated as negative samples. During training, the same number of positive pairs (positive sample - positive sample, labeled 1) and negative pairs (negative sample - positive sample, labeled 0) are used to ensure sample balance. During testing, the electromyographic (EMG) signal features of the test samples and the pre-stored EMG signal feature templates of activated gestures are input into two symmetrical structures of the conjoint network, respectively. The posterior probability of the EMG signal features is obtained, representing the similarity between the EMG signal feature and the EMG feature of the activated gesture, ranging from 0 to 1. A suitable threshold is set for binarization, i.e., determining whether the frame is an activated or inactive frame.

[0114] In practical applications, the user's electromyography (EMG) signals are collected in sleep mode, and the EMG signals are subjected to sliding window framing, preprocessing, active segment detection, and feature extraction.

[0115] The features of each frame of electromyography (EMG) signal extracted are input into the connected network of the mode switching unit to calculate the similarity with the pre-stored EMG features of the activated gesture; based on a preset threshold, it is determined whether the acquired EMG signal is an activated frame.

[0116] Specifically, in this embodiment, five frames are accumulated for a mode decision. If more than three of the five frames are identified as active frames, the gesture is determined to be an active gesture; otherwise, it is an inactive gesture.

[0117] Furthermore, there are five categories of gestures in the daily control mode: activation gestures (gesture 1) and four daily control gestures (gestures 2-5). Gesture recognition algorithms can be modeled using existing classification algorithms. The signal preprocessing, sliding window framing, preprocessing, active segment detection, and feature extraction processes are the same as described above.

[0118] Specifically, this embodiment uses a variant of the recurrent neural network structure as the basic structure of the deep learning model for recognizing everyday control gestures, constructing a stacked LSTM model consisting of two LSTM layers and one bidirectional LSTM layer. The model's input is all features of a single frame of electromyography (EMG) signal. The first LSTM layer is set to 64 nodes, with dropout, recurrent dropout, and return sequences parameters of 0.1, 0.5, and True, respectively. Batch normalization is performed on the output of this layer. Then, the second LSTM layer is set to 64 nodes, with the activation function 'ReLU', a dropout rate of 0.1, a recurrent dropout rate of 0.5, and return sequences set to True. Batch normalization is performed again on the output of this layer. The third layer is a bidirectional LSTM layer with 64 nodes, the activation function 'ReLU', a dropout rate of 0.1, and a recurrent dropout rate of 0.5. The output of this layer after batch normalization is sent to the output layer, which uses the sigmoid function as its activation function and outputs the predicted probabilities of five gestures (gesture 1 to gesture 5). The model's loss function is defined as the classification cross-entropy function, and the error is backpropagated using the rmsprop optimizer.

[0119] When building a deep learning model for recognizing everyday control gestures, we also optimize the model by using residual connections, regularization, Dropout, batch normalization, activation functions, etc., to improve model performance.

[0120] The data in the daily control electromyography dataset were divided into training, testing, and validation sets. After training the model using the data in the training set, the model parameters were iteratively adjusted using the validation set. First, the model was allowed to overfit, then regularization and other optimization methods were added. This iterative optimization process continued until the set of hyperparameters that performed best on the validation set was selected as the final test model.

[0121] Furthermore, the intelligent prosthetic limb system of this embodiment also includes an authentication function, implemented through an authentication unit. The authentication unit includes a pre-trained authentication model used to authenticate the user wearing the intelligent prosthetic limb before use; the training of the authentication model includes:

[0122] Obtain the electromyographic signals of authorized users’ password gestures and construct a training dataset of password gesture electromyographic signals. The training dataset includes N sets of password gesture electromyographic signals of authorized users and corresponding user labels.

[0123] The single-classification model is trained using the training data of each authorized user in the training dataset, and a threshold is set to obtain the identity model of each user.

[0124] The process of verifying the identity of users wearing smart prosthetics includes:

[0125] The electromyography (EMG) signals of a user wearing a smart prosthesis are acquired, preprocessed, and then the preprocessed EMG signals are subjected to active segment detection and feature extraction. The extracted feature data is then input into an identity verification model.

[0126] Retrieve the pre-established identity model of the authorized user to be matched, and determine whether the user to be verified is an authorized user or an unauthorized user based on a pre-set threshold.

[0127] Specifically, the first step is to collect electromyographic signals of the password gestures and then construct a training dataset.

[0128] Electrode pads are attached to the authorized user's arm to collect electromyographic (EMG) signals from the user's arm surface. This embodiment of the intelligent prosthetic system can be used not only for user authentication but also in conventional authentication scenarios, replacing fingerprint and facial recognition methods. This embodiment involves at least 20 users and sets at least 3 password gestures. Gestures can be open palms, complex static gestures, or dynamic gestures such as wrist rotation. Each signal acquisition requires collecting all password gestures for each user, with each gesture executed at least 10 times, collecting 10 sets of EMG signals. Each password gesture lasts approximately 3 seconds. All signals are used for subsequent identity modeling.

[0129] After acquiring the electromyographic signals, signal preprocessing is required. Specifically, the raw electromyographic signals can be filtered out by a Butterworth 4th order 30-500Hz bandpass filter to remove baseline drift and motion artifacts, and then filtered out by a 50Hz notch filter to remove power frequency interference.

[0130] Then, active segment detection is performed. Active segment detection is performed on each channel of the multi-channel surface electromyography signal. The signal is divided into frames through a 128ms window with a movement step of 64ms. The root mean square (RMS) of the signal in each window is calculated.

[0131] To further determine the threshold, statistical methods were used to sort the RMS values ​​of all frame signals to find a suitable RMS threshold. Signal segments below the threshold represent resting electromyographic signals. Then, a two-round segmentation was performed. In the first round, resting frames and active frames were labeled, with 0 representing resting frames and 1 representing active frames. In the second round, the length of consecutive active frames was determined. If the duration did not exceed 1 second, it was considered noise and classified as a resting frame; if the duration exceeded 1 second, it was considered an active segment and its active frame label was retained. This dual detection method ensures the accuracy of active segment detection.

[0132] Further feature extraction is performed by splicing consecutive activity frames into electromyographic activity segments corresponding to the complete hand gesture execution through activity segment detection. Each activity segment signal serves as the unit for feature extraction. This embodiment employs two types of feature extraction techniques: intra-channel features and inter-channel features. Intra-channel features calculate the characteristics of a single-channel signal, including three main categories: time-domain features, frequency-domain features, and autoregressive coefficients. Time-domain features typically include root mean square (RMS), waveform length, short-time zero-crossing rate, mean absolute value, and rate of change of slope sign. Frequency-domain features include mean amplitude spectrum, median frequency, and mean power frequency. Autoregressive coefficients are generally selected from 1st to 4th order autoregressive coefficients. Inter-channel features calculate the characteristics between pairs of channels. Inter-channel features include mutual information and Pearson correlation coefficients.

[0133] After extracting the electromyographic signal features, they can be used for model training. This embodiment uses the training data of each authorized user in the training dataset to train a single-classification model, obtaining an identity model for each user, including:

[0134] N sets of training data from authorized users are used as positive class training samples, and the positive class training sample data are mapped from the original space to the feature space through a nonlinear transformation function.

[0135] Using the support vector description algorithm, a minimum hypersphere is trained based on the training data of each authorized user mapped to the feature space. The data of all users are input into the identity verification model to obtain the distance of each user's data to the surface of the minimum hypersphere. A threshold is set based on the distance to obtain the corresponding user identity model.

[0136] Specifically, for identity verification, all users other than authorized users are considered imposters. Since it's impossible to collect data from all imposters when building the model, this embodiment uses a one-class classifier for modeling, as it only uses data from authorized users as positive class samples for training. Specifically, Support Vector Data Description (SVDD) can be used as the one-class classifier, also known as a One-class Support Vector Machine (OCSVM), which can distinguish between target and non-target samples. For a set of positive class training data (i.e., electromyography data from authorized users) x∈R... n×d Where n is the number of samples and d is the feature dimension. First, the sample data is mapped from the original space to the feature space using a nonlinear transformation function Φ:x→F. Then, a hypersphere with the smallest possible volume is found in the feature space to contain as much feature data as possible within the smallest possible hypersphere. This means that the radius of the hypersphere needs to be constrained during its construction; furthermore, this hypersphere needs to have a certain degree of fault tolerance, allowing some outlier data to fall outside the hypersphere. To construct such a minimum hypersphere, the following constrained optimization problem is modeled using SVDD:

[0137]

[0138]

[0139] Where R is the radius of the hypersphere, a is the center of the sphere, ξ is the relaxation factor, C is the penalty parameter balancing the hypersphere and the error rate, and x i This represents the i-th feature data of the electromyography signal mapped to the feature space for a specific authorized user. In this embodiment, the Radial Basis Function (RBF) is chosen as the nonlinear transformation function Φ:x→F for SVDD. SVDD trains each user's individual data into an identity model belonging to that user, used for identity verification during model deployment.

[0140] In the application of SVDD for authentication, through nonlinear mapping, if the image of a user's electromyography (EMG) signal sample point in the feature space falls within the optimal hypersphere, the sample is considered a normal point, i.e., an authorized user's sample; if it falls outside the optimal hypersphere, the sample is considered an outlier. The output of SVDD is the distance between the sample and the surface of the optimal hypersphere. Therefore, it is necessary to determine a decision threshold for the identity model for each user to determine whether the accessing user is an authorized or unauthorized user.

[0141] For each authorized user's SVDD identity model, the collected electromyography (EMG) data of other users is input, resulting in a set of distance-based outputs. The more similar the input EMG data is to the data used in modeling the authorized user's identity model, the smaller the absolute value of the output; conversely, the less similar the input EMG data is to the data used in modeling the authorized user's identity model, the larger the absolute value of the output. Based on multiple distance-based outputs, a threshold is set for the authorized user's identity model; when the absolute value of the output is less than this threshold, the user is determined to be an authorized user; otherwise, the user is considered an unauthorized user.

[0142] In practical applications, when any user accesses the system, they first select a pre-stored authorized user identity model corresponding to their access. Then, they wear electrode pads and perform a password gesture, each gesture lasting approximately 3 seconds. During this time, the electromyography (EMG) signal detection sensor continuously collects EMG signals and transmits them to the EMG signal recognition control module. This module first performs real-time filtering and framing of the signals, using the RMS threshold determined during identity modeling to detect active segments. Once a complete active segment is detected, its features are calculated, forming a feature vector, which is then fed into the SVDD identity model. The SVDD identity model outputs the distance between the feature vector of the current EMG sample and the selected pre-stored authorized user identity model. Based on the pre-determined identity model threshold, it determines the current user's identity type. If the distance between the current sample and the pre-stored authorized user identity model is less than the threshold, verification is successful, and the device is unlocked; otherwise, verification fails, and the user is asked to retry. If three consecutive verification failures are detected, the system automatically locks the device for a certain period, during which no user access is permitted.

[0143] The hardware architecture of the intelligent prosthetic system in this embodiment includes two subsystems, such as... Figure 4 The diagram shows the signal acquisition subsystem and the signal processing subsystem. The signal acquisition subsystem mainly consists of three modules: a signal acquisition module, a power management module, and a Bluetooth master control module. The power module is responsible for powering the entire system, powered by a 3.7V lithium battery, and then the voltage is converted to provide the required voltage to other modules of the system through a buck-boost circuit and a voltage regulator circuit. This invention uses the nRF52832 as the Bluetooth master control chip. The low-power, 24-bit analog front-end chip ADS1298 for physiological signal measurement is controlled via the SPI communication protocol to acquire 8-channel surface electromyography signals at the corresponding location. The signals are then preprocessed and packaged by the corresponding analog front-end, and then transmitted to the host computer via serial port (wired transmission) or Bluetooth (wireless transmission) through the nRF52832 processor. The host computer used in this invention is a Cortex-A72 (ARM v8) 64-bit SoC@1.5GHz Raspberry Pi 4B / 8GB platform. An authentication method is embedded in this platform to collect the signals of visiting users in real time and output the confirmation result, displaying whether access is allowed or denied.

[0144] In another embodiment of the present invention, a storage medium is provided that stores computer instructions for execution by a computer to implement the aforementioned text conversion method for surface electromyography signals.

[0145] In summary, the embodiments of the present invention provide a text conversion method for surface electromyography (EMG) signals. By collecting and amplifying EMG signals of text characters, an EMG text conversion model combining an EMG model and a language model is constructed, realizing EMG-to-text sequence recognition. The present invention can restore the ability of amputees to input characters and, combined with gesture recognition, enable them to grasp everyday objects, thereby maximizing the restoration of amputees' self-care abilities.

[0146] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0147] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for text conversion of surface electromyography signals, characterized in that, Includes the following steps: Acquire the surface electromyography (EMG) signals corresponding to character writing gestures, and preprocess the EMG signals. The preprocessed surface electromyography (EMG) signal is input into a pre-trained EMG model to obtain the character prediction probability corresponding to the surface EMG signal. The electromyography model was trained based on a pre-built electromyography text conversion dataset; Based on the character prediction probabilities corresponding to each gesture output by the electromyography (EMG) model, a word search is performed using a language model to obtain the converted text, including: saving the top m characters with the highest prediction probabilities for each gesture; if no EMG signal is detected within a preset time, it is considered a space, and all characters between the spaces are taken as a prediction combination; in descending order of prediction probabilities, with the top m predicted characters corresponding to the first gesture in the string between spaces as the root node, based on the order of each gesture and the prediction probabilities of the top m predicted characters corresponding to each gesture, the predicted characters corresponding to each gesture are combined and used as child nodes, and a search tree is built sequentially. Each branch of the search tree is a group of prediction combinations, and the prediction combinations are matched in the corpus; if a prediction combination exists in the corpus, it is identified as the final recognition result of the word, and the building of the search tree is terminated; otherwise, the search tree is built and the search is performed until all possible combinations are searched; if none of the obtained combinations appear in the corpus, the character combination with the highest prediction probability for each character is taken as the final prediction result of the word; If the length of the predicted combination is greater than the preset length Then only the first part of the predicted combination is extracted. Each character is considered as a word; the corpus contains words longer than a certain length. The word is truncated to match the output of the string search tree; if the string matches successfully, the original word with the truncated word is taken as the recognition result; the language model is constructed based on the string search model.

2. The text conversion method for surface electromyography signals according to claim 1, characterized in that, The electromyography model includes a SepConv2D layer, a pooling layer, a Dense layer, and an output layer; The SepConv2D layer is used to extract features from the input electromyography signal data through convolution operations. The pooling layer is used to downsample and reduce the dimension of the output of the SepConv2D layer to obtain a dimension-reduced electromyographic signal with the original electromyographic signal characteristics; The Dense layer is used to extract the correlation between the features of the dimensionality-reduced electromyographic signals through nonlinear transformation and map them to the output space; The output layer uses the softmax activation function to output the character prediction probability corresponding to each surface electromyography signal.

3. The text conversion method for surface electromyography signals according to claim 2, characterized in that, The training of the electromyographic model includes: Multiple electromyographic (EMG) signals of character writing gestures were collected from multiple users to obtain an initial EMG-text conversion dataset. The character writing gesture EMG signals include the corresponding EMG signals obtained by controlling wrist movements to perform writing actions of multiple characters. Data augmentation was performed on the electromyographic signals in the initial text conversion dataset to construct an electromyographic text conversion dataset. The electromyography (EMG) text conversion dataset is input into a depthwise separable convolutional network model, and the EMG model is obtained by iteratively updating the model through a loss function.

4. The text conversion method for surface electromyography signals according to claim 3, characterized in that, The characters include English letters and punctuation marks; the initial electromyography text conversion dataset includes a training set, a validation set, and a test set; The data for each session in the training set includes electromyographic signals corresponding to 26 lowercase English letters, commas, and periods; The verification set includes at least one session containing multiple common words, as well as comma and period gestures; at least one session contains multiple common phrases, as well as electromyographic signals corresponding to commas and periods. Each session in the test set includes multiple text-related electromyographic signals.

5. The text conversion method for surface electromyography signals according to claim 3, characterized in that, Data augmentation can be performed using any of the following methods: The waveform is altered to amplify the data, including: multiplying the sampled value of the acquired electromyographic signal by a preset ratio to change the amplitude; interpolating and downsampling the acquired electromyographic signal to change the velocity, thus obtaining the amplified electromyographic signal; Data augmentation is performed by perturbing the detection results, including: after obtaining the complete electromyographic activity segment of the electromyographic signal through the electromyographic activity segment detection method, obtaining different activity segments of the electromyographic signal by moving the detection window left and right to obtain the amplified electromyographic signal; or by increasing the judgment threshold of the activity segment to reduce the length of the activity segment; or by decreasing the judgment threshold of the activity segment to increase the length of the activity segment to obtain the amplified electromyographic signal. Data augmentation is achieved by modifying the spectrum, including altering the spectrum of the electromyographic signal through time warping, frequency masking, and temporal masking to obtain an amplified electromyographic signal.

6. The text conversion method for surface electromyography signals according to claim 1, characterized in that, The preprocessing of the surface electromyography signal includes: The electromyographic signal is divided into frames by a preset sliding window and step size, and the framed electromyographic signal is then filtered. Active segment detection is performed on the filtered frame signal to divide the frame signal into active frame signal and rest frame signal; The continuous activity frame signals are spliced ​​together to form the electromyographic activity segment signals generated when the corresponding complete gesture is executed. A fixed length of signal is extracted, and the time spectrum of all channels is extracted to obtain the preprocessed surface electromyographic signal.

7. The text conversion method for surface electromyography signals according to claim 6, characterized in that, The process of detecting active segments in the filtered framed signal includes: The feature matrix is ​​obtained by calculating the root mean square of the subbands of the framed signal using the three-frequency subband filtering method. The data in the feature matrix are iterated using the expectation-maximization clustering algorithm. Similar signals in the feature matrix are aggregated. The maximum a posteriori probability estimate of the Gaussian mixture model parameters is obtained based on the aggregated data. The framed signal is divided into active frame signal and resting frame signal based on the a posteriori probability.

8. A storage medium, characterized in that, The storage medium stores computer instructions that are executed by a computer to implement the text conversion method for surface electromyography signals according to any one of claims 1-7.

Images