A silent speech recognition method based on fusion of motor unit activity and surface electromyography signals

Abstract

The application discloses a silent speech recognition method based on motion unit activity and surface electromyogram signal fusion. First, a plurality of channel surface electrode arrays are arranged on the relevant vocal muscle groups of the face and neck of a subject through an electromyogram signal acquisition device to collect surface electromyogram signals generated by the subject during the execution of a silent speech task. Then, the collected original multi-channel electromyogram signals are decomposed and processed to extract discharge activity information of motion units. In combination with the original multi-channel electromyogram signals and the discharge information of the motion units, peak-to-peak value features related to the motion units in the time-frequency domain are extracted, multi-modal feature inputs are constructed, and a multi-modal fusion neural network is constructed to realize the classification and recognition of silent speech instructions.

Description

Technical Field

[0001] This invention relates to the field of bioelectrical signal processing and human-computer voice interaction technology, specifically to a silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals. Background Technology

[0002] Against the backdrop of rapid development in human-computer interaction technology, speech recognition, as a natural and efficient interaction method, has been widely applied in various fields such as smart homes, medical assistance, and vehicle control. However, traditional speech recognition systems heavily rely on audible sound input, are susceptible to environmental noise interference, and cannot function properly in special scenarios such as quiet environments, users with speech impairments, or people who are deaf. Therefore, silent speech recognition (SSR) technology has become a research hotspot in recent years.

[0003] Currently, most silent speech recognition methods focus on extracting macroscopic features such as time-domain, frequency-domain, and time-frequency-domain features from surface electromyography (sEMG) signals. While these can reflect the overall activation state of muscles, the fact that sEMG signals are essentially a mixture of the firing activities of multiple motor units means that a single macroscopic feature cannot capture finer-grained neural drive information, thus limiting the performance of recognition models in complex tasks. Motor unit (MU) activity represents the most basic functional unit of muscle. By decomposing surface EMG signals to extract motor unit action potentials, more microscopic muscle recruitment information can be obtained. These microscopic features not only reveal the fine control patterns of the nervous system over muscles but also possess good adaptability to individual differences and strong robustness.

[0004] However, most current SSR systems still rely on single-scale analysis of macroscopic sEMG features, failing to fully integrate microscopic features of motor units with traditional raw electromyographic signal features, and lacking in-depth modeling of neural drive information. This limitation restricts the accuracy of silent speech recognition systems in practical applications. Summary of the Invention

[0005] To address the problem in existing silent speech recognition systems that rely solely on macroscopic surface electromyography (EMG) signals, which fails to accurately capture microscopic motor unit activities and thus results in insufficient recognition accuracy, this invention aims to provide a silent speech recognition method based on the fusion of motor unit activity and surface EMG signals. This invention extracts microscopic motor unit activity features by performing motor unit action potential decomposition on surface EMG signals, while simultaneously acquiring and utilizing macroscopic time-frequency features of the original EMG signals. Specifically, this invention constructs a multi-scale feature fusion model to fuse microscopic motor unit activity information with macroscopic EMG features, achieving multi-level and multi-dimensional representation of silent speech signals. The introduction of motor unit features more realistically reflects the muscle nerve drive mechanism, improving the model's ability to capture details of speech movement; the fusion with surface EMG signals enhances the richness and robustness of the features. Through this optimized design, the accuracy of silent speech recognition is improved.

[0006] The objective of this invention is achieved through the following technical solutions.

[0007] A silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals includes the following steps:

[0008] Step 1: Using an electromyography (EMG) signal acquisition device, a multi-channel surface electrode array is deployed on the relevant vocal muscles of the subject's face and neck to acquire the raw multi-channel surface EMG signals generated by the subject during the performance of a silent speech task; this step includes the following sub-steps:

[0009] Step 1.1: A multi-channel surface electrode array is arranged on the relevant vocal muscles of the subject's face and neck. The electrode array contains m channels, and the specific arrangement area of ​​the electrodes includes the perioral muscles, buccal muscles, mental muscles, and sternocleidomastoid muscles, which are closely related to speech production.

[0010] Step 1.2: The subject performs K speech tasks, each task repeated R times. Electromyography (EMG) waveform data for each channel is synchronously recorded during task execution using an EMG signal acquisition device, obtaining the original multi-channel surface EMG signal matrix x(k):

[0011] x(k) = [x1(k), x2(k), ..., x m (k)] T

[0012] k = 0, ..., D R

[0013] Where x i (k) represents the electromyographic signal of the i-th channel at the k-th discrete time, D RThe duration of the recording; the output of this step is the raw multichannel surface electromyography signal data x, which serves as the basic input for subsequent electromyography decomposition and feature extraction processing.

[0014] Step 2: Decompose and process the raw multichannel surface electromyography signals acquired in Step 1 to extract the firing activity information of the motor units; this step includes the following sub-steps:

[0015] Step 2.1: Treat the original multichannel surface electromyography (EMG) signal as a mixture of signals from multiple unobservable motor units, satisfying the following convolutional mixture model:

[0016]

[0017] Where: x i (k) represents the electromyographic signal acquired by the i-th surface electrode channel at the k-th discrete time, s j (k) represents the discharge activity sequence of the j-th motion unit at time k, reflecting whether it discharges at that time. ij (l) represents the response of channel i to the j-th motion unit at delay l, n i (k) represents the noise of the i-th channel at time k, H(l) is the convolutional mixing matrix, s(kl) represents the column vector composed of the discharge activities of all n motion units at discrete time kl, n(k) is the additive noise term, n is the number of motion units, and m is the number of channels;

[0018] Step 2.2: Perform observational amplification processing on the original multichannel surface electromyography signals:

[0019] To improve the stability and decomposition accuracy of the mixing matrix estimation in blind source separation, the original fully determined or underdetermined system is transformed into an overdetermined system. A delayed embedding strategy is used to add R delayed versions to each channel, resulting in an extended observation matrix.

[0020]

[0021] in This represents the expanded m*(R+1) samples; This represents the convolutional blending matrix after channel expansion. This represents the discharge vector of the moving unit after channel expansion. This represents the additive noise term after channel expansion;

[0022] Step 2.3: Expand the observation matrix Whitening is performed to reduce redundancy between features in the input data;

[0023] In the whitening process, the data must first be centered by normalizing the mean of the input signal X to zero, so as to ensure that the data is zero-mean.

[0024] X centered =XE[X]

[0025] Where X centered It is a centralized extended observation data matrix;

[0026] Next, the centered data is projected onto a new coordinate system so that its covariance matrix becomes the identity matrix, thereby removing the correlation of the data. The formula is as follows:

[0027] X whitened =D -1 / 2 E T X centered

[0028] Among them, X whitened It is the whitened extended observation data matrix, E is the eigenvector of the covariance matrix, and D is the eigenvalue diagonal matrix;

[0029] Step 2.4: After whitening the extended observation matrix in Step 2.3, an improved Fast ICA (Fast-Independent Component Analysis) algorithm is used to extract independent components from the whitened electromyography (EMG) data to estimate the firing sequence of motor cells. In the Fast ICA algorithm, the goal is to learn a separation matrix W such that the whitened observation data, through the whitened extended observation data matrix X... whitened After projection, several statistically independent component signals can be recovered, which is the estimated firing sequence of the moving unit. Specifically, each row in the separation matrix This represents the projection direction of the i-th independent component. During initialization, a uniform random distribution can be used to generate an initial vector with the unit norm.

[0030] Step 2.4.1: Randomly initialize the weight vector w of the separation matrix;

[0031] Step 2.4.2: Iteratively update the weight vector w using a nonlinear function g(x) (such as tanh(x) or x). 3 Perform independent component maximization processing:

[0032]

[0033] Where w (k) Let X represent the weight vector at the k-th iteration, X represent the whitened extended observation data matrix, and g(·) be a nonlinear function used to maximize the non-Gaussianity of independent components. Common forms include g(x) = tanh(x) or g(x) = x 3g'(·) is the derivative of the function g(·). This represents the expectation operation over the sample, i.e., the average value over all samples;

[0034] Step 2.4.3: Perform Gram-Schmidt orthogonal normalization on all updated weight vectors w to ensure orthogonality between multiple independent components;

[0035] Step 2.4.4: Check convergence: If the weight vector changes ||w| between two adjacent iterations + If -w|| is less than the set threshold, it is considered converged and proceeds to the next stage; otherwise, return to step 2.4.2 to continue iterating.

[0036] Step 2.4.5: For the currently estimated source signal s i (k)=w i (n) T x(k) is used for peak detection to obtain the discharge pulse train PT n ={t j}, and calculated its coefficient of variation based on the discharge peak sequence. n ;

[0037]

[0038] Where σ(PT) n ) is the standard deviation of the discharge pulse train, μ(PT) n () is the mean of the discharge pulse train;

[0039] Step 2.4.6: Perform iterative optimization with coefficient of variation constraints: if CoV n-1 >CoV n This indicates improved discharge stability, so the weight vector is updated:

[0040]

[0041] Where w i (n+1) is the updated weight vector of the i-th independent component in the (n+1)-th iteration, t j It is the time point of discharge peak j, where J is the number of peaks, x(t) j ) is at time point t j Above, the signal window extracted from the whitened data X;

[0042] If CoV n ≥CoV n-1 If the source update process is terminated, the current optimal weight vector w is retained. i ;

[0043] Step 2.4.7: Perform an independence quality assessment on the separated source signals, using the Silhouette Coefficient (SIL) to measure it.

[0044]

[0045] Where, d A d represents the average distance (compactness) between points within the source signal. B The average distance (separability) between a point within the source signal and the nearest point of another class is used. Only when SIL > 0.9 is the source signal considered an effective motion unit activity. This step filters out a set of estimated source signals:

[0046] S = {s1(k), s2(k), ..., s} n (k)}

[0047] Each s j (k) represents the discharge sequence generated by the corresponding motion unit during the silent speech task;

[0048] Step 2.4.8: To further obtain the spatial discharge characteristics of motor units, the corresponding average action potential waveform (MUAP waveform template) can be extracted from the original multi-channel electromyography signal based on the discharge peak sequence of each motor unit; at each discharge time point t j ∈PT n Centered on the original signal from multiple channels, waveform segments of fixed window length are extracted. The waveform segments extracted from the same motion unit at multiple discharge times are aligned and averaged to obtain the average action potential waveform of the motion unit on each channel.

[0049] Finally, the output of step 2 is the MUAP waveform template W = {w1, w2, ..., w n}, where w j The average MUAP waveform of the j-th motion unit.

[0050] Step 3: Combining the raw multi-channel surface electromyography (EMG) signals obtained in Step 1 with the motor unit discharge activity information and MUAP waveform template extracted in Step 2, extract the peak-to-peak features related to the motor unit in the time-frequency domain to construct a multimodal feature input for training and testing the silent speech recognition model. This includes the following two sub-steps:

[0051] Step 3.1: For the raw multi-channel surface electromyography (EMG) signal data obtained in Step 1, the time-domain signal is mapped to the time-frequency domain using Short-Time Fourier Transform (STFT) to obtain a more time-varying spectrogram. Specific processing includes: performing STFT processing on the signal of each acquisition channel at different repetition counts; extracting the spectral amplitude using a fixed window length and sliding step size; converting the spectrogram into a logarithmic amplitude plot in decibels to enhance high-frequency detail representation; and finally, standardizing the spectrogram size (e.g., 65×100) to construct a standardized STFT feature matrix.

[0052] Step 3.2: Based on the MUAP waveform template extracted in Step 2, in order to further reflect the electrophysiological activity intensity of each motor unit in different channels, its peak-to-peak P2P characteristics are calculated. The specific process includes: for each motor unit, extract its MUAP waveform template in different channels, calculate the difference between the peak and valley values ​​of the MUAP signal in each channel as the peak-to-peak index, and finally form a multi-channel peak-to-peak vector corresponding to each motor unit to represent its discharge intensity and spatial activation characteristics.

[0053] Through the above two sub-steps, features are extracted from two dimensions: macroscopic time-frequency dynamics (STFT) and microscopic MUAP potential change (P2P), respectively, to provide multimodal input information for subsequent silent speech recognition models.

[0054] Step 4: Construct a silent speech recognition model to classify and recognize silent speech commands: To effectively fuse complementary information from sEMG signals at both macroscopic and microscopic scales, a dual-branch multimodal fusion neural network structure is designed for the classification and recognition of silent speech commands. This step mainly includes the following sub-processes:

[0055] Step 4.1: Construct a dual-branch multimodal fusion neural network structure to build a neural network model based on the dual-branch structure as a silent speech recognition model. The macro-scale input is an m-channel two-dimensional time-frequency image (STFT image), which is convolutionally and pooled using a two-dimensional convolutional neural network (2D-CNN) to extract its spatial structural features. The micro-scale input is an m-channel × MU-number P2P feature vector, which is processed using a multilayer perceptron (MLP) to extract higher-order representations. The deep semantic features output from the two branches are concatenated in the fusion stage and input into a fully connected layer to complete classification prediction.

[0056] Step 4.2: Model Training and Evaluation Strategies

[0057] The model training employed K-Fold Cross Validation to systematically evaluate the generalization ability of the silent speech recognition model. The loss function used was cross-entropy, and the optimizer was Adam. In each training epoch, the accuracy on the validation set was recorded to track the performance changes of the silent speech recognition model during training. The stability and robustness of the silent speech recognition model across multiple sets of samples were evaluated by measuring the accuracy fluctuations at different numbers of folds.

[0058] Compared with the prior art, the method of the present invention has the following advantages:

[0059] 1) By integrating the time-frequency characteristics of macroscopic raw electromyographic signals with the activity characteristics of microscopic motor units, a multi-scale representation of the muscle nerve drive mechanism was achieved, overcoming the problem of detail loss caused by traditional methods that rely solely on macroscopic features.

[0060] 2) Enhance the interpretability of physiological mechanisms. The microscopic features extracted from the decomposition of motor units are directly related to the neuromuscular control mechanism, making the decision-making process of the silent speech recognition model more closely related to physiological activities.

[0061] 3) Improve recognition accuracy: By introducing the microscopic features of the action potential of motor units, the silent speech recognition model can more meticulously depict the neural driving mechanism of muscle movement, thereby enhancing the ability to recognize complex speech movement details and improving the accuracy of silent speech recognition. Attached Figure Description

[0062] Figure 1 This is a flowchart of the overall workflow of a silent speech recognition system.

[0063] Figure 2 This is a schematic diagram of the decomposition process of action potentials in the motor unit of electromyography signals.

[0064] Figure 3 This is a flowchart of the extraction and fusion of microscopic motion unit features and macroscopic time-frequency features.

[0065] Figure 4 This is a schematic diagram of a multimodal fusion neural network structure. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It is obvious that the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] like Figure 1As shown, the present invention provides a silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals, comprising the following steps:

[0068] Step 1: A 13-channel electromyography (EMG) electrode array was deployed in the relevant muscle areas of the subject's face and neck, specifically covering the perioral muscles, buccal muscles, mental muscles, and sternocleidomastoid muscle. The subject performed 10 speech tasks, each repeated 50 times. The EMG waveforms of each channel were synchronously recorded during the task execution using an EMG signal acquisition device to obtain raw multi-channel surface EMG signals.

[0069] Step 2: As Figure 2 As shown, the raw multi-channel surface electromyography signals acquired in step 1 are decomposed and processed to extract the firing activity information of the motor units. The specific steps are as follows:

[0070] The original multi-channel electromyography (EMG) signals were extended using a delayed embedding strategy, with an extension number of 10, meaning 10 delayed versions were added to each channel signal. The extended signals were then centered and whitened to remove correlations between data points.

[0071] The Fast Blind Source Separation (FastICA) algorithm is used to estimate the discharge sequence of the moving units. First, the weight vector of the separation matrix is ​​randomly initialized. Then, the weights are adjusted iteratively, with a maximum number of iterations set to 50. After each iteration, the weights are Gram-Schmidt orthogonal normalized to ensure that the independent components are uncorrelated. If the weight change between two consecutive iterations is less than the convergence threshold, the algorithm is considered convergent, with the convergence threshold set to 1 × 10⁻⁶. -4 .

[0072] Peak detection is performed on the estimated source signal, and the coefficient of variation of its discharge interval is calculated to measure the stability of the discharge. If the coefficient of variation decreases during the optimization process, the weight vector is updated, and the optimal weights are recalculated using the detected discharge times; otherwise, the optimization is terminated, and the current optimal solution is retained.

[0073] The quality of the separated source signals is evaluated using the profile coefficient (SIL), and only reliable motion unit discharge sequences with a profile coefficient greater than 0.9 are retained to ensure the accuracy of the decomposition results.

[0074] Based on the selected discharge times of the motion units, corresponding waveform segments are extracted from the original multi-channel signal, aligned and averaged to obtain the average action potential waveform (MUAP template) of each motion unit on each channel, which serves as the spatial feature for subsequent analysis.

[0075] Step 3: First, perform a Short-Time Fourier Transform (STFT) on the original multi-channel electromyography (EMG) signals. For each acquisition channel, perform STFT processing at different repetition counts, setting the sliding window length to 256 ms and the sliding step size to 100 ms. Extract the spectral amplitude and convert the spectrogram to a logarithmic amplitude plot in decibels to enhance high-frequency detail. Finally, standardize the spectrogram size to 65 × 100 and construct a standardized STFT feature matrix. Next, for the MUAP waveform templates obtained in Step 2, calculate the difference between the peak and valley values ​​of the MUAP signal in each channel as the peak-to-peak value index, ultimately forming a multi-channel peak-to-peak value vector for each motor unit. (Example:) Figure 3 As shown, features are extracted from two dimensions: macroscopic time-frequency dynamics (STFT) and microscopic MUAP potential change (P2P), providing multimodal input information for feature fusion and classification recognition in step 4.

[0076] Step 4: Construct a multimodal fusion neural network to achieve classification and recognition of silent speech commands. For example... Figure 4 As shown, the network comprises two branches: the STFT branch takes a 13×65×100 two-dimensional time-spectrum as input and extracts spatial structure and local energy variation features through two layers of two-dimensional convolution (Conv2D) and pooling (MaxPooling) modules; the P2P branch takes a 3×13-dimensional peak-to-peak sequence as input and uses a two-layer fully connected network (Fully Connected Encoder) for nonlinear compression and feature transformation, with each layer containing a linear mapping and a ReLU activation function. The high-dimensional feature vectors output from the two branches are concatenated and fused to form a multi-scale joint representation, and finally, a fully connected classifier (FC+Softmax) outputs the predicted class.

[0077] After a 5-fold cross-validation experiment, the recognition accuracy of the multimodal fusion neural network reached 0.9618, which is a significant improvement over the convolutional neural network that only uses STFT features (accuracy 0.9153).

[0078] The recognition method of this invention integrates macroscopic surface electromyography signal features with microscopic motor unit discharge activity features to achieve multi-scale and all-round characterization of silent speech signals, thereby improving the accuracy of silent speech recognition and providing more reliable technical support for intelligent interaction and assisted communication.

Claims

1. A silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals, characterized in that, Includes the following steps: Step 1: Using an electromyography (EMG) signal acquisition device, a multi-channel surface electrode array is deployed on the relevant vocal muscles of the subject's face and neck to acquire the raw multi-channel surface EMG signals generated by the subject during the performance of a silent speech task. Step 2: Decompose the raw multichannel surface electromyography signals collected in Step 1 to extract the discharge activity information of the motor units; Step 3: Combine the original multi-channel surface electromyography signals obtained in Step 1 with the motor unit discharge activity information and MUAP waveform template extracted in Step 2, extract the peak-to-peak features related to the motor unit in the time-frequency domain, construct multimodal feature input, and use it for training and testing of the silent speech recognition model; Step 4: Construct a silent speech recognition model to classify and recognize silent speech commands: In order to effectively integrate the complementary information of sEMG signals at the macro and micro scales, a dual-branch multimodal fusion neural network model is designed for the classification and recognition of silent speech commands. Step 3 includes the following sub-steps: Step 3.1: For the original multi-channel surface electromyography signal data obtained in Step 1, the time-domain signal is mapped to the time-frequency domain using the short-time Fourier transform (STFT) to obtain a more time-varying spectrum. Step 3.2: Based on the MUAP waveform template extracted in Step 2, in order to further reflect the electrophysiological activity intensity of each motor unit in different channels, its peak-to-peak P2P characteristics are calculated. The specific process includes: for each motor unit, extract its MUAP waveform template in different channels, calculate the difference between the peak and valley values ​​of the MUAP signal in each channel as the peak-to-peak index, and finally form a multi-channel peak-to-peak vector corresponding to each motor unit to represent its discharge intensity and spatial activation characteristics. Through the above two sub-steps, features are extracted from two dimensions: macroscopic time-frequency dynamics and microscopic MUAP potential changes, respectively, to provide multimodal input information for subsequent silent speech recognition models; Step 4 includes the following sub-steps: Step 4.1: Construct a dual-branch multimodal fusion neural network structure and build a neural network model based on the dual-branch structure as a silent speech recognition model: The macro-scale input is an m-channel two-dimensional time-frequency map, i.e., a STFT image, which is convolutional and pooling operations performed by a two-dimensional convolutional neural network to extract its spatial structure features; the micro-scale input is a P2P feature vector with m channels × MU, which is processed by a multilayer perceptron (MLP) to extract higher-order representations; the deep semantic features output from the two branches are concatenated in the fusion stage and input into a fully connected layer to complete classification prediction; Step 4.2: Model Training and Evaluation Strategies The model training employed a K-fold cross-validation method to systematically evaluate the generalization ability of the silent speech recognition model. The loss function used was cross-entropy, and the optimizer was Adam. In each training epoch, the accuracy of the validation set was recorded to track the performance changes of the silent speech recognition model during the training process. The stability and robustness of the silent speech recognition model on multiple sets of samples were evaluated by the accuracy fluctuations at different fold numbers.

2. The silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals according to claim 1, characterized in that, Step 1 includes the following sub-steps: Step 1.1: A multi-channel surface electrode array is arranged on the relevant vocal muscles of the subject's face and neck. The electrode array contains m channels. The specific electrode arrangement areas include the perioral muscles, buccal muscles, mental muscles, and the sternocleidomastoid muscle, which are closely related to speech production. Step 1.2: The subject performs K speech tasks, each task repeated R times. Electromyography (EMG) waveform data of each channel is synchronously recorded during the task execution using an EMG signal acquisition device to obtain the raw multi-channel surface EMG signal matrix. : in Indicates the first i The first channel in the k Electromyographic signals at discrete moments, The duration of the recording; the output of this step is the raw multichannel surface electromyography signal data. x This serves as the basic input for subsequent electromyography decomposition and feature extraction processing.

3. The silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals according to claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Treat the original multichannel surface electromyography (EMG) signal as a mixture of signals from multiple unobservable motor units, satisfying the following convolutional mixture model: in: Indicates the first The surface electrode channel in the first Electromyography signals acquired at discrete time points Indicates the first Each motion unit at time... The discharge activity sequence reflects whether a discharge occurs at that moment, h ij (l) indicates a channel For the first Each motion unit in delay The response at the location, Indicates the first Each channel at time The noise, H(l) is the convolutional mixing matrix, Indicates at discrete time At that time, all A column vector composed of the discharge activities of each motion unit. For additive noise, For the number of motion units, Number of channels; Step 2.2: Perform observational amplification processing on the original multichannel surface electromyography signals: To improve the stability and decomposition accuracy of the mixing matrix estimation in blind source separation, the original fully determined or underdetermined system is transformed into an overdetermined system, and a delayed embedding strategy is used to add [something] to each channel. R Each delayed version yields an extended observation matrix. : in Indicates the extended One sample; This represents the convolutional blending matrix after channel expansion. This represents the discharge vector of the moving unit after channel expansion. This represents the additive noise term after channel expansion; Step 2.3: Expand the observation matrix Whitening is performed to reduce redundancy between features in the input data; In the whitening process, the data must first be centered, and the input signal... Perform mean normalization to ensure that the data is zero-mean; in It is a centralized extended observation data matrix; Next, the centered data is projected onto a new coordinate system so that its covariance matrix becomes the identity matrix, thereby removing the correlation of the data. The formula is as follows: in, It is the whitened extended observation data matrix. These are the eigenvectors of the covariance matrix. It is an eigenvalue diagonal matrix; Step 2.4: After completing the whitening process of the extended observation matrix in Step 2.3, an improved fast blind source separation algorithm is used to extract independent components from the whitened electromyography data to estimate the firing sequence of motor units. In the improved fast blind source separation algorithm, the goal is to learn a separation matrix. This allows the whitened observation data to be processed through the whitened extended observation data matrix. After projection, several statistically independent component signals can be recovered, which is the estimated firing sequence of the moving unit. Specifically, each row in the separation matrix Indicates the first The projection directions of each independent component can be initialized using a uniform random distribution to generate an initial vector with the unit norm. .

4. The silent speech recognition method based on the fusion of motor unit activity and surface electromyography signals according to claim 3, characterized in that, Step 2.4 includes the following sub-steps: Step 2.4.1: Randomly initialize the weight vector of the separation matrix. ; Step 2.4.2: Iteratively update the weight vector Using nonlinear functions Perform independent component maximization processing: in Indicates the first The weight vector at the next iteration This represents the whitened extended observation data matrix. It is a nonlinear function used to maximize the non-Gaussianity of independent components. It is a function The derivative, This represents the expectation operation over the sample, i.e., the average value over all samples; Step 2.4.3: For all updated weight vectors Gram-Schmidt orthogonal normalization is performed to ensure orthogonality among multiple independent components; Step 2.4.4: Check convergence: If the weight vector changes between two adjacent iterations... If the value is less than the set threshold, it is considered to have converged and proceeds to the next stage; Otherwise, return to step 2.4.2 and continue the iteration; Step 2.4.5: For the currently estimated source signal Peak detection is performed to obtain the discharge pulse train. { }, and calculated its coefficient of variation based on the discharge peak sequence. ; in It is the standard deviation of the discharge pulse train. It is the average value of the discharge pulse train; Step 2.4.6: Perform iterative optimization with coefficient of variation constraints: if This indicates improved discharge stability, so the weight vector is updated: in It is the first The independent component in the first The updated weight vector in +1 iterations It is the discharge peak value The point in time, It is the peak number. At a certain point in time Above, the signal window extracted from the whitened data X; like If the update process fails, the current optimal weight vector is retained. ; Step 2.4.7: Perform an independence quality assessment on the separated source signals using profile coefficients. Measure it: in, This represents the average distance between points within the source signal. This is the average distance between a point within the source signal and the nearest point of another class; only when SIL > 0.9 is the source signal determined to be an effective motion unit activity. This step filters out a set of estimated source signals: Each of them This represents the discharge sequence generated by the corresponding motion unit during the silent sound production task; Step 2.4.8: To further obtain the spatial discharge characteristics of motor units, based on the discharge peak sequence of each motor unit, the corresponding average action potential waveform, i.e., the MUAP waveform template, is extracted from the original multi-channel electromyography signal; at each discharge time point... Centered on the original signal from multiple channels, waveform segments of fixed window length are extracted. The waveform segments extracted from the same motion unit at multiple discharge times are aligned and averaged to obtain the average action potential waveform of the motion unit on each channel.

Images