Semi-supervised decoupled multi-stage myoelectric speech recognition method and system

Abstract

The application discloses a kind of semi-supervised decoupling multi-stage myoelectric speech recognition method and system, belong to speech recognition field.The method of the present application includes using the semi-supervised decoupling myoelectric speech recognition training including three stages to obtain myoelectric data speech recognition model: first stage, pre-train dual-branch decoupling generator using the data samples of two modalities of voiced speech myoelectric data and silent speech myoelectric data of source domain, second stage, retain mode branch in dual-branch decoupling generator, semi-supervised fine-tuning is carried out to mode branch to realize the adaptation of voiced speech myoelectric data mode from source domain to target domain;Third stage, further semi-supervised fine-tuning is carried out to realize the adaptation of voiced speech myoelectric data mode of target domain to silent speech myoelectric data mode to obtain myoelectric data speech recognition model.The present application aims to alleviate the influence of individual difference in myoelectric speech recognition, improve the individual recognition accuracy of data distribution in myoelectric speech recognition.

Description

Technical Field

[0001] This invention relates to the field of speech recognition, specifically to a semi-supervised decoupled multi-stage electromyography speech recognition method and system. Background Technology

[0002] Speech recognition is the cornerstone of human communication. Silent speech recognition (SSR) aims to infer a speaker's intent without requiring audible sound signals, thereby enabling communication in noisy environments, protecting privacy during interactions, and providing assistance to individuals with impaired speech abilities. In recent years, surface electromyography (sEMG), which records the electrical activity generated by muscle fibers, has become a promising technological approach for silent speech recognition. Silent speech recognition was chosen because it can capture neuromuscular activation patterns associated with speech and supports phoneme and word-level recognition. However, surface electromyography-based speech recognition still faces challenges due to variations in the distribution of users, recorded sessions, and speaking conditions, which limits its practical application in silent speech recognition.

[0003] To reduce the specific calibration work required before each use, transfer learning (TL), a method that improves the performance of the target domain by transferring knowledge from a relevant source domain, is gradually becoming a promising strategy for reducing calibration workload. Existing transfer learning research has made significant progress in cross-scene generalization in relevant sEMG speech tasks, including adaptive adjustments to accents, speakers, and recorded conversations. For example, existing techniques (S. Ullah and D. Kim, “Multiaccent emg-to-speech optimized translation with perfl and maml adaptations,” IEEE Transactions on Instrumentation and Measurement, vol. 73, 2024, publisher Copyright: © 2024 IEEE. Personal use is permitted, but republication / redistribution requires IEEE permission) proposed a model-independent meta-learning method based on a Squeeze-and-Excitation network and a gated multilayer perceptron to adapt to data with different accents, reducing word error rate by 1.3% to 3.5% on public datasets. Existing techniques (Q. Cui, X. Zhang, Y. Zhang, C. Zheng, L. Xie, Y. Yan, EQ Wu, and E. Yin, “A simplified adversarial architecture for cross-subject silent speech recognition using electromyography,” J. NeuralEng., 2024.) propose a simplified differential adversarial network for cross-speaker SSR tasks, achieving a 10.07% performance improvement over state-of-the-art methods. Existing techniques (M. Wand, T. Schultz, and J. Schmidhuber, “Domain-adversarial training for session independent emg-based speech recognition,” 09 2018, pp. 3167–3171.) introduce domain adversarial training techniques, enabling the neural network front-end to adapt to new recorded sessions without supervised registration, resulting in a 17.0% relative reduction in word error rate on the development dataset and a 4.7% reduction on the evaluation dataset.These studies support the idea that sEMG characterization contains transferable structures, and that well-designed adaptive adjustments can alleviate the burden of specific data collection.

[0004] While these studies demonstrate the value of transfer learning in sEMG-based silent speech recognition applications, their transfer settings only partially meet the practical needs of silent speech recognition deployments. Cross-subject methods primarily address the issues arising from individual user differences under similar speaking conditions, while cross-conversation methods mainly improve robustness to recording instability or electrode reattachment in the same recognition task. However, in real-world silent speech recognition applications, the target context is often the silent utterances of new users. Because silent speech lacks observable acoustic output for direct verification, obtaining large-scale, high-quality labeled data is extremely difficult. This challenge is even more pronounced in clinical populations: they struggle to maintain lengthy calibration processes or repeated standardized tests, and existing systems still face numerous obstacles in terms of data acquisition, calibration burden, and evaluation in real-world clinical settings. Furthermore, silent speech exhibits weaker motor amplitude and less consistent articulation effort, especially for users with speech production difficulties. These factors make supervised training solely based on target silent speech both costly and unstable. In contrast, spoken speech is easier to elicit and more reliably labeled, thus providing more readily available speech category supervision signals. Therefore, an important practical problem is how to leverage spoken speech information to improve silent speech recognition performance for a specific user when available labeled target silent speech data is limited. This problem is only meaningful when the two speech modalities share transferable speech-related information. Transition strategies from spoken to silent speech are physiologically feasible because spoken and silent speech share the same speech content and involve locally overlapping neuromuscular control of the articulatory organs. This relationship has prompted recent research to utilize spoken or cross-modal speech information to support silent electromyographic signal modeling. For example, existing techniques (D. Gaddy and D. Klein, “Digital Voicing of Silent Speech,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, 2020, pp. 5521–5530.) convert audio target data from vocal recordings into silent electromyographic signals. EMG speech synthesis research has shown that vocalized speech can provide effective supervision for modeling silent speech.

[0005] Research on electromyographic speech synthesis has shown that vocalizations can provide effective supervision for modeling silent speech. Existing techniques (T. Benster, G. Wilson, R. Elisha, FR Willett, and S. Druckmann, “ACross-Modal Approach to Silent Speech with LLM-Enhanced Recognition,” 2024.) further propose a cross-modal silent speech recognition framework that aligns EMG signals with audio features and leverages pure audio datasets to improve silent speech recognition performance. These studies demonstrate that information from relevant vocalization conditions is helpful for silent speech modeling. However, they do not directly address the problem of class-discriminative EMG speech recognition when both the subject and modality change simultaneously. For silent speech recognition classification, the model should form a discriminative speech representation under the target silent speech condition, rather than simply reconstructing or synthesizing speech from EMG signals. This requirement makes vocalizations an attractive source domain because they not only provide richer and more reliable class supervision information but also share the articulation intent with silent speech. However, the distribution of spoken and silent speech differs. Spoken speech also involves vocal cord vibration, airflow control, and acoustic feedback. Previous studies have shown that vocal patterns and the lack of acoustic feedback affect speech recognition based on electromyography (sEMG). Therefore, while spoken speech data can provide valuable category-related information, its direct transfer to silent speech recognition is insufficient. Reducing the distributional differences influenced by subject and modality has become a critical technical problem that urgently needs to be solved in the field of silent speech recognition based on surface electromyography (sEMG) (electromyography-based speech recognition). Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a semi-supervised decoupled multi-stage electromyography (EMG) speech recognition method and system to address the above-mentioned problems in the prior art. The present invention aims to alleviate the influence of individual differences in EMG speech recognition and improve the recognition accuracy of individuals with different data distributions during EMG speech recognition.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A semi-supervised decoupled multi-stage electromyography (EMG) speech recognition method includes training with three stages of semi-supervised decoupled EMG speech recognition to obtain an EMG data speech recognition model: In the first stage, a dual-branch decoupling generator is pre-trained using data samples from two modalities: spoken speech electromyography (EMG) data and silent speech EMG data from the source domain. The dual-branch decoupling generator includes a pattern branch and an individual branch. The pattern branch is used to learn speech-related representations, while the individual branch is used to capture individual-related differences. In the second stage, the pattern branch in the dual-branch decoupling generator is retained, and the pattern branch is semi-supervised fine-tuned using data samples of the spoken speech electromyography data modal from both the source and target domains to achieve the adaptation of the spoken speech electromyography data modal from the source domain to the target domain. In the third stage, for the fine-tuned pattern branches, data samples from two modalities—voiced speech EMG data and silent speech EMG data—from the source domain, as well as data samples from silent speech EMG data in the target domain, are used to further semi-supervised fine-tun the pattern branches to achieve adaptation from the voiced speech EMG data modal to the silent speech EMG data modal in the target domain. This results in an EMG data speech recognition model composed of pattern branches, which can be used for speech recognition of silent speech EMG data in the target domain.

[0008] Optionally, the pattern branches of the dual-branch decoupling generator are generated by the pattern encoder. and speech classifier Composition, the pattern encoder Used for processing input feature maps Encode and generate pattern latent variables The speech classifier Used for latent variables of the pattern The classification identifies the corresponding speech category prototype in the pattern space; the individual branches of the dual-branch decoupling generator are generated by individual encoders. and individual classifiers Composition, the individual encoder Used for processing input feature maps Encode to generate individual latent variables The individual classifier Used for individual latent variables The classification identifies the corresponding individual identity prototype in the individual subject space, where the input feature map The feature map is extracted from labeled acoustic or aphonic electromyography (EMG) data of individuals in the source domain.

[0009] Optionally, the dual-branch decoupling generator further includes a shared decoder. and the gradient inversion layer GRL, the shared decoder Latent variables used to determine the pattern based on the input and individual latent variables Perform reconstruction to obtain reconstruction feature maps The gradient inversion layer (GRL) is used to invert the gradient direction of the auxiliary classifier during backpropagation to suppress pattern latent variables. Individual identity information and individual latent variables The speech category information in the data; when pre-training the dual-branch decoupling generator using data samples from two modalities—voiced speech EMG data and silent speech EMG data—in the source domain, the loss function used is the first-stage loss, and the functional expression of the first-stage loss is: ; ; ; ; in, For the losses in the first phase, and These are the weighting coefficients. To constrain the input feature map and its reconstructed feature map Reconstruction loss due to differences between them To reduce model latent variables and individual latent variables The decoupling loss of information coupling between them The alignment loss is the corresponding speech category prototype in the pattern space and the corresponding individual identity prototype in the individual subject space. and These are the weighting coefficients. The self-reconstruction loss is used to constrain the shared decoder. Based on the pattern latent variables of the same sample and individual latent variables Reconstruct the original input electromyographic feature map to ensure that the latent variables retain effective information of the input signal; The cross-variable cycle consistency loss is used to constrain the pattern latent variables and the permuted individual latent variables through a shared decoder. After reconstruction, the pattern latent variables and individual latent variables obtained by re-encoding are consistent with the original pattern latent variables and the replaced individual latent variables, respectively. For balance coefficient, This is an adversarial classification loss implemented through a gradient inversion layer (GRL). For use as latent variables in constrained patterns and individual latent variables Orthogonal constraint loss of geometric correlation The loss for class discrimination of the speech category prototype is given by the pattern branch. The individual branch represents the individual discrimination loss of the individual identity prototype.

[0010] Optionally, rebuild loss The expression for the computation function is: ; ; , , ; Adversarial classification loss The expression for the computation function is: ; Orthogonal constraint loss The expression for the computation function is: ; Category-based loss The expression for the computation function is: ; Individual discrimination loss The expression for the computation function is: ; in, For batch size, and The first Reconstructed feature map and input feature map of each sample. Indicates shared decoder , and These represent the mode encoders respectively. and individual encoders , and For feature map Corresponding pattern latent variables and individual latent variables Feature map for and Both are achieved through a shared decoder The feature map obtained by decoding, For the first random permutation within a batch Individual latent variables of a sample, among which This indicates that the index of the sample within the batch has been randomly permuted and compared with the first sample. Index of sample pairings For individual classifiers When the input is a pattern latent variable After Gradient Reversal Layer (GRL), the prediction is performed for the first... Each sample belongs to the individual identity prototype label. The probability, and The first Pattern latent variables and individual latent variables for each sample. For the first Prototype labels for the speech categories of each sample. For individual classifiers, For speech classifiers, for transpose, Pattern branching based on given latent variables Predicting speech category prototype labels The probability, These are the regularization weight coefficients. For the first branch in the pattern Pattern latent variables for each sample To its voice category prototype label The Mahalanobis distance corresponding to the Gaussian distribution, Prototype labels for the speech category in the pattern branch The precision matrix, wherein the precision matrix is ​​the inverse of the covariance matrix; Given individual latent variables in individual branches Time prediction source domain first Each sample belongs to the individual identity prototype label. The probability, These are the regularization weight coefficients. For the source domain in the individual branch Individual latent variables of a sample To its individual identity prototype label The Mahalanobis distance corresponding to the Gaussian distribution, Individual identity prototype label in individual branches The accuracy matrix.

[0011] Optionally, the pattern encoder and individual encoders Both have the same structure, including multiple cascaded encoding convolutional blocks, flattening layers, and latent variable layers. Each encoding convolutional block consists of sequentially connected 2D convolutional layers (Conv2d), batch normalization (BN) layers, and LReLU activation functions. The output of the last convolutional block is flattened by the flattening layer and then input to the latent variable layer. This latent variable layer is a linear layer used to map the input features to pattern latent variables of a specified dimension. or individual latent variables The shared decoder It includes an input layer, multiple cascaded decoding convolutional blocks, and an output layer, where the input layer is used to process the pattern encoder through linear layers. and individual encoders The concatenated features are linearized and output to the first decoding convolutional block. This decoding convolutional block consists of a sequentially connected upsampling module, a 2D convolutional layer (Conv2d), a batch normalization (BN) layer, and an LReLU activation function. The output layer consists of a 2D convolutional layer (Conv2d) and a hyperbolic tangent layer (Tanh) to reconstruct the input feature map into a reconstructed feature map. .

[0012] Optionally, the loss function used in the second stage is the second-stage loss, and the functional expression of the second-stage loss is: ; in, For the losses in the second phase, , , and For the loss weight in the second stage, The second stage uses cross-entropy classification loss for speech category supervision based on labeled samples. The second stage uses supervised representation constraint loss to enhance the clustering of similar samples and the separation of dissimilar samples. This is the alignment loss for the statistical distribution of the prototype speech categories in the pattern space corresponding to the source and target domain electromyographic data of spoken speech in the second stage. This is the prototype anchoring loss used in the second stage to constrain the class prototype offset during the target domain adaptation process.

[0013] Optionally, the loss function used in the third stage is the third-stage loss, and the functional expression of the third-stage loss is: ; in, For the losses in the third phase, , , and For the loss weight in the third stage, The third stage uses cross-entropy classification loss for speech category supervision based on labeled samples. The third stage uses supervised representation constraint loss to enhance the clustering of similar samples and the separation of dissimilar samples. The modal alignment loss in the third stage is used to align the statistical distribution differences between spoken and silent speech EMG data in the pattern latent variable space. This is the prototype anchoring loss used in the third stage to constrain the class prototype offset during the target domain adaptation process.

[0014] Optionally, the alignment loss calculation function for the corresponding speech category prototype statistical distributions of the source and target domain spoken electromyography data in the pattern space during the second stage is expressed as follows: ; ; ; The global CORAL alignment loss for spoken speech electromyography data modality is used to align the covariance of the overall representation distributions of the source and target domains. This is a category-level CORAL alignment loss for spoken speech electromyography data modalities, used to class-wise align the covariance of the source and target domain representations. These are the weighting coefficients for the second stage. Latent variables of the model and individual latent variables Feature dimensions, To find the covariance matrix, and These are the sets of pattern latent variables for modal samples of spoken speech electromyography data from the source and target domains, respectively. For the set of categories participating in category-level CORAL alignment, and The first and second digits in the source and target domains, respectively. The set of pattern latent variables for modal samples of electromyography data in the form of spoken speech; The expression for the modal alignment loss used in the third stage to align the statistical distribution differences between spoken and silent speech EMG data in the pattern latent variable space is as follows: ; ; ; in, For the third stage modal alignment loss, The third stage global CORAL alignment loss, For the third stage, class-level CORAL alignment loss, These are the weighting coefficients for the third-stage category-level CORAL alignment loss. Latent variables of the model Feature dimensions, The function for calculating the covariance matrix. The third stage of electromyography samples of spoken language were processed by a pattern encoder. The set of pattern latent variables obtained after encoding and normalization This is the set of pattern latent variables for all silent speech electromyography samples that can be used for global modality alignment in the third stage. Let be the set of latent variables for the electromyographic sample patterns of spoken speech corresponding to the c-th speech category. Let c be the set of latent variables representing the labeled silent speech electromyography (EMG) sample patterns corresponding to the c-th speech category in the target domain. It is a set of speech categories that exist simultaneously in both spoken samples and labeled silent samples in the target domain; The calculation function expression for the cross-entropy classification loss is as follows: ; ; ; , ; ; ; ; , ; in, The cross-entropy classification loss is used for labeled samples in the source domain of spoken speech electromyography data modal. These are the weighting coefficients. The cross-entropy classification loss is used for a small number of labeled samples in the target domain under the spoken speech electromyography data modality. and These are the batch sizes for the source and destination domains, respectively. For the first in the source domain A sample in a given Predict speech category under the condition The probability, For the first in the source domain Voice category labels for a single audio-visual electromyography (EMG) sample. For the first in the source domain Pattern latent variables of electromyography samples of spoken speech. For the first in the target domain A given set of labeled spoken electromyographic samples Predict speech category under the condition The probability, For the first in the target domain Voice category labels for labeled electromyographic samples of spoken voice. For the target domain, there are pattern latent variables for electromyographic samples of labeled spoken language. For normalization operations, For the first in the source domain There are 3 samples of spoken speech electromyography (EMG) data, where s represents the source domain and v represents the spoken speech EMG data modality. For the first in the target domain There are 3 labeled electromyographic (EMG) data samples of spoken language, where t represents the target domain and lab represents the labeled sample. and These represent the mode encoders respectively. and individual encoders , These are the weighting coefficients. The cross-entropy classification loss is used for the spoken speech EMG data samples in the third stage to maintain the discriminative ability of the pattern branch for spoken speech categories. The cross-entropy classification loss for labeled silent speech EMG data samples in the target domain in the third stage is used to semantically correct the pattern branch using a small number of silent speech calibration samples. The batch size for the electromyography (EMG) data samples in the third phase. The batch size for labeled silent speech electromyography data samples in the target domain during the third stage. For the first in the target domain A sample in a given Predict speech category under the condition The probability, For the first in the target domain Voice category labels for a single audio-visual electromyography (EMG) sample. For the first in the source domain Pattern latent variables of electromyography samples of spoken speech. For the i-th labeled silent speech electromyography sample in the target domain, given its pattern latent variables Under the given conditions, it is predicted by the speech classifier as the speech category label. The probability of; The speech category label is the i-th labeled silent speech electromyography sample in the target domain; For the i-th labeled silent speech electromyography sample in the target domain, the pattern encoder The pattern latent variables obtained after encoding and normalization; The expression for the calculation function of the supervised representation constraint loss is: ; ; ; ; in, and These are the weighting coefficients. To monitor and compare learning loss, For unlabeled samples Figure 1 Sexual damage, Let i be the set of positive samples for the i-th sample. The set of positive samples of the i-th sample is the same as that of the j-th sample. The set of sample indices for each sample i belonging to the same speech category. For similarity function, For the first Pattern latent variables for each sample For pattern latent variables, For the first The pattern latent variables of the nth sample, the th Each sample is a negative sample in contrastive learning. For temperature coefficient, For unlabeled samples, For the target domain Unlabeled type The first modality of spoken electromyography data One sample, , and This is a random enhancement function for the samples, used to add random noise to the samples; For the first in the target domain One unlabeled sample of electromyographic data with spoken words, including This indicates unlabeled samples; The expression for the calculation function of the prototype anchoring loss is: ; ; ; in, The total number of voice categories. This serves as the category prototype for the c-th speech category in the target domain spoken speech electromyography data modality obtained in the second stage of learning. The category prototype for the c-th speech category in the source domain spoken speech electromyography data modality is derived from the pattern branch prototype obtained in the first stage of pre-training. The set of categories appearing in labeled silent samples in the target domain. The pattern latent variable centers are defined for the c-th class of labeled silent speech electromyography samples in the target domain. The target domain is defined as the number of labeled silent speech electromyography samples belonging to class c. This indicates that the summation is only performed on the speech category label. The target domain contains labeled silent samples. For the first in the target domain One labeled silent speech electromyography sample.

[0015] Optionally, after obtaining the electromyographic data speech recognition model by employing a three-stage semi-supervised decoupled electromyographic speech recognition training, the method further includes using the electromyographic data speech recognition model to perform electromyographic data speech recognition in the target domain. S1, acquire multi-channel surface electromyography signals at the vocal muscle groups of an individual in the target domain when uttering silent speech; S2, preprocesses and reduces noise from multi-channel surface electromyography signals; S3 performs abnormal channel correction on the pre-processed and noise-reduced multi-channel surface electromyography signals; S4. Feature extraction is performed on the multi-channel surface electromyography (EMG) signal after abnormal channel correction. This includes using a sliding window strategy to segment the multi-channel EMG signal. For each segment, the root mean square (RMS) and waveform length are extracted from that segment. The features of the extracted segments are used to construct a feature tensor based on the electrode spatial layout of each channel as the obtained target domain silent speech EMG data. The target domain silent speech EMG data is then input into the EMG data speech recognition model to obtain the syllables corresponding to the silent speech of an individual in the target domain.

[0016] The present invention also provides a semi-supervised decoupled multi-stage electromyographic speech recognition system, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the semi-supervised decoupled multi-stage electromyographic speech recognition method.

[0017] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The method of the present invention includes a three-stage semi-supervised decoupled electromyography (EMG) speech recognition training to obtain an EMG data speech recognition model: In the first stage, a dual-branch decoupling generator is pre-trained using data samples from two modalities, namely, spoken EMG data and silent EMG data from the source domain, to learn a decoupled representation that can separate speech pattern information from individual differences in EMG signals; In the second stage, the pattern branch in the dual-branch decoupling generator is retained, and the pattern branch is semi-supervisedly fine-tuned to achieve adaptation of the spoken EMG data modality from the source domain to the target domain; In the third stage, further semi-supervised fine-tuning is performed to achieve adaptation of the spoken EMG data modality from the target domain to the silent EMG data modality to obtain an EMG data speech recognition model. Thus, feature decoupling is used to perform semi-supervised training for speech recognition based on multi-channel surface EMG signals, realizing cross-modal transfer from spoken to silent pronunciation, adapting this modal representation to the vocal modality of the target subject, and aligning the adapted representation with silent pronunciation. This invention, through the above three stages of semi-supervised decoupled electromyographic speech recognition training, can effectively alleviate the influence of individual differences in electromyographic speech recognition and significantly improve the recognition accuracy of individuals with different data distributions during electromyographic speech recognition. Attached Figure Description

[0018] Figure 1This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the principle of the three-stage semi-supervised decoupled electromyography speech recognition training in this embodiment of the invention.

[0020] Figure 3 This is a schematic diagram illustrating the characterization changes in the three stages of an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram illustrating the feature extraction principle in an embodiment of the present invention.

[0022] Figure 5 This invention relates to the data differences between two modalities: audible speech electromyography (EMG) data and silent speech EMG data, as described in this embodiment. Figure 5 (a) is the root mean square plot of electromyography data from spoken speech. Figure 5 (b) is the root mean square plot of the electromyography data for silent speech. Figure 5 (c) is a plot of the root mean square difference between the two modalities of electromyography (EMG) data: spoken speech and silent speech.

[0023] Figure 6 These are the root mean square plots of different individuals in this embodiment of the invention, wherein... Figure 6 (a)~ Figure 6 (l) are the root mean square plots of individuals S1 to S12 respectively.

[0024] Figure 7 These are violin diagrams showing the distance between the centroids under the same and different conditions in embodiments of the present invention. Figure 7 (a) is a violin diagram of the distance between the centroids under the same conditions. Figure 7 (b) is a violin diagram of the distance between the centroids under different conditions.

[0025] Figure 8 The centroid coordinates divided by mode and the sound in the embodiments of the present invention The distance between the centroids in the silent state, where Figure 8 (a) is the x-coordinate of the centroid. Figure 8 (b) is the y-coordinate of the centroid. Figure 8 (c) is audio The distance between the centroids in a silent state.

[0026] Figure 9 The centroid coordinates divided by individual units and the sound in the embodiments of the present invention The distance between the centroids in a silent state, where Figure 9 (a) is the x-coordinate of the centroid. Figure 9 (b) is the y-coordinate of the centroid. Figure 9 (c) is audio The distance between the centroids in a silent state.

[0027] Figure 10 This represents the nearest prototype probe in the first stage latent space and the nearest prototype confusion matrix before and after the second and third stages in this embodiment of the invention. Figure 10 (a) represents the latent variables of the first-stage model. The most recent prototype probe, Figure 10 (b) represents the individual latent variables in the first stage. The most recent prototype probe, Figure 10 (c) is the most recent prototype confusion matrix before the second stage. Figure 10 (d) is the most recent prototype confusion matrix after the second stage. Figure 10 (e) is the difference matrix between the nearest prototype confusion matrices before and after the second stage. Figure 10 (f) is the most recent prototype confusion matrix before the third stage. Figure 10 (g) is the most recent prototype confusion matrix after the third stage. Figure 10 (h) is the difference matrix between the nearest prototype confusion matrices before and after the third stage.

[0028] Figure 11 This is a schematic diagram of feature space comparison in an embodiment of the present invention, wherein... Figure 11 (a) is the initial feature space. Figure 11 (b) is the final feature space.

[0029] Figure 12 For the purpose of comparing the profile coefficient analysis results on the retained samples in this embodiment of the invention, wherein Figure 12 (a) shows the contour coefficient analysis results in the initial feature space. Figure 12 (b) shows the contour coefficient analysis results of the final feature space.

[0030] Figure 13 This invention presents a comparison of the accuracy and macro-mean F1 score of CDAN and DMST at the subject level in this embodiment. Figure 13 (a) represents the accuracy rate. Figure 13 (b) is the macro average F1 score.

[0031] Figure 14 This is a comparison of the recall rates of CDAN and DMST across different categories in this embodiment of the invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] like Figure 1 and Figure 2As shown, the semi-supervised decoupled multi-stage electromyography (EMG) speech recognition method in this embodiment includes training with three stages of semi-supervised decoupled EMG speech recognition to obtain an EMG data speech recognition model: In the first stage, a dual-branch decoupling generator is pre-trained using data samples from two modalities: spoken speech electromyography (EMG) data and silent speech EMG data from the source domain. The dual-branch decoupling generator includes a pattern branch and an individual branch. The pattern branch is used to learn speech-related representations, while the individual branch is used to capture individual-related differences. In the second stage, the pattern branch in the dual-branch decoupling generator is retained, and the pattern branch is semi-supervised fine-tuned using data samples of the spoken speech electromyography data modal from both the source and target domains to achieve the adaptation of the spoken speech electromyography data modal from the source domain to the target domain. In the third stage, for the fine-tuned pattern branches, data samples from two modalities—voiced speech EMG data and silent speech EMG data—from the source domain, as well as data samples from silent speech EMG data in the target domain, are used to further semi-supervised fine-tun the pattern branches to achieve adaptation from the voiced speech EMG data modal to the silent speech EMG data modal in the target domain. This results in an EMG data speech recognition model composed of pattern branches, which can be used for speech recognition of silent speech EMG data in the target domain.

[0034] In this embodiment, data samples for both spoken and silent speech electromyography (EMG) modalities were acquired using four 8×8 electrode arrays, A1 to A4. The vocal muscles used were the following muscles closely related to speech: zygomaticus major, orbicularis oris, levator anguli oris, platysma, sternocleidomastoid, cricothyroid, and thyrohyoid muscles. Each of the four electrode arrays A1 to A4 contained 8×8 electrodes, and their channels were arranged in a 16×16 spatial grid. Electrode arrays A1 and A2 were used to acquire activity signals from the zygomaticus major, orbicularis oris, and levator anguli oris; electrode arrays A3 and A4 were used to record signals from the platysma, sternocleidomastoid, cricothyroid, and thyrohyoid muscles. A reference electrode was placed on the mastoid process behind each ear. Throughout the experiment, the electrode placement remained consistent for all subjects. The surface EMG signals were recorded using a Quattrocento system with an ADC resolution of 16 bits, a sampling rate of 2048 Hz, and a gain of 150. During data collection, participants sat comfortably and followed on-screen instructions. Two consecutive sessions were recorded without changing electrode positions—one in spoken mode and the other in silent mode. In spoken mode, participants pronounced each phoneme at a moderate volume, following natural pronunciation habits; in silent mode, they pronounced the phonemes clearly but without acoustic noise. Before the formal experiment, participants practiced switching between spoken and silent pronunciation until they were proficient in silent pronunciation. Each session contained 10 consonant tasks presented in a randomized order, using the same experimental paradigm for both spoken and silent modes. These phonemes were accompanied by visual cues. For each phoneme, participants completed two test blocks. Each test block contained three 1-second trial-rest pairs, with a 1-second rest interval between each trial. A 5-second rest period was provided between the two test blocks to reduce muscle fatigue. Auditory cues were used to mark the start of each trial, while synchronized trigger signals were sent at the start and end of each trial to accurately segment the surface electromyography (EMG) signals. A total of 120 consonant trials were included for each subject across both modes. Each trial contained 2048 sampling points. Subjects could pause or continue the experiment as needed; missed or incorrect trials were recorded and excluded. On average, 0.4 ± 0.8 consonant trials were excluded in the SS dataset (silent speech EMG dataset), while 1.3 ± 1.1 consonant trials were excluded in the VS dataset (voiced speech EMG dataset).

[0035] After acquiring multi-channel surface electromyography (EMG) signals from the vocal muscles of an individual during silent speech, further processing is required, including preprocessing for noise reduction, outlier channel correction, feature extraction, and additional labeling of labeled samples. This yields data samples in two modalities: spoken and silent speech EMG. Outlier channel correction mitigates the impact of local channel failures or anomalous sampling on feature stability. Channels with feature values ​​exceeding three standard deviations from the regional mean are identified as outliers and scaled proportionally based on the statistical data of adjacent channels, for example, by scaling to the mean of adjacent channels.

[0036] In this embodiment, preprocessing noise reduction includes: first, filtering the multi-channel surface electromyography (EMG) signals through a Butterworth filter with a high-pass cutoff frequency of 10Hz and a low-pass cutoff frequency of 500Hz (using a zero-phase digital filter, processed separately in the forward and reverse directions, order eight). Processing is performed separately for each direction. 10Hz is chosen as the high-pass cutoff frequency to attenuate motion artifacts, while the 500Hz low-pass cutoff frequency is used to suppress high-frequency noise. Subsequently, a set of notch filters is applied to remove electric field interference at 50Hz and its harmonics, as multi-channel EMG signal recordings are often affected by such interference. In subsequent analysis, the first 0.25 seconds of data related to the expected reaction time are excluded, retaining only a stable 0.75-second segment for further processing.

[0037] In this embodiment, feature extraction includes: segmenting the multi-channel surface electromyography (EMG) signal using a sliding window strategy; extracting the root mean square (RMS) and waveform length from each segment; and constructing a feature tensor based on the electrode spatial layout of each channel to obtain the target domain silent speech EMG data. To fully utilize the spatial distribution information of the multi-channel EMG signal, this embodiment maps each channel of the multi-channel EMG signal to a two-dimensional topological representation based on the spatial structure of the electrodes, such as... Figure 4As shown, four 8×8 electrode regions (A1-A4) are combined into a unified 16×16 grid, thus aligning the positions on the feature map with adjacent recording points on the skin surface. When segmenting the signal using a sliding window strategy, a 200-millisecond (ms) sliding window with a 50-millisecond step is used for each segment, and the root mean square (RMS) and waveform length (WL) are extracted from each segment. Thus, the three segments within each window, with RMS and waveform length extracted from each segment, can ultimately be represented as a two-dimensional feature map containing six feature planes. To mitigate the impact of absolute amplitude differences between samples, an intra-sample normalization method is employed. Normalization is performed on each feature channel of each sample. Finally, the model input is represented as a 6×16×16 feature tensor. This representation not only preserves the spatial relationships between channels but also matches the input format of the convolutional neural network (CNN) in the encoder, enabling simultaneous encoding of spatiotemporal frequency multidimensional information in the sEMG signal.

[0038] like Figure 2 As shown, in this embodiment, the pattern branches of the dual-branch decoupling generator are generated by the pattern encoder. and speech classifier Composition, the pattern encoder Used for processing input feature maps Encode and generate pattern latent variables The speech classifier Used for latent variables of the pattern The classification identifies the corresponding speech category prototype in the pattern space; the individual branches of the dual-branch decoupling generator are generated by individual encoders. and individual classifiers Composition, the individual encoder Used for processing input feature maps Encode to generate individual latent variables The individual classifier Used for individual latent variables The classification identifies the corresponding individual identity prototype in the individual subject space, where the input feature map Feature maps are extracted from labeled acoustic or silent electromyography (EMG) data of individuals in the source domain. In the first stage, a two-branch decoupling generator is trained; in the subsequent second and third stages, only the pattern branch from the two-branch decoupling generator is retained and fine-tuned and adapted. For example... Figure 3 As shown, in the first stage, the original feature map is divided into pattern latent variables. and individual latent variables The two types of features are: the second stage is to adapt the spoken speech electromyography data modality from the source domain to the target domain, and the third stage is to adapt the spoken speech electromyography data modality in the target domain to the silent speech electromyography data modality.

[0039] like Figure 2 As shown, the dual-branch decoupling generator in this embodiment also includes a shared decoder. and the gradient inversion layer GRL, the shared decoder Latent variables used to determine the pattern based on the input and individual latent variables Perform reconstruction to obtain reconstruction feature maps The gradient inversion layer (GRL) is used to invert the gradient direction of the auxiliary classifier during backpropagation to suppress pattern latent variables. Individual identity information and individual latent variables The voice category information in the text can be represented as: ; The reconstructed paths prevent latent variables from discarding information needed to describe the original sEMG feature maps, while the dual-branch structure provides independent paths for pattern-related factors and subject-related factors. To assign explicit semantics to these two latent spaces, prototype-based classifiers are attached to the two branches respectively. Connected speech classifier Define speech category prototypes in pattern space, and with Connected individual classifiers Then, a subject identity prototype is defined in the subject space. Furthermore, adversarial classification heads are connected via a gradient inversion layer (GRL) to suppress information leakage between branches: the GRL is used to invert the gradient direction of the auxiliary classifier during backpropagation to suppress pattern latent variables. Individual identity information and individual latent variables The speech category information. These auxiliary individual classifiers and shared decoder This is only used in the first phase. In subsequent migration phases, only the pattern encoder for the pattern branch is retained. and its pattern classifier This enables personalized targeting and the conversion of spoken electromyography (EMG) data to silent EMG data (VS). to SS alignment provides a transferable representation.

[0040] In this embodiment, when pre-training the two-branch decoupling generator using data samples from both spoken and silent speech electromyography (EMG) modalities in the source domain, the loss function used is the first-stage loss. In the first stage, under a leave-one-out method, the generator is trained using spoken and silent data from multiple subjects. The first-stage loss aims to establish transferable pattern latent variables for subsequent stages. Its goal is to... It is discriminatory in terms of pronunciation patterns but insensitive to the identity of the subject; at the same time, it makes It can reflect the subject-specific variation information and minimizes the predictive power for speech categories. The functional expression of the loss in the first stage is: ; ; ; ; in, For the losses in the first phase, and These are the weighting coefficients. To constrain the input feature map and its reconstructed feature map Reconstruction loss due to differences between them To reduce model latent variables and individual latent variables The decoupling loss of information coupling between them The alignment loss is the corresponding speech category prototype in the pattern space and the corresponding individual identity prototype in the individual subject space. and These are the weighting coefficients. The self-reconstruction loss is used to constrain the shared decoder. Based on the pattern latent variables of the same sample and individual latent variables Reconstruct the original input electromyographic feature map to ensure that the latent variables retain effective information of the input signal; The cross-variable cycle consistency loss is used to constrain the pattern latent variables and the permuted individual latent variables through a shared decoder. After reconstruction, the re-encoded pattern latent variables and individual latent variables are consistent with the original pattern latent variables and the permuted individual latent variables, respectively. This loss can prevent the pattern branches and individual branches from learning mixed representations through ordinary reconstruction tasks and promote the separability and composability of pattern latent variables and individual latent variables in the generation space. For balance coefficient, This is an adversarial classification loss implemented through a gradient inversion layer (GRL). For use as latent variables in constrained patterns and individual latent variables Orthogonal constraint loss of geometric correlation The loss for class discrimination of the speech category prototype is given by the pattern branch. This is an individual discrimination loss for individual branches on individual identity prototypes. The loss represents each speech category or individual identity as a Gaussian prototype distribution in the latent variable space, constraining samples to cluster towards their corresponding category prototypes through Mahalanobis distance, while maintaining distinction from other category prototypes. This prototype structure not only improves the discriminativeness of the decoupled representation in the first stage but also provides a stable semantic reference for prototype anchoring in the second and third stages.

[0041] Among them, reconstruction losses The expression for the computation function is: ; ; , , ; Adversarial classification loss The expression for the computation function is: ; , ; Orthogonal constraint loss The expression for the computation function is: ; Category-based loss The expression for the computation function is: ; Individual discrimination loss The expression for the computation function is: ; in, For batch size, and The first Reconstructed feature map and input feature map of each sample. For shared decoder, and These represent the mode encoders respectively. and individual encoders , and For feature map Corresponding pattern latent variables and individual latent variables Feature map for and Both obtain feature maps through a shared decoder. For the first random permutation within a batch Individual latent variables of a sample, among which This indicates that the index of the sample within the batch has been randomly permuted and compared with the first sample. Index of sample pairings For individual classifiers When the input is a pattern latent variable After Gradient Reversal Layer (GRL), the prediction is performed for the first... Each sample belongs to the individual identity prototype label. The probability, and The first Pattern latent variables and individual latent variables for each sample. For the first Prototype labels for the speech categories of each sample. For individual classifiers, For speech classifiers, for transpose, Pattern branching based on given latent variables Predicting speech category prototype labels The probability, These are the regularization weight coefficients. For the first branch in the pattern Pattern latent variables for each sample To its voice category prototype label The Mahalanobis distance corresponding to the Gaussian distribution, Prototype labels for the speech category in the pattern branch The precision matrix, wherein the precision matrix is ​​the inverse of the covariance matrix; Given individual latent variables in individual branches Time prediction source domain first Each sample belongs to the individual identity prototype label. The probability, These are the regularization weight coefficients. For the source domain in the individual branch Individual latent variables of a sample To its individual identity prototype label The Mahalanobis distance corresponding to the Gaussian distribution, Individual identity prototype label in individual branches The accuracy matrix. Here, the Mahalanobis distance is a known existing distance algorithm, using... For example, its calculation function expression is: ; in, The first Pattern latent variables for each sample for To its voice category prototype label The Mahalanobis distance corresponding to the Gaussian distribution, For the first branch in the pattern Each speech category corresponds to a Gaussian prototype center. Prototype labels for the speech category in the pattern branch The precision matrix, in the superscript This indicates transpose.

[0042] In this embodiment, the pattern encoder and individual encoders Both have the same structure and can be configured with the desired encoder structure. For example, both include multiple cascaded encoding convolutional blocks, flattening layers, and latent variable layers. The encoding convolutional block consists of sequentially connected 2D convolutional blocks (Conv2d), batch normalization (BN) layers, and LReLU activation functions. The output of the last convolutional block is flattened by the flattening layer and then input to the latent variable layer. The latent variable layer is a linear layer used to map the input features to pattern latent variables of a specified dimension. or individual latent variables The shared decoder It includes an input layer, multiple cascaded decoding convolutional blocks, and an output layer, where the input layer is used to process the pattern encoder through linear layers. and individual encoders The concatenated features are linearized and output to the first decoding convolutional block. This decoding convolutional block consists of a sequentially connected upsampling module, a 2D convolutional layer (Conv2d), a batch normalization (BN) layer, and an LReLU activation function. The output layer consists of a 2D convolutional layer (Conv2d) and a hyperbolic tangent layer (Tanh) to reconstruct the input feature map into a reconstructed feature map. .

[0043] The second stage adapts the VS (voiced speech electromyography data) representations from the source subjects to the target subjects. In this stage, a semi-supervised learning framework is employed under the LOSO setting. The source domain contains labeled VS samples from the source subjects, providing a global speech category structure learned from multiple subjects. The target domain contains VS samples from the retained subjects, which are divided into three subsets at the trial level: a small labeled calibration subset, an unlabeled adaptation subset, and a validation subset. This design achieves small-sample supervised adaptation while avoiding information leakage due to overlapping sliding window samples. After pre-training in the first stage, the subject branch is discarded. Pattern latent variables are fine-tuned in the second stage, where low-level parameters remain unchanged, while high-level pattern encoding layers and target-specific prototype parameters are updated. This design preserves general feature extraction capabilities while allowing high-level decision boundaries to be adapted to the target subjects. This embodiment... The loss function used in the second stage is the second-stage loss, and its functional expression is as follows: ; in, For the losses in the second phase, , , and For the loss weight in the second stage, The second stage uses cross-entropy classification loss for speech category supervision based on labeled samples. The second stage uses supervised representation constraint loss to enhance the clustering of similar samples and the separation of dissimilar samples. This is the alignment loss for the statistical distribution of the prototype speech categories in the pattern space corresponding to the source and target domain electromyographic data of spoken speech in the second stage. This is the prototype anchoring loss used in the second stage to constrain the class prototype shift during the target domain adaptation process. Cross-entropy classification loss. Labeled source and target speech subsets are used to preserve supervisory information during the adaptation process. Supervised representation constraint loss. The target structure is further strengthened by enhancing compactness within the same category and separation between different categories. Alignment loss is used to reduce inter-subject variability between the source and target. It will align their feature statistics in the model's latent variables. At the prototype level, this is the prototype anchoring loss. The updated prototype is anchored to the category structure learned from the first stage, preventing the representation from deviating from the global speech manifold. These losses collectively adapt the high-level pattern space to the target subject while preserving the pattern structure required for subsequent speech category transformation.

[0044] The third stage employs the same semi-supervised adaptation structure as the second stage, but changes the transfer objective to modal alignment between VS (voiced speech electromyography data) and SS (silent speech electromyography data). The training data consists of four parts: labeled VS samples and unlabeled SS samples from source subjects, a small subset of labeled SS calibration data from target subjects, and unlabeled SS samples. The SS data related to the target subjects is partitioned at the trial level, as in the second stage, to ensure that the final tests remain independent. This setup allows the model to utilize both the silence domain structure of the source subjects and the target-specific silence calibration data. The model and the voiced prototypes trained in the second stage are used for initialization; therefore, the pattern encoder and prototype classifier already contain target subject adaptation information learned from the VS modality. The loss function used in the third stage of this embodiment is the third-stage loss, and its functional expression is: ; in, For the losses in the third phase, , , and For the loss weight in the third stage, The third stage uses cross-entropy classification loss for speech category supervision based on labeled samples. The third stage uses supervised representation constraint loss to enhance the clustering of similar samples and the separation of dissimilar samples. The modal alignment loss in the third stage is used to align the statistical distribution differences between spoken and silent speech EMG data in the pattern latent variable space. The prototype anchoring loss, used in the third stage to constrain the class prototype shift during target domain adaptation, and the modal alignment loss, used to align the statistical distribution differences between spoken and silent speech EMG data in the pattern latent variable space. In this embodiment, the function expression for calculating the alignment loss is: ; ; ; in, The global CORAL alignment loss for spoken speech electromyography data modality is used to align the covariance of the overall representation distributions of the source and target domains. This is a category-level CORAL alignment loss for spoken speech electromyography data modalities, used to class-wise align the covariance of the source and target domain representations. These are the weighting coefficients for the second stage. Latent variables of the model Feature dimensions, To find the covariance matrix, and These are the sets of pattern latent variables for modal samples of spoken speech electromyography data from the source and target domains, respectively. For the set of categories participating in category-level CORAL alignment, and The first and second digits in the source and target domains, respectively. A set of pattern latent variables for modal samples of speech electromyography data.

[0045] ; ; ; in, For the third stage modal alignment loss, The third stage global CORAL alignment loss, For the third stage, class-level CORAL alignment loss, These are the weighting coefficients for the third-stage category-level CORAL alignment loss. Latent variables of the model Feature dimensions, The function for calculating the covariance matrix. The third stage of electromyography samples of spoken language were processed by a pattern encoder. The set of pattern latent variables obtained after encoding and normalization This is the set of pattern latent variables for all silent speech electromyography samples that can be used for global modality alignment in the third stage. Let be the set of latent variables for the electromyographic sample patterns of spoken speech corresponding to the c-th speech category. Let c be the set of latent variables representing the labeled silent speech electromyography (EMG) sample patterns corresponding to the c-th speech category in the target domain. It is a set of speech categories that exist simultaneously in both spoken samples and labeled silent samples in the target domain.

[0046] In this embodiment, the calculation function expression for the cross-entropy classification loss is: ; ; ; , ; ; ; ; , ; in, The cross-entropy classification loss is used for labeled samples in the source domain of spoken speech electromyography data modal. These are the weighting coefficients. The cross-entropy classification loss is used for a small number of labeled samples in the target domain under the spoken speech electromyography data modality. and These are the batch sizes for the source and destination domains, respectively. For the first in the source domain A sample in a given Predict speech category under the condition The probability, For the first in the source domain Voice category labels for a single audio-visual electromyography (EMG) sample. For the first in the source domain Pattern latent variables of electromyography samples of spoken speech. For the first in the target domain A given set of labeled spoken electromyographic samples Predict speech category under the condition The probability, For the first in the target domain Voice category labels for labeled electromyographic samples of spoken voice. For the target domain, there are pattern latent variables for electromyographic samples of labeled spoken language. For normalization operations, For the first in the source domain There are 3 samples of spoken speech electromyography (EMG) data, where s represents the source domain and v represents the spoken speech EMG data modality. For the first in the target domain There are 3 labeled electromyographic (EMG) data samples of spoken language, where t represents the target domain and lab represents the labeled sample. and These represent the mode encoders respectively. and individual encoders , These are the weighting coefficients. The cross-entropy classification loss is used for the spoken speech EMG data samples in the third stage to maintain the discriminative ability of the pattern branch for spoken speech categories. The cross-entropy classification loss for labeled silent speech EMG data samples in the target domain in the third stage is used to semantically correct the pattern branch using a small number of silent speech calibration samples. The batch size for the electromyography (EMG) data samples in the third phase. The batch size for labeled silent speech electromyography data samples in the target domain during the third stage. For the first in the target domain A sample in a given Predict speech category under the condition The probability, For the first in the target domain Voice category labels for a single audio-visual electromyography (EMG) sample. For the first in the source domain Pattern latent variables of electromyography samples of spoken speech. For the i-th labeled silent speech electromyography sample in the target domain, given its pattern latent variables Under the given conditions, it is predicted by the speech classifier as the speech category label. The probability of; The speech category label is the i-th labeled silent speech electromyography sample in the target domain; For the i-th labeled silent speech electromyography sample in the target domain, the pattern encoder The pattern latent variables are obtained after encoding and normalization.

[0047] In this embodiment, the expression for the calculation function of the supervised representation constraint loss is: ; ; ; ; in, and These are the weighting coefficients. To monitor and compare learning loss, For unlabeled samples Figure 1 Sexual damage, Let i be the set of positive samples for the i-th sample. The set of positive samples of the i-th sample is the same as that of the j-th sample. The set of sample indices for each sample i belonging to the same speech category. For similarity function, For the first Pattern latent variables for each sample For pattern latent variables, For the first The pattern latent variables of the nth sample, the th Each sample is a negative sample in contrastive learning. For temperature coefficient, For unlabeled samples, For the target domain Unlabeled type The first modality of spoken electromyography data One sample, , and This is a random enhancement function for the samples, used to add random noise to the samples; For the first in the target domain One unlabeled sample of electromyographic data with spoken words, including This indicates an unlabeled sample.

[0048] In this embodiment, the expression for calculating the prototype anchoring loss is: ; ; ; in, The total number of voice categories. This serves as the category prototype for the c-th speech category in the target domain spoken speech electromyography data modality obtained in the second stage of learning. The category prototype for the c-th speech category in the source domain spoken speech electromyography data modality is derived from the pattern branch prototype obtained in the first stage of pre-training. The set of categories appearing in labeled silent samples in the target domain. The pattern latent variable centers are defined for the c-th class of labeled silent speech electromyography samples in the target domain. The target domain is defined as the number of labeled silent speech electromyography samples belonging to class c. This indicates that the summation is only performed on the speech category label. The target domain contains labeled silent samples. For the first in the target domain One labeled silent speech electromyography sample.

[0049] In this embodiment, after obtaining the electromyographic data speech recognition model by employing a three-stage semi-supervised decoupled electromyographic speech recognition training, the method further includes using the electromyographic data speech recognition model to perform electromyographic data speech recognition in the target domain. S1, acquire multi-channel surface electromyography signals at the vocal muscle groups of an individual in the target domain when uttering silent speech; S2, preprocesses and reduces noise from multi-channel surface electromyography signals; S3 performs abnormal channel correction on the pre-processed and noise-reduced multi-channel surface electromyography signals; S4. Feature extraction is performed on the multi-channel surface electromyography (EMG) signal after abnormal channel correction. This includes using a sliding window strategy to segment the multi-channel EMG signal. For each segment, the root mean square (RMS) and waveform length are extracted from that segment. The features of the extracted segments are used to construct a feature tensor based on the electrode spatial layout of each channel as the obtained target domain silent speech EMG data. The target domain silent speech EMG data is then input into the EMG data speech recognition model to obtain the syllables corresponding to the silent speech of an individual in the target domain.

[0050] To verify the source of differences in the original multichannel surface electromyography signals, this embodiment analyzes the root mean square (RMS) activation maps of representative speech categories. Figure 5 To illustrate the data differences between the two modalities of electromyography (EMG) data—voiced speech and silent speech—in this embodiment, the following methods are used: Figure 5 (a)~ Figure 5 (c) It can be seen that the two modalities of electromyography data with and without speech have similar main activation regions on the electrode grid, but their spatial distribution and local activation intensity are different. Figure 6 This is the root mean square plot of different individuals in this embodiment. Figure 6 (a)~ Figure 6 (l) Further, it is shown that under the same speech category and the same modality, there are obvious individual differences in the location and shape of the high activation region in the root mean square activation maps of individuals S1 to S12. Figure 7 This is a violin diagram showing the distance between the centroids under the same and different conditions in an embodiment of the present invention. Figure 7 (a) and Figure 7 (b) Centroid distance analysis showed that the centroid distance between subjects was greater than the intra-subject variability, and the centroid distance between modalities was greater than the intra-modal variability. Figure 8 In this embodiment, the centroid coordinates are divided according to the mode, and the sound... Distance between the centroids in a silent state Figure 9 In this embodiment, the centroid coordinates are divided by individual units, and the sound... Distance of the center of mass in a silent state Figure 8 (a)~ Figure 8 (c) and Figure 9 (a)~ Figure 9 Further statistical results in (c) showed that the cross-modal centroid distance for all speech tags was significantly greater than the intramodal reference distance, and the inter-subject centroid distance for all subjects was also significantly greater than the intra-subject variability. These results indicate that both modal and individual differences exist in the original multichannel surface electromyography signals.

[0051] To verify whether the multi-stage transfer process of the semi-supervised decoupled multi-stage electromyography speech recognition method in this embodiment achieves the expected goal, the nearest prototype diagnostic confusion matrix is ​​used for analysis. Figure 10 In this embodiment, the nearest prototype probe in the first stage latent space and the nearest prototype confusion matrix before and after the second and third stages are shown below. Figure 10 As shown in (a), the model latent variable z in the first stage p A clear category diagonal structure is formed, and the individual latent variable z sA clear diagonal structure of the subject was formed, successfully separating speech pattern-related information from subject-related information. Figure 10 (b)~ Figure 10 (e) shows that before the second-stage adaptation, there was significant class confusion between the target subject and the VS (voiced speech electromyography data) samples; after the second-stage adaptation, the diagonal structure was enhanced, indicating that the target subject's voice prototype assignment was improved. Figure 10 (f)~ Figure 10 (h) shows that in the third stage, the target SS (silent speech electromyography data) samples also suffered from severe confusion before adaptation; after modal adaptation from spoken to silent, most categories formed a clear diagonal structure. The difference matrix further shows that both the second and third stages mainly improved the allocation ratio of the correct categories and reduced some confusion of incorrect prototypes.

[0052] Further prototype-based diagnostic indicators were used to quantify the effects at each stage, and the results are shown in Table 1.

[0053] Table 1: Prototype-based staged transfer diagnostic evidence

[0054] As shown in Table 1, in the second stage, the class conditional gap ratio decreased from 2.132 to 0.617, the prototype margin increased from -0.483 to 0.277, and the nearest prototype accuracy improved from 10.8% to 68.1%. In the third stage, the class conditional gap ratio decreased from 1.453 to 1.171, the prototype margin increased from -0.482 to 0.823, and the nearest prototype accuracy improved from 18.8% to 84.6%. These changes are consistent with the confusion matrix results, indicating that the second stage effectively reduced the difference between the source and target subjects, and the third stage effectively improved the modal difference between the target subject's VS and SS.

[0055] Further visualization and clustering evaluation of the initial feature space and the final pattern embedding space yielded the following results: Figure 11 and Figure 12 As shown. Figure 11 (a)~ Figure 11 (b) shows that before the migration, the samples of different speech labels were obviously mixed, and the source domain VS and the target domain SS did not form a stable category structure; after the complete three-stage processing, the samples of the same speech category formed a more compact cluster. Figure 12 (a)~ Figure 12 (b) shows that the category contour coefficient increases significantly after the transfer, while the domain contour coefficient decreases significantly, indicating that the final embedding space is mainly organized according to speech categories, while weakening the separation between the source and target domains.

[0056] To verify the effectiveness of the semi-supervised decoupled multi-stage electromyography speech recognition method in this embodiment, the method in this embodiment was compared with the source domain only, source domain + target domain calibration, source domain pre-training + target domain calibration, DANN, CDAN, MME, CE-SFDA, CoTMix, CDAC and CDDG methods in terms of the generalization performance of silent speech recognition without seeing the target speaker. Accuracy and macro-average F1 score were used as indicators. The source domain only method used only the source domain voiceprint. The results are shown in Table 2.

[0057]

[0058] As shown in Table 2, when comparing the performance of the three baseline methods of the semi-supervised decoupled multi-stage EMG speech recognition method in this embodiment—source domain only, source domain + target domain calibration, and source domain pre-training + target domain calibration—the accuracy was 15.24% and the macro-average F1 score was 10.58% when using training data and directly testing the target SS test samples. The source domain + target domain calibration method, by adding a small number of SS calibration samples from the target subjects during semi-supervised training, improved the accuracy and macro-average F1 score to 63.21% and 60.45%, respectively. The source domain pre-training + target domain calibration method further employed pre-training techniques, achieving an accuracy of 0.03% and a macro-average F1 score of 58.07%. These results indicate that directly migrating from source VS data to target SS data yields poor results, while using target SS calibration samples significantly improves the results (p-value < 0.05). Subsequently, the semi-supervised decoupled multi-stage EMG speech recognition method of this embodiment was compared with existing domain adaptation methods (DANN, CDAN, MME) and semi-supervised adaptation methods (CE-SFDA, CoTMix, CDAC, CDDG). The performance of all seven methods was superior to those using only source domain data. Among them, CDAN-SSDA achieved an accuracy of 66.39% and a macro-average F1 score of 63.90%, making it the most competitive result. CDDG also performed close to this level, with an accuracy of 65.89% and a macro-average F1 score of 63.60%. However, the performance of these methods still fell short of the EMG data speech recognition model (DMST) of this embodiment. TDMST achieved an accuracy of 83.11% and a macro-average F1 score of 81.92%, which is 16.72% higher than the best competing method, CDAN-SSDA, and 18.02% higher in macro-average F1 score. Compared to all baselines and state-of-the-art methods, the proposed method achieves significant improvements in both accuracy and macro-average F1 score (p < 0.001). These results demonstrate that the proposed method's electromyographic data-based speech recognition (DMST) model achieves significant improvements in cross-subject speech separation tasks.

[0059] To verify whether this improvement is consistent across the retained samples and speech labels, the electromyography data speech recognition model (DMST) of this embodiment was further compared with the baseline model CDAN. The results are as follows: Figure 13 and Figure 14 As shown. From Figure 13 (a)~ Figure 13 (b) and Figure 14 As can be seen, in the sample-by-sample pairwise comparison, each line connects the performance data of the same retained sample under CDAN-SSDA and DMST. Most samples showed improvements in both accuracy and Macro-F1 score, indicating that the advantage of DMST is not driven by a small number of samples. The performance of sample S6 decreased slightly, but its baseline performance under CDAN-SSDA was already high, suggesting that the benefits of DMST may vary depending on the difficulty of the samples themselves. Further analysis of category-level recall on the target SS test set revealed that DMST's recall was higher than CDAN's. SSDA metrics for all speech tags indicate that the performance improvement is not limited to a few categories. Overall, these topic-level and category-level comparisons confirm the robustness of the EMG data speech recognition model DMST of this embodiment under the LOS0 target-SS evaluation setting.

[0060] In addition, to verify the contribution of each of the three stages in the semi-supervised decoupled multi-stage electromyography speech recognition method of this embodiment, an ablation experiment was conducted in the same LOSO evaluation setting, and the results are shown in Table 3.

[0061] Table 3: Ablation Experiment Results

[0062] As shown in Table 3, using only the first stage results in an accuracy of only 12.75% and a macro-average F1 score of 7.87%. Using only stage 2 also yields poor performance, with an accuracy of 14.83% and a macro-average F1 score of 10.78%. Adding the second stage to the first stage slightly improves performance, achieving an accuracy of 16.88% and a macro-average F1 score of 12.62%, but the results remain close to random levels. These results indicate that decoupling or voice personalization techniques alone are insufficient for identifying the target speaker in a silent state. In contrast, the combination including the third stage achieves higher performance. The third stage alone achieves an accuracy of 70.85% and a macro-average F1 score of 69.06%. The combination of the second and third stages achieves an accuracy of 74.08% and a macro-average F1 score of 72.62%. Combining the first and third stages increases the accuracy to 79.97% and the macro-average F1 score to 78.72%, but this is still significantly inferior to the complete model. Overall, the complete three-stage framework of the method in this embodiment achieves optimal performance, which confirms that decoupling, target subject personalization, and modal alignment in the method of this embodiment can bring complementary advantages.

[0063] This embodiment also provides a semi-supervised decoupled multi-stage electromyographic speech recognition system, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the semi-supervised decoupled multi-stage electromyographic speech recognition method.

[0064] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0065] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A semi-supervised decoupled multi-stage electromyography-speech recognition method, characterized in that, This includes employing a three-stage semi-supervised, decoupled electromyography (EMG) speech recognition training method to obtain an EMG data speech recognition model: In the first stage, a dual-branch decoupling generator is pre-trained using data samples from two modalities: spoken speech electromyography (EMG) data and silent speech EMG data from the source domain. The dual-branch decoupling generator includes a pattern branch and an individual branch. The pattern branch is used to learn speech-related representations, while the individual branch is used to capture individual-related differences. In the second stage, the pattern branch in the dual-branch decoupling generator is retained, and the pattern branch is semi-supervised fine-tuned using data samples of the spoken speech electromyography data modal from both the source and target domains to achieve the adaptation of the spoken speech electromyography data modal from the source domain to the target domain. In the third stage, for the fine-tuned pattern branches, data samples from two modalities—voiced speech EMG data and silent speech EMG data—from the source domain, as well as data samples from silent speech EMG data in the target domain, are used to further semi-supervised fine-tun the pattern branches to achieve adaptation from the voiced speech EMG data modal to the silent speech EMG data modal in the target domain. This results in an EMG data speech recognition model composed of pattern branches, which can be used for speech recognition of silent speech EMG data in the target domain.

2. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 1, characterized in that, The pattern branches of the dual-branch decoupling generator are generated by the pattern encoder. and speech classifier Composition, the pattern encoder Used for processing input feature maps Encode and generate pattern latent variables The speech classifier Used for latent variables of the pattern Classify and identify the corresponding speech category prototype in the pattern space; The individual branches of the dual-branch decoupling generator are generated by individual encoders. and individual classifiers Composition, the individual encoder Used for processing input feature maps Encode to generate individual latent variables The individual classifier Used for individual latent variables The classification identifies the corresponding individual identity prototype in the individual subject space, where the input feature map The feature map is extracted from labeled acoustic or aphonic electromyography (EMG) data of individuals in the source domain.

3. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 2, characterized in that, The dual-branch decoupling generator also includes a shared decoder. and the gradient inversion layer GRL, the shared decoder Latent variables used to determine the pattern based on the input and individual latent variables Perform reconstruction to obtain reconstruction feature maps The gradient inversion layer (GRL) is used to invert the gradient direction of the auxiliary classifier during backpropagation to suppress pattern latent variables. Individual identity information and individual latent variables The speech category information in the data; when pre-training the dual-branch decoupling generator using data samples from two modalities—voiced speech EMG data and silent speech EMG data—in the source domain, the loss function used is the first-stage loss, and the functional expression of the first-stage loss is: ； ； ； ； in, For the losses in the first phase, and These are the weighting coefficients. To constrain the input feature map and its reconstructed feature map Reconstruction loss due to differences between them To reduce model latent variables and individual latent variables The decoupling loss of information coupling between them The alignment loss is the corresponding speech category prototype in the pattern space and the corresponding individual identity prototype in the individual subject space. and These are the weighting coefficients. The self-reconstruction loss is used to constrain the shared decoder. Based on the pattern latent variables of the same sample and individual latent variables Reconstruct the original input electromyographic feature map to ensure that the latent variables retain effective information of the input signal; The cross-variable cycle consistency loss is used to constrain the pattern latent variables and the permuted individual latent variables through a shared decoder. After reconstruction, the pattern latent variables and individual latent variables obtained by re-encoding are consistent with the original pattern latent variables and the replaced individual latent variables, respectively. For balance coefficient, This is an adversarial classification loss implemented through a gradient inversion layer (GRL). For use as latent variables in constrained patterns and individual latent variables Orthogonal constraint loss of geometric correlation The loss for class discrimination of the speech category prototype is given by the pattern branch. The individual branch represents the individual discrimination loss of the individual identity prototype.

4. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 3, characterized in that, Reconstruction losses The expression for the computation function is: ； ； ， ， ； Adversarial classification loss The expression for the computation function is: ； Orthogonal constraint loss The expression for the computation function is: ； Category-based loss The expression for the computation function is: ； Individual discrimination loss The expression for the computation function is: ； in, For batch size, and The first Reconstructed feature map and input feature map of each sample. Indicates shared decoder , and These represent the mode encoders respectively. and individual encoders , and For feature map Corresponding pattern latent variables and individual latent variables Feature map for and Both are achieved through a shared decoder The feature map obtained by decoding, For the first random permutation within a batch Individual latent variables of a sample, among which This indicates that the index of the sample within the batch has been randomly permuted and compared with the first sample. Index of sample pairings For individual classifiers When the input is a pattern latent variable After Gradient Reversal Layer (GRL), the prediction is performed for the first... Each sample belongs to the individual identity prototype label. The probability, and The first Pattern latent variables for each sample and individual latent variables , For the first Prototype labels for the speech categories of each sample. For individual classifiers, For speech classifiers, for transpose, Pattern branching based on given latent variables Predicting speech category prototype labels The probability, These are the regularization weight coefficients. For the first branch in the pattern Pattern latent variables for each sample To its voice category prototype label The Mahalanobis distance corresponding to the Gaussian distribution, Prototype labels for the speech category in the pattern branch The precision matrix, wherein the precision matrix is ​​the inverse of the covariance matrix; Given individual latent variables in individual branches Time prediction source domain first Each sample belongs to the individual identity prototype label. The probability, These are the regularization weight coefficients. For the source domain in the individual branch Individual latent variables of a sample To its individual identity prototype label The Mahalanobis distance corresponding to the Gaussian distribution, Individual identity prototype label in individual branches The accuracy matrix.

5. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 3 or 4, characterized in that, The mode encoder and individual encoders Both have the same structure, including multiple cascaded encoding convolutional blocks, flattening layers, and latent variable layers. Each encoding convolutional block consists of sequentially connected 2D convolutional layers (Conv2d), batch normalization (BN) layers, and LReLU activation functions. The output of the last convolutional block is flattened by the flattening layer and then input to the latent variable layer. This latent variable layer is a linear layer used to map the input features to pattern latent variables of a specified dimension. or individual latent variables The shared decoder It includes an input layer, multiple cascaded decoding convolutional blocks, and an output layer, where the input layer is used to process the pattern encoder through linear layers. and individual encoders The concatenated features are linearized and output to the first decoding convolutional block. This decoding convolutional block consists of a sequentially connected upsampling module, a 2D convolutional layer (Conv2d), a batch normalization (BN) layer, and an LReLU activation function. The output layer consists of a 2D convolutional layer (Conv2d) and a hyperbolic tangent layer (Tanh) to reconstruct the input feature map into a reconstructed feature map. .

6. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 5, characterized in that, The loss function used in the second stage is the second-stage loss, and its functional expression is as follows: ； in, For the losses in the second phase, , , and For the loss weight in the second stage, The second stage uses cross-entropy classification loss for speech category supervision based on labeled samples. The second stage uses supervised representation constraint loss to enhance the clustering of similar samples and the separation of dissimilar samples. This is the alignment loss for the statistical distribution of the prototype speech categories in the pattern space corresponding to the source and target domain electromyographic data of spoken speech in the second stage. This is the prototype anchoring loss used in the second stage to constrain the class prototype offset during the target domain adaptation process.

7. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 6, characterized in that, The loss function used in the third stage is the third-stage loss, and the functional expression of the third-stage loss is as follows: ； in, For the losses in the third phase, , , and For the loss weight in the third stage, The third stage uses cross-entropy classification loss for speech category supervision based on labeled samples. The third stage uses supervised representation constraint loss to enhance the clustering of similar samples and the separation of dissimilar samples. The modal alignment loss in the third stage is used to align the statistical distribution differences between spoken and silent speech EMG data in the pattern latent variable space. This is the prototype anchoring loss used in the third stage to constrain the class prototype offset during the target domain adaptation process.

8. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 7, characterized in that, The function expression for calculating the alignment loss of the source and target domain spoken electromyographic data in the pattern space corresponding to the prototype statistical distribution of speech categories is as follows: ； ； ； The global CORAL alignment loss for spoken speech electromyography data modality is used to align the covariance of the overall representation distributions of the source and target domains. This is a category-level CORAL alignment loss for spoken speech electromyography data modalities, used to class-wise align the covariance of the source and target domain representations. These are the weighting coefficients for the second stage. Latent variables of the model and individual latent variables Feature dimensions, To find the covariance matrix, and These are the sets of pattern latent variables for modal samples of spoken speech electromyography data from the source and target domains, respectively. For the set of categories participating in category-level CORAL alignment, and The first and second digits in the source and target domains, respectively. The set of pattern latent variables for modal samples of electromyography data in the form of spoken speech; The expression for the modal alignment loss used in the third stage to align the statistical distribution differences between spoken and silent speech EMG data in the pattern latent variable space is as follows: ； ； ； in, For the third stage modal alignment loss, The third stage global CORAL alignment loss, For the third stage, class-level CORAL alignment loss, These are the weighting coefficients for the third-stage category-level CORAL alignment loss. Latent variables of the model Feature dimensions, The function for calculating the covariance matrix. The third stage of electromyography samples of spoken language were processed by a pattern encoder. The set of pattern latent variables obtained after encoding and normalization This is the set of pattern latent variables for all silent speech electromyography samples that can be used for global modality alignment in the third stage. Let be the set of latent variables for the electromyographic sample patterns of spoken speech corresponding to the c-th speech category. Let c be the set of latent variables representing the labeled silent speech electromyography (EMG) sample patterns corresponding to the c-th speech category in the target domain. It is a set of speech categories that exist simultaneously in both spoken samples and labeled silent samples in the target domain; The calculation function expression for the cross-entropy classification loss is as follows: ； ； ； ， ； ； ； ； ， ； in, The cross-entropy classification loss is used for labeled samples in the source domain of spoken speech electromyography data modal. These are the weighting coefficients. The cross-entropy classification loss is used for a small number of labeled samples in the target domain under the spoken speech electromyography data modality. and These are the batch sizes for the source and destination domains, respectively. For the first in the source domain A sample in a given Predict speech category under the condition The probability, For the first in the source domain Voice category labels for a single audio-visual electromyography (EMG) sample. For the first in the source domain Pattern latent variables of electromyography samples of spoken speech. For the first in the target domain A given set of labeled spoken electromyographic samples Predict speech category under the condition The probability, For the first in the target domain Voice category labels for labeled electromyographic samples of spoken voice. For the target domain, there are pattern latent variables for electromyographic samples of labeled spoken language. For normalization operations, For the first in the source domain There are 3 samples of spoken speech electromyography (EMG) data, where s represents the source domain and v represents the spoken speech EMG data modality. For the first in the target domain There are 3 labeled electromyographic (EMG) data samples of spoken language, where t represents the target domain and lab represents the labeled sample. and These represent the mode encoders respectively. and individual encoders , These are the weighting coefficients. The cross-entropy classification loss is used for the spoken speech EMG data samples in the third stage to maintain the discriminative ability of the pattern branch for spoken speech categories. The cross-entropy classification loss for labeled silent speech EMG data samples in the target domain in the third stage is used to semantically correct the pattern branch using a small number of silent speech calibration samples. The batch size for the electromyography (EMG) data samples in the third phase. The batch size for labeled silent speech electromyography data samples in the target domain during the third stage. For the first in the target domain A sample in a given Predict speech category under the condition The probability, For the first in the target domain Voice category labels for a single audio-visual electromyography (EMG) sample. For the first in the source domain Pattern latent variables of electromyography samples of spoken speech. For the i-th labeled silent speech electromyography sample in the target domain, given its pattern latent variables Under the given conditions, it is predicted by the speech classifier as the speech category label. The probability of; The speech category label is the i-th labeled silent speech electromyography sample in the target domain; For the i-th labeled silent speech electromyography sample in the target domain, the pattern encoder The pattern latent variables obtained after encoding and normalization; The expression for the calculation function of the supervised representation constraint loss is: ； ； ； ； in, and These are the weighting coefficients. To monitor and compare learning loss, For view consistency loss of unlabeled samples, Let i be the set of positive samples for the i-th sample. The set of positive samples of the i-th sample is the same as that of the j-th sample. The set of sample indices for each sample i belonging to the same speech category. For similarity function, For the first Pattern latent variables for each sample For pattern latent variables, For the first The pattern latent variables of the nth sample, the th Each sample is a negative sample in contrastive learning. For temperature coefficient, For unlabeled samples, For the target domain Unlabeled type The first modality of spoken electromyography data One sample, , and This is a random enhancement function for the samples, used to add random noise to the samples; For the first in the target domain One unlabeled sample of electromyographic data with spoken words, including This indicates unlabeled samples; The expression for the calculation function of the prototype anchoring loss is: ； ； ； in, The total number of voice categories. This serves as the category prototype for the c-th speech category in the target domain spoken speech electromyography data modality obtained in the second stage of learning. The category prototype for the c-th speech category in the source domain spoken speech electromyography data modality is derived from the pattern branch prototype obtained in the first stage of pre-training. The set of categories appearing in labeled silent samples in the target domain. The pattern latent variable centers are defined for the c-th class of labeled silent speech electromyography samples in the target domain. The target domain is defined as the number of labeled silent speech electromyography samples belonging to class c. This indicates that the summation is only performed on the speech category label. The target domain contains labeled silent samples. For the first in the target domain One labeled silent speech electromyography sample.

9. The semi-supervised decoupled multi-stage electromyography-speech recognition method according to claim 1, characterized in that, After obtaining the electromyographic data speech recognition model by employing a three-stage semi-supervised decoupled electromyographic speech recognition training method, the method further includes using the electromyographic data speech recognition model to perform electromyographic data speech recognition in the target domain. S1, acquire multi-channel surface electromyography signals at the vocal muscle groups of an individual in the target domain when uttering silent speech; S2, preprocesses and reduces noise from multi-channel surface electromyography signals; S3 performs abnormal channel correction on the pre-processed and noise-reduced multi-channel surface electromyography signals; S4. Feature extraction is performed on the multi-channel surface electromyography (EMG) signal after abnormal channel correction. This includes using a sliding window strategy to segment the multi-channel EMG signal. For each segment, the root mean square (RMS) and waveform length are extracted from that segment. The features of the extracted segments are used to construct a feature tensor based on the electrode spatial layout of each channel as the obtained target domain silent speech EMG data. The target domain silent speech EMG data is then input into the EMG data speech recognition model to obtain the syllables corresponding to the silent speech of an individual in the target domain.

10. A semi-supervised, decoupled, multi-stage electromyography-based speech recognition system, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the semi-supervised decoupled multi-stage electromyography speech recognition method according to any one of claims 1 to 9.

Images