A multi-modal adaptive fusion speech decoding system and method

Abstract

This invention discloses a multimodal adaptive fusion speech decoding system and method, belonging to the field of speech robust decoding technology. The system includes a feature extraction layer and a multimodal fusion network. The feature extraction layer includes a speech feature extraction module and an electromyography (EMG) feature extraction module. The multimodal fusion network includes a speech modality expert network, an EMG modality expert network, a multimodal interaction expert network, and a gating network. The speech modality expert network and the EMG modality expert network retain modality-specific discrimination information. The multimodal interaction expert network explicitly models intermodal dependencies through a cross-attention mechanism, capturing cross-modal complementary information. The adaptive gating network dynamically allocates weights to each expert based on the input signal quality, performing weighted fusion of expert outputs. This invention can balance decoding accuracy under high signal-to-noise ratio (SNR) conditions and decoding robustness under low SNR conditions in scenarios with dynamically fluctuating modality quality.

Description

Technical Field

[0001] This invention belongs to the field of speech robust decoding technology, and more specifically, relates to a multimodal adaptive fusion speech decoding system and method. Background Technology

[0002] Speech disorders typically refer to impairments in speech production, fluency of expression, or vocalization. Mild speech disorders are characterized by significant deficiencies in speech clarity and articulation coordination, but without a complete loss of vocal ability. Studies show that although mild speech disorders do not lead to complete aphasia, they can easily cause decreased communication efficiency due to unclear pronunciation, significantly impacting patients' psychological and social participation abilities. With the increasing aging of society, the communication problems of patients with mild speech disorders, primarily the elderly, are receiving growing attention. Therefore, researching effective decoding algorithms to assist these patients in daily communication is of great significance for improving their quality of life.

[0003] Currently, the mainstream speech decoding methods are mainly divided into two categories: one is Automatic Speech Recognition (ASR) based on acoustic signals, and the other is Silent Speech Recognition (SSR) based on non-acoustic signals.

[0004] Automatic speech recognition (ASR) based on acoustic signals has become an important means of human-computer interaction. Its core objective is to process speech signals from a microphone and convert them into text or commands, enabling machines to understand and respond to spoken input. Modern ASR is widely used in everyday scenarios, such as voice assistants recognizing user voice requests to complete information searches, and smart homes recognizing user voice commands to control home appliances.

[0005] The primary goal of silent speech recognition (SSR) is to decode speech content in the absence of audible speech output. Its core lies in extracting effective information from non-acoustic physiological signals related to speech production, such as electroencephalography (EEG) and electromyography (EMG), rather than relying on traditional speech acoustic signals. Currently, commonly used physiological signals in SSR research include EMG and EEG signals. Unlike EEG signals that reflect brain neural activity, EMG directly records the electrical activity of muscles related to articulation, allowing for a closer association with articulation movements of the face, jaw, and larynx, resulting in higher discriminability and practicality in silent speech recognition tasks. Therefore, EMG can be used to decode continuous phrases and achieve high recognition accuracy in classification tasks. For patients with mild speech disorders who still possess muscle activity, EMG is a convenient and reliable decoding signal.

[0006] Although ASR is widely used, its recognition accuracy drops significantly in noisy environments or when acoustic signal acquisition is limited. While SSR has a significant advantage in noise sensitivity, its overall decoding accuracy is relatively low and it is susceptible to individual physiological differences and electrode position shifts. For individuals with mild speech disorders, when the signal-to-noise ratio (SNR) is low, relying solely on acoustic signals can lead to a significant decrease in decoding robustness due to unclear pronunciation or background noise interference; while in high SNR scenarios, relying solely on non-acoustic signals will result in the loss of effective acoustic information.

[0007] This shows that existing speech decoding technologies have the technical problem of being unable to balance decoding accuracy under high signal-to-noise ratio conditions and decoding robustness under low signal-to-noise ratio conditions. Summary of the Invention

[0008] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a multimodal adaptive fusion speech decoding system and method, thereby solving the technical problem that existing speech decoding technologies struggle to balance decoding accuracy under high signal-to-noise ratio conditions with decoding robustness under low signal-to-noise ratio conditions.

[0009] To achieve the above objectives, according to a first aspect of the present invention, a multimodal adaptive fusion speech decoding system is provided, comprising: a feature extraction layer and a multimodal fusion network; The feature extraction layer includes a speech feature extraction module and an electromyography (EMG) feature extraction module, which are used to extract speech features and EMG features from speech signals and EMG signals, respectively. The multimodal fusion network includes a speech modality expert network, an electromyography modality expert network, a multimodal interaction expert network, and a gating network; The speech modality expert network learns speech-related discriminative information from the joint features obtained by splicing speech features and electromyography features, performs category prediction, and obtains the first prediction result. The electromyography modal expert network learns discriminative information related to electromyography from the joint features obtained by splicing speech features and electromyography features, performs category prediction, and obtains a second prediction result. The multimodal interaction expert network explicitly models speech features and electromyographic features through bidirectional cross-attention to obtain cross-modal interaction representations, which are used for category prediction to obtain a third prediction result. The gating network takes the joint features of concatenated speech features and electromyography features as input and outputs the gating weights of the speech monomodal expert network, the electromyography monomodal expert network, and the multimodal interaction expert network. These weights are used to adaptively weight and fuse the first, second, and third prediction results to obtain the speech category prediction result.

[0010] Furthermore, the gating network assigns the largest gating weight to the multimodal interaction expert network. When the signal-to-noise ratio of the joint features is within a preset range of high signal-to-noise ratio, the gating weight of the speech single-modal expert network is controlled to be greater than that of the electromyography single-modal expert network. When the signal-to-noise ratio of the joint features decreases, the gating weight of the speech single-modal expert network is reduced, and the gating weight of the multimodal interaction expert network is increased.

[0011] Furthermore, the multimodal interaction expert network first aligns speech features and electromyography (EMG) features to the same embedding dimension using a learnable projection matrix. Then, it calculates bidirectional cross-attention weights between speech features and EMG features, as well as between EMG features and speech features. Based on these bidirectional attention weights, it weights and fuses speech features and EMG features to obtain a cross-modal interaction representation. Finally, the cross-modal interaction representation is mapped to the output dimension through a linear layer to obtain a third prediction result.

[0012] Furthermore, the speech modality expert network uses a multilayer perceptron structure to adaptively mine and extract discriminative information related to speech modality from the joint features obtained by splicing speech features and electromyography features.

[0013] Furthermore, the electromyography modality expert network uses a multilayer perceptron structure to adaptively mine and extract discriminative information related to electromyography modality from the joint features obtained by splicing speech features and electromyography features.

[0014] According to a second aspect of the present invention, a training method for a multimodal adaptive fusion speech decoding system is provided, the training method employing a two-stage training strategy: In the first stage, the speech feature extraction module and the electromyography (EMG) feature extraction module are trained independently. The outputs of the speech feature extraction module and the EMG feature extraction module are connected to the speech classification head and the EMG classification head, respectively, to obtain speech signals and EMG signals labeled with the real speech categories. The speech signals labeled with the real speech categories are then input into the speech feature extraction module and the speech classification head in sequence to output the predicted speech category. The error between the predicted speech category and the real speech category is used as the loss function, and backpropagation is performed to update the parameters of the speech feature extraction module and the speech classification head. Similarly, the EMG signals labeled with the real speech categories are input into the EMG feature extraction module and the EMG classification head in sequence to output the predicted speech category. The error between the predicted speech category and the real speech category is then used as the loss function, and backpropagation is performed to update the parameters of the EMG feature extraction module and the EMG classification head. In the second stage, the parameters of the speech feature extraction module and the electromyography (EMG) feature extraction module obtained in the first stage of training are fixed, and only the multimodal fusion network is trained. The speech feature extraction module and the EMG feature extraction module obtained in the first stage of training are used to extract speech features and EMG features from the speech signal and EMG signal, respectively. The speech features and EMG features are input into the multimodal fusion network. The error between the speech category prediction result output by the multimodal fusion network and the true speech category is used as the loss function. The parameters of the multimodal fusion network are updated by backpropagation to obtain the trained speech decoding system.

[0015] Furthermore, the calculation of the loss function in the second stage includes: Calculate the cross-entropy loss between the speech category prediction result and the actual speech category, and calculate the load balancing loss between the speech category prediction result and the actual speech category. Use the weighted sum of the cross-entropy loss and the load balancing loss as the loss function.

[0016] According to a third aspect of the present invention, a speech decoding method with multimodal adaptive fusion is provided, comprising: The speech decoding system is trained by inputting the speech signal and electromyographic signal to be decoded into a multimodal adaptive fusion speech decoding system or a training method for a multimodal adaptive fusion speech decoding system, and outputs the speech category.

[0017] According to a fourth aspect of the present invention, an electronic device is provided, including a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used to implement a training method for a multimodal adaptive fusion speech decoding system or a multimodal adaptive fusion speech decoding method.

[0018] According to a fifth aspect of the present invention, a computer product is provided, which, when running, enables the computer to execute a training method for a multimodal adaptive fusion speech decoding system or a multimodal adaptive fusion speech decoding method.

[0019] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: (1) This invention combines a single-modal expert network and a multimodal interactive expert network to fully preserve the independent discrimination information of each modality while explicitly mining the complementary information between speech and electromyographic signals using a bidirectional cross-attention mechanism. It also utilizes a gating network to adaptively weight and fuse the outputs of each expert, achieving adaptive modeling of dynamic changes in modal quality. Based on the "divide and conquer" principle, this invention divides the model into multiple "expert" networks using a multimodal fusion network and adaptively weights and fuses them through the routing mechanism of the gating network. This allows all expert modules to collaboratively process information from different modalities. The cross-attention mechanism enables features of one modality to focus on features of another, thereby capturing the complementarity between different modalities at the feature level. This invention's system fully learns speech and electromyographic features, avoiding the significant decrease in decoding robustness caused by unclear pronunciation or background noise interference when relying solely on acoustic signals, and preventing the loss of effective acoustic information in high signal-to-noise ratio scenarios. Experimental verification shows that the method provided by this invention maintains high decoding accuracy under high signal-to-noise ratio conditions while effectively improving decoding robustness under low signal-to-noise ratio conditions, demonstrating strong comprehensive performance.

[0020] (2) The gating network of this invention assigns the largest gating weight to the multimodal interaction expert network, indicating that the multimodal interaction expert always dominates under various noise conditions, while the single-modal expert mainly plays an auxiliary role. This is because the multimodal interaction expert network explicitly models the intermodal dependencies through a cross-attention mechanism, which can capture cross-modal complementary information and improve decoding accuracy. Under high signal-to-noise ratio (SNR) conditions, the weight of the speech single-modal expert is higher than that of the electromyography single-modal expert, and the model mainly relies on the interaction expert and the speech expert for decoding. As the SNR gradually decreases, the gating network reduces the weight of the speech expert and increases the weight allocation to the interaction expert. This is because under low SNR conditions, the model's decoding process relies more on the interaction expert and the electromyography expert, indicating that the gating-weighted fusion method can adaptively adjust the fusion strategy of multimodal information according to the noise level. Therefore, this invention can balance decoding accuracy under high SNR conditions and decoding robustness under low SNR conditions in scenarios with dynamic fluctuations in modal quality, effectively solving the problem that fixed fusion strategies are difficult to adaptively adjust in communication with mild speech disorders.

[0021] (3) Speech and electromyography are asynchronous in physical space (e.g., muscles contract first, then sound is emitted). This invention aligns the two to the same embedding dimension through a learnable projection matrix, effectively establishing a connection in the semantic space. Under high signal-to-noise ratio (SNR), bidirectional attention acts as a "high-resolution fusion agent," utilizing the complementary information of the two modalities to solve the "confusion" problem present in a single modality. Under low SNR, bidirectional attention acts as a "noise reduction filter" and an "information compensator." As long as one modality is clear, the interaction expert network can "recover" the lost information through cross-modal association. This balances decoding accuracy under high SNR conditions with decoding robustness under low SNR conditions.

[0022] (4) This invention decouples feature extraction and modality fusion processes through a two-stage training strategy, reducing cross-modal interference in the early stages of training and ensuring the feature quality of the input fusion module. The first stage trains two single-modal feature extraction modules, speech and electromyography, respectively. The goal of this stage is to obtain stable single-modal representations so that the single-modal features can contain sufficient discriminative information. The second stage fixes the parameters of the feature extraction modules obtained in the first stage and trains only the multimodal fusion network. The goal of the second stage is to enable the fusion network to learn the fusion relationship of different modal information based on reliable single-modal feature representations. The second stage only updates the parameters of the fusion network, without updating the parameters of the two single-modal feature extractors, to improve training efficiency and reduce training instability. A two-stage training strategy is adopted to achieve stable feature learning and efficient modality fusion.

[0023] (5) To avoid assigning too low weights to some experts during the training process, this invention introduces an additional load balancing constraint during the second stage of training, applying an entropy-based regularization term to the weight distribution of the gating network output. This constraint encourages a more balanced gating distribution and prevents the model from over-relying on a single expert. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the UC-MoE structure provided in an embodiment of the present invention; Figure 2 This is a heatmap of expert weight distribution in UC-MoE under different signal-to-noise ratio conditions provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the two-stage training strategy provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0026] A multimodal adaptive fusion speech decoding system includes: a feature extraction layer and a multimodal fusion network; The feature extraction layer includes a speech feature extraction module and an electromyography (EMG) feature extraction module, which are used to extract speech features and EMG features from speech signals and EMG signals, respectively. After normalizing the speech features and electromyographic features respectively, the joint features are obtained by concatenating them. The multimodal fusion network includes a speech modality expert network, an electromyography modality expert network, a multimodal interaction expert network, and a gating network; The speech modality expert network learns speech-related discriminative information from joint features, performs category prediction, and obtains a first prediction result; The electromyography modal expert network learns discriminative information related to electromyography from joint features, performs category prediction, and obtains a second prediction result. The multimodal interaction expert network explicitly models speech features and electromyographic features through bidirectional cross-attention to obtain cross-modal interaction representations, which are used for category prediction to obtain a third prediction result. The gating network takes joint features as input and outputs gating weights for the speech monomodal expert network, electromyography monomodal expert network, and multimodal interaction expert network. These weights are used to adaptively weight and fuse the first, second, and third prediction results to obtain the speech category prediction result.

[0027] The system training process employs a two-stage training strategy: In the first stage, the speech and electromyography single-modal feature extraction modules are trained independently. Each modal feature extraction module is followed by a corresponding classification head, and each module is supervised using the cross-entropy loss of its respective modality. In the second stage, the parameters of the feature extraction module obtained in the first stage are fixed, and only the multimodal fusion network is trained. The parameters of the fusion network are updated using the sum of cross-entropy loss and load balancing loss as the total loss function, resulting in a trained speech decoding system.

[0028] This invention combines a single-modal expert network and a multimodal interactive expert network to fully preserve the independent discriminative information of each modality while explicitly mining complementary information between speech and electromyographic signals using a bidirectional cross-attention mechanism. It employs a gating network to adaptively weight and fuse the outputs of each expert, achieving adaptive modeling of dynamic changes in modality quality. Furthermore, a two-stage training strategy decouples feature extraction from the modality fusion process, reducing cross-modal interference in the early stages of training and ensuring the feature quality of the fusion network input. Simultaneously, a load-balanced loss constraint is introduced to control the distribution of gating weights, preventing the model from over-relying on a single expert. The method provided by this invention maintains high decoding accuracy under high signal-to-noise ratio conditions while effectively improving decoding robustness under low signal-to-noise ratio conditions, demonstrating strong overall performance.

[0029] Example 1 like Figure 1 As shown, a multimodal adaptive fusion speech decoding system (UC-MoE) includes: a feature extraction layer and a multimodal fusion network; The feature extraction layer includes a speech feature extraction module and an electromyography (EMG) feature extraction module, which are used to extract speech features and EMG features from speech signals and EMG signals, respectively. After normalizing the speech features and electromyographic features respectively, the joint features are obtained by concatenating them. The multimodal fusion network includes a speech modality expert network, an electromyography modality expert network, a multimodal interaction expert network, and a gating network; The speech modality expert network learns speech-related discriminative information from joint features, performs category prediction, and obtains a first prediction result; The electromyography modal expert network learns discriminative information related to electromyography from joint features, performs category prediction, and obtains a second prediction result. The multimodal interaction expert network explicitly models speech features and electromyographic features through bidirectional cross-attention to obtain cross-modal interaction representations, which are used for category prediction to obtain a third prediction result. The gating network takes joint features as input and outputs gating weights for the speech monomodal expert network, electromyography monomodal expert network, and multimodal interaction expert network. These weights are used to adaptively weight and fuse the first, second, and third prediction results to obtain the speech category prediction result.

[0030] Next, we will introduce the structure of each module.

[0031] UC-MoE comprises two unimodal expert networks, one for speech and one for electromyography (EMG), both employing a multilayer perceptron (MLP) architecture. Each MLP consists of two linear layers and one ReLU activation unit, emphasizing the modeling of discriminative information relevant to speech decoding within its respective modality. For the input speech and EMG features, normalization is first performed separately. The normalized speech and EMG features are then concatenated to obtain a joint feature representation, which is then passed through the two unimodal expert networks to obtain the corresponding category prediction results.

[0032] In addition to unimodal experts, UC-MoE introduces a multimodal interaction expert network. This network explicitly models the dependency relationship between speech and electromyography (EMG) features through a bidirectional cross-attention mechanism. Compared to simple feature concatenation mapping, the interaction expert based on the cross-attention mechanism can explicitly model the dependency relationship between different modalities to fully capture interaction feature information. First, the features of the two modalities are aligned to the same embedding dimension using a learnable projection matrix. Then, the features are explicitly modeled through bidirectional cross-attention: the bidirectional cross-attention weights for speech to EMG and EMG to speech are calculated separately. The two modal features are then weighted and fused based on the bidirectional attention weights, and finally mapped to the output dimension through a linear layer to obtain the prediction result of the interaction expert.

[0033] UC-MoE employs a soft-gating mechanism to weight and fuse the outputs of different experts. This gating mechanism allows the model to adaptively adjust the contributions of different experts based on the overall distribution of the input features, thereby enhancing the model's robustness. Similar to unimodal expert networks, the gating network first normalizes the two input features separately and concatenates them to obtain a joint feature representation. This representation is then passed through a multilayer perceptron and a softmax function with a temperature coefficient to output the gating weights of the three experts, with the sum of all weights being 1. The final output of the model is the weighted result of the gating network on the expert network.

[0034] in, , , These are the gating weights for voice experts, interaction experts, and electromyography experts, respectively. , , These are the predictions from three experts.

[0035] like Figure 2As shown, the multimodal interaction expert consistently dominates under various noise conditions, while the single-modal expert mainly plays a supporting role. Under high signal-to-noise ratio (SNR) conditions, the speech expert has a higher weight than the electromyography (EMG) expert, and the model primarily relies on the interaction and speech experts for decoding. As the SNR gradually decreases, the weight allocation of the speech expert significantly decreases, and the gating network further increases the weight allocation of the interaction expert, while the weight change of the EMG expert is relatively small. Under low SNR conditions, the model's decoding process relies more heavily on the interaction and EMG experts, indicating that the gating-weighted fusion method can adaptively adjust the multimodal information fusion strategy according to the noise level.

[0036] Example 2 like Figure 3 As shown, a training method for a multimodal adaptive fusion speech decoding system includes: Phase 1: Single-modal representation learning. Two single-modal feature extraction modules, one for speech and one for electromyography, are trained separately. The goal of this phase is to obtain stable single-modal representations that contain sufficient discriminative information. This can be obtained from the following formula:

[0037] in, and These represent the speech and electromyography feature extraction functions, respectively. and These represent the extracted speech features and electromyography (EMG) features, respectively. The features are mapped to the output dimension via a classification head, and the parameters of the two feature extractors are updated using the supervision signals for their respective modalities.

[0038] Phase Two: Multimodal Fusion Network Learning. The parameters of the feature extraction modules obtained in Phase One are fixed, and only the multimodal fusion network is trained. The goal of this phase is to enable the fusion network to learn the fusion relationships of different modal information based on reliable single-modal feature representations. This can be obtained from the following formula:

[0039] in, This represents a multimodal fusion network. This represents the model's predicted output. In this stage, only the parameters of the fusion network are updated, without updating the parameters of the two unimodal feature extractors, to improve training efficiency and reduce training instability.

[0040] During the testing phase, the original electromyographic signals and speech signals at different signal-to-noise ratios are input into the trained model to obtain the corresponding output.

[0041] Considering that different modalities may interfere with each other in the early stage of training, leading to unstable training, this invention adopts a two-stage training strategy to ensure reliable single-modal feature quality. This strategy decouples feature extraction from modality fusion, thereby reducing cross-modal interference in the early stage of training, ensuring the feature quality of the input fusion module, and achieving stable feature learning and efficient modality fusion.

[0042] In the single-modal representation learning stage, the feature extraction modules for the speech modality and electromyography modality are trained separately. In this stage, the feature extraction module for each modality is followed by the corresponding classification head, and supervised learning is performed using standard cross-entropy loss.

[0043] In the multimodal fusion network learning phase, the parameters of the feature extraction module obtained in the first phase are fixed, and only the multimodal fusion network is trained. The basic supervision signal still uses cross-entropy loss. Simultaneously, to avoid the gating network assigning excessively low weights to some experts during training, a load balancing constraint is introduced in this phase, applying an entropy-based regularization term to the weight distribution of the gating network output. This constraint encourages a more balanced gating distribution and prevents the model from over-relying on a single expert. The total training loss function is defined as:

[0044] in, Represents cross-entropy loss, This indicates load balancing losses. The hyperparameter represents the balance between the two loss terms. The UC-MoE multimodal fusion network is trained by minimizing the above total loss function, enabling it to dynamically adjust the contributions of each expert according to changes in modal quality under different scenarios, thus achieving adaptive modeling of modal dynamic reliability.

[0045] Load balancing loss is defined as the average of the negative entropy of the gated weight distribution:

[0046] in, For the sample size, For the first The first sample The gating weights corresponding to each expert To prevent numerically unstable small constants.

[0047] Table 1

[0048] Experiments demonstrate that, on publicly available multimodal speech datasets, the method of this invention (UC-MoE in Table 1) achieves superior sentence classification accuracy under various signal-to-noise ratio (SNR) conditions compared to classic decision-level fusion methods (DLF-MAX), traditional machine learning fusion methods (Random Forest RF, Naive Bayes NBC, Decision Tree DT), and deep fusion model methods (DCCA, GRU, Transformer, MTREE-Net, I²MoE, MMR-DAF). Its overall recognition performance significantly outperforms various baseline methods. Specific comparison results are shown in Table 1. Table 1 shows the classification accuracy of different methods under different SNR conditions, with the highest result for each SNR highlighted in bold.

[0049] In summary, this invention proposes a multimodal fusion speech decoding system, UC-MoE, based on a hybrid expert model, to address speech decoding problems in communication scenarios involving mild speech disorders. By designing a single-modal expert network, speech and electromyographic (EMG) signals are modeled independently, fully preserving the advantages of individual decoding for each modality. Simultaneously, a multimodal interactive expert network based on a cross-attention mechanism is constructed to explicitly mine the interaction relationships and complementary features between speech and EMG signals. Furthermore, an adaptive gating network is introduced, which dynamically adjusts the weights of each expert's output according to changes in modal quality under different scenarios. Finally, UC-MoE achieves adaptive modeling of modal dynamic reliability through modular training and optimization.

[0050] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal adaptive fusion speech decoding system, characterized in that, include: Feature extraction layer and multimodal fusion network; The feature extraction layer includes a speech feature extraction module and an electromyography (EMG) feature extraction module, which are used to extract speech features and EMG features from speech signals and EMG signals, respectively. The multimodal fusion network includes a speech modality expert network, an electromyography modality expert network, a multimodal interaction expert network, and a gating network; The speech modality expert network learns speech-related discriminative information from the joint features obtained by splicing speech features and electromyography features, performs category prediction, and obtains the first prediction result. The electromyography modal expert network learns discriminative information related to electromyography from the joint features obtained by splicing speech features and electromyography features, performs category prediction, and obtains a second prediction result. The multimodal interaction expert network explicitly models speech features and electromyographic features through bidirectional cross-attention to obtain cross-modal interaction representations, which are used for category prediction to obtain a third prediction result. The gating network takes the joint features of concatenated speech features and electromyography features as input and outputs the gating weights of the speech monomodal expert network, the electromyography monomodal expert network, and the multimodal interaction expert network. These weights are used to adaptively weight and fuse the first, second, and third prediction results to obtain the speech category prediction result.

2. The speech decoding system with multimodal adaptive fusion as described in claim 1, characterized in that, The gating network assigns the largest gating weight to the multimodal interaction expert network. When the signal-to-noise ratio of the joint features is within a preset range of high signal-to-noise ratio, the gating weight of the speech single-modal expert network is controlled to be greater than that of the electromyography single-modal expert network. When the signal-to-noise ratio of the joint features decreases, the gating weight of the speech single-modal expert network is reduced and the gating weight of the multimodal interaction expert network is increased.

3. A multimodal adaptive fusion speech decoding system as described in claim 1 or 2, characterized in that, The multimodal interaction expert network first aligns speech features and electromyography (EMG) features to the same embedding dimension using a learnable projection matrix. Then, it calculates bidirectional cross-attention weights between speech features and EMG features, as well as between EMG features and speech features. Based on these bidirectional attention weights, it weights and fuses speech features and EMG features to obtain a cross-modal interaction representation. Finally, the cross-modal interaction representation is mapped to the output dimension through a linear layer to obtain the third prediction result.

4. A multimodal adaptive fusion speech decoding system as described in claim 1 or 2, characterized in that, The speech modality expert network uses a multilayer perceptron structure to adaptively mine and extract discriminative information related to speech modality from the joint features obtained by splicing speech features and electromyography features.

5. A multimodal adaptive fusion speech decoding system as described in claim 1 or 2, characterized in that, The electromyography modality expert network uses a multilayer perceptron structure to adaptively mine and extract discriminative information related to electromyography modality from the joint features obtained by splicing speech features and electromyography features.

6. A training method for a multimodal adaptive fusion speech decoding system as described in any one of claims 1-5, characterized in that, The training method employs a two-stage training strategy: In the first stage, the speech feature extraction module and the electromyography (EMG) feature extraction module are trained independently. The outputs of the speech feature extraction module and the EMG feature extraction module are connected to the speech classification head and the EMG classification head, respectively, to obtain speech signals and EMG signals labeled with the real speech categories. The speech signals labeled with the real speech categories are then input into the speech feature extraction module and the speech classification head in sequence to output the predicted speech category. The error between the predicted speech category and the real speech category is used as the loss function, and backpropagation is performed to update the parameters of the speech feature extraction module and the speech classification head. Similarly, the EMG signals labeled with the real speech categories are input into the EMG feature extraction module and the EMG classification head in sequence to output the predicted speech category. The error between the predicted speech category and the real speech category is then used as the loss function, and backpropagation is performed to update the parameters of the EMG feature extraction module and the EMG classification head. In the second stage, the parameters of the speech feature extraction module and the electromyography (EMG) feature extraction module obtained in the first stage of training are fixed, and only the multimodal fusion network is trained. The speech feature extraction module and the EMG feature extraction module obtained in the first stage of training are used to extract speech features and EMG features from the speech signal and EMG signal, respectively. The speech features and EMG features are input into the multimodal fusion network. The error between the speech category prediction result output by the multimodal fusion network and the true speech category is used as the loss function. The parameters of the multimodal fusion network are updated by backpropagation to obtain the trained speech decoding system.

7. The training method as described in claim 6, characterized in that, The calculation of the loss function in the second stage includes: Calculate the cross-entropy loss between the speech category prediction result and the actual speech category, and calculate the load balancing loss between the speech category prediction result and the actual speech category. Use the weighted sum of the cross-entropy loss and the load balancing loss as the loss function.

8. A speech decoding method with multimodal adaptive fusion, characterized in that, include: The speech signal and electromyographic signal to be decoded are input into a speech decoding system of multimodal adaptive fusion as described in any one of claims 1-5 or a speech decoding system trained by the training method described in claim 6 or 7, and the speech category is output.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used to implement a training method for a multimodal adaptive fusion speech decoding system as described in claim 6 or 7, or a multimodal adaptive fusion speech decoding method as described in claim 8.

10. A computer product, characterized in that, When the product is running, it enables the computer to execute the training method of the multimodal adaptive fusion speech decoding system as described in claim 6 or 7, or the multimodal adaptive fusion speech decoding method as described in claim 8.

Images