A multi-modal collaborative swallowing simulation electrical stimulation method and system of brain-computer interface

Abstract

The application discloses a kind of brain-computer interface multimodal collaborative swallowing simulation electric stimulation method and system, the method acquisition electroencephalogram signal, oral cavity signal and electromyogram signal three modal signals, through 1D-CNN+Transformer hybrid establishment time sequence movement algorithm model, fusion electroencephalogram signal, oral cavity signal and electromyogram signal feature, output laryngeal movement delay and muscle group activation space-time sequence, then, oral cavity biomechanics simulation model is combined with food physical property, generates and exports neuromuscular electrical stimulation NMES sequence output, improve the accuracy of swallowing treatment.

Description

Technical Field

[0001] This invention relates to the field of swallowing therapy technology, and in particular to a multimodal collaborative swallowing simulation electrical stimulation method and system using a brain-computer interface. Background Technology

[0002] Existing swallowing electrical stimulation techniques mostly identify swallowing intentions and execute swallowing actions by acquiring single-modal signals such as electroencephalogram (EEG) or electromyography (EMG). They do not integrate EEG and EMG signals for multimodal analysis, nor do they consider oral signals, and therefore lack effective optimization of swallowing therapy based on the physiological characteristics of swallowing. Furthermore, neuromuscular electrical stimulation (NMES) lacks the ability to dynamically adjust based on the patient's real-time physiological signals. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal collaborative swallowing simulation electrical stimulation method and system for brain-computer interfaces to solve one or more of the above-mentioned technical problems.

[0004] To achieve this objective, the present invention adopts the following technical solution: A multimodal collaborative swallowing simulation electrical stimulation method using a brain-computer interface includes: S1: Signal Acquisition Collect electroencephalogram (EEG) signals, oral cavity signals, and pharyngeal electromyography (EMG) signals; S2: Establishment of the temporal correlation model: A temporal correlation model is established between the collected EEG signals, oral signals, and pharyngeal electromyography signals using a 1D-CNN+Transformer architecture; this model promotes a nonlinear mapping relationship between the above three modal signals and the activity of the pharyngeal muscle groups; and generates and outputs the spatiotemporal sequences of pharyngeal movement delay and muscle activation. S3: Swallowing simulation: Establish an oropharyngeal biomechanical simulation model to simulate the food swallowing process based on food properties, pharyngeal movement delay, and spatiotemporal sequence of muscle activation, and generate and output quantifiable parameters of swallowing efficacy and temporal stimulation commands. S4: Output of the neuromuscular electrical stimulation (NMES) sequence: Based on the pharyngeal effect energy quantification parameters and timing stimulation instructions, a neuromuscular electrical stimulation (NMES) sequence is generated and output.

[0005] In some implementations, in step S1, the oral cavity signal includes a tongue pressure signal; Or the oral signals may include tongue pressure signals, tooth occlusal force signals, and oral temperature and humidity signals.

[0006] In some implementations, in step S1 or step S2, electroencephalogram (EEG) signals, oral cavity signals, and pharyngeal electromyography (EMG) signals are acquired synchronously to ensure that the three modal signals have a unified timestamp. By downsampling or interpolation, the acquired EEG signals, oral cavity signals, and pharyngeal electromyography signals are unified to the same temporal resolution; The timing alignment of acquired EEG, oral cavity, and pharyngeal electromyography signals can be dynamically achieved through the TAD-CAT algorithm, or by combining the TAD-CAT algorithm with the Transformer timing attention mechanism.

[0007] In some implementations, in step S2, 1D-CNN extracts local features of each modality signal, and 1D-CNN+Transformer establishes long-range temporal dependencies between each modality signal. It combines at least the transient rhythm of EEG signals, the burst onset of EMG signals, and the peak height of tongue pressure to learn the dependencies between different time steps and between different modality features, capture cross-modal delays, and then performs training.

[0008] In some implementations, during the training process in step S2, corresponding records are made based on the characteristics of the population, the performance of the model in different populations is observed, and domain adaptation or multi-task learning is performed. Among these, population characteristics include age, gender, and disease.

[0009] In some implementations, the time-series correlation model can calculate the correlation coefficient between the predicted delay and the actual delay; Specifically, this includes: calculating the Pearson correlation coefficient and root mean square error (RMSE) between the predicted waveform and the actual waveform; The temporal association model can use Dynamic Time Warped Distance (DTW) to evaluate shape similarity.

[0010] In some implementations, in step S3, the food properties include: elastic modulus, adhesive force, and rheological constitutive parameters; The quantitative parameters of swallowing efficiency include: pharyngeal contraction rate, food transport rate, and food residue.

[0011] In some implementations, in step S3, the maximum mean difference (MMD) distance between the real-time features and the initial model features is calculated to determine whether it is the first use or feature drift, wherein the threshold for the maximum mean difference (MMD) distance is 0.15. If it is the first time using the technology or if there is feature drift, a meta-learning architecture based on Prototypical Network is adopted, which uses individual effective data to fine-tune the parameters of the upper attention layer and classifier of Transformer. If the features do not drift, the previously stored individual parameters are directly called.

[0012] In some implementations, step SF1 is further included between steps S2 and S3; Step SF1: Verification of the validity of the swallowing intention: Based on the triggering of swallowing intention, the effectiveness of swallowing intention is verified by dual threshold rules and temporal constraints; Among them, the double threshold rule is: the amplitude of the oral signal is greater than or equal to the preset physiological threshold, and the probability of intention of the EEG signal is greater than or equal to 0.85; the preset physiological threshold is 1.5 times the individual baseline value; Timing constraint: EEG signals are valid within 200-500ms after oral signal triggering; If the dual threshold and timing constraints are met, it is determined to be a valid intent; otherwise, it is determined to be a false trigger or no intent, and the process returns to step S1: signal acquisition.

[0013] In some implementations, in step S4, while outputting the neuromuscular electrical stimulation (NMES) sequence, electromyographic signals of the swallowing muscle group after stimulation are simultaneously acquired.

[0014] In some implementations, the method further includes: step S5; S5: Dynamically optimize model parameters: By comparing the feedback electromyographic signals, the expected contraction intensity, and the deviation from the timing, the error value is calculated. If the error is ≤5%, save the current parameters; If the error is greater than 5%, adaptive updates are initiated, filtering data sets through a sliding window mechanism and adjusting model weights and stimulus parameters.

[0015] A brain-computer interface-based multimodal collaborative swallowing simulation electrical stimulation system includes: The signal acquisition module is used to acquire electroencephalogram (EEG) signals, oral cavity signals, and electromyogram (EMG) signals. The temporal correlation module is used to establish a temporal correlation model and generate and output the spatiotemporal sequences of pharyngeal movement delay and muscle activation. The swallowing intent verification module is used to verify the validity of swallowing intent; The swallowing simulation module is used to establish an oropharyngeal biomechanical simulation model. It simulates the food swallowing process based on the food properties, pharyngeal movement delay, and spatiotemporal sequence of muscle activation, and generates and outputs quantified parameters of swallowing efficacy and temporal stimulation commands. The electrical stimulation output module generates and outputs a neuromuscular electrical stimulation (NMES) sequence based on the pharyngeal effect energy quantification parameters and timing stimulation instructions. The feedback optimization module optimizes the model parameters based on the feedback electromyographic signals.

[0016] The beneficial effects of this invention are as follows: By integrating the features of three modal signals—EEG, oral cavity, and EMG—through a 1D-CNN+Transformer hybrid temporal motion algorithm model, the invention outputs a spatiotemporal sequence of pharyngeal movement delay and muscle activation. Then, the oropharyngeal biomechanical simulation model, combined with food properties, generates and outputs a neuromuscular electrical stimulation (NMES) sequence, thereby improving the accuracy of swallowing therapy. Attached Figure Description

[0017] Figure 1 This is one of the step diagrams of a multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interface according to the present invention; Figure 2 This is the second step diagram of a multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interface according to the present invention; Figure 3 This is a schematic diagram of a multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interface according to the present invention; Figure 4 This is a structural diagram of a multimodal collaborative swallowing simulation electrical stimulation system for brain-computer interface according to the present invention.

[0018] Noun Analysis: BCI: Brain-Computer Interface; EEG: Electroencephalography, or electroencephalogram; NMES: Neuromuscular Electrical Stimulation; sEMG: Surface Electromyography; TAD-CAT: Temporal Attention-based Dual-modal Conditional AlignmentTransformer, a model based on temporal attention for dual-modal conditional alignment; Transformer: A deep learning model based on attention mechanisms.

[0019] MMD: Maximum Mean Discrepancy; RMS: Root Mean Square; FOIS: Functional Oral Intake Scale.

[0020] Prototypical Network: A classic few-shot learning model proposed in 2017. Its core idea is to use "class prototypes" to represent categories and classify based on distance.

[0021] CSP space: Common Spatial Pattern, is a core technology for classifying multi-channel temporal signals such as EEG.

[0022] CNN; Convolutional Neural Network.

[0023] 1D-CNN: One-dimensional temporal signals in convolutional neural networks. It refers to a network that processes one-dimensional data structures, and these data typically have a sequential order (i.e., temporal dependencies). Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings.

[0025] refer to Figures 1 to 3 A multimodal collaborative swallowing simulation electrical stimulation method using a brain-computer interface, comprising: S1: Signal Acquisition Electroencephalogram (EEG) signals, oral cavity signals, and pharyngeal electromyography (EMG) signals were collected.

[0026] S2: Establishment of temporal correlation model: that is, to establish a temporal motion algorithm model; A temporal correlation model is established using a 1D-CNN+Transformer architecture to collect EEG signals, oral signals, and pharyngeal electromyography signals; this model promotes a nonlinear mapping relationship between the above three modal signals and the activity of the pharyngeal muscle groups; and generates and outputs the spatiotemporal sequence of pharyngeal movement delay and muscle activation.

[0027] S3: Swallowing simulation: An oropharyngeal biomechanical simulation model was established to simulate the food swallowing process based on food properties, pharyngeal movement delay, and spatiotemporal sequence of muscle activation. The model generates and outputs quantifiable parameters of swallowing efficacy and temporal stimulation commands. The temporal stimulation commands are used to control the frequency, pulse width, target, timing, and intensity of electrical stimulation.

[0028] S4: Output of the neuromuscular electrical stimulation (NMES) sequence: Based on the pharyngeal effect energy quantification parameters and timing stimulation instructions, a neuromuscular electrical stimulation (NMES) sequence is generated and output. The NMES sequence includes the frequency, pulse width, target location, timing, and intensity of the electrical stimulation.

[0029] Therefore, three modal signals—EEG, oral cavity, and pharyngeal electromyography—are collected. A nonlinear mapping relationship between the three modal signals is formed through a temporal motion algorithm, and the spatiotemporal sequence of pharyngeal movement delay and muscle activation is output. Then, the swallowing process is simulated through swallowing simulation to generate and output quantifiable parameters of swallowing efficacy and temporal stimulation instructions. Finally, a neuromuscular electrical stimulation (NMES) sequence is generated and output to electrically stimulate the pharyngeal muscles. This allows for deep fusion and analysis of multimodal signals, optimization for swallowing physiological characteristics, and the realization of fine temporal synergistic activation of the glossopharyngeal muscles, thereby improving the therapeutic effect of electrical stimulation.

[0030] In step S1, the oral cavity signal includes a tongue pressure signal; Or the oral signals may include tongue pressure signals, tooth biting force signals, and oral temperature and humidity signals.

[0031] In step S1, the signal acquisition module 1 acquires each modal signal and performs filtering and other processing on each modal signal.

[0032] EEG signals are acquired via an EEG acquisition device 11, a head-mounted array electrode structure with at least 8 channels. It features a high-impedance, low-noise front end and, combined with optimized lead layout and motor cortex-oriented sampling, stably captures high signal-to-noise ratio signals related to swallowing imagery / intention. A decoding engine based on machine learning and deep learning performs in-depth analysis of the temporal and frequency characteristics and spatial patterns of the EEG signals to accurately distinguish between swallowing preparation / intention states and resting states, outputting highly reliable trigger commands. The system incorporates ensemble learning, convolutional networks, and online adaptive mechanisms, enabling it to dynamically optimize recognition thresholds and stimulation parameters based on individual patient physiological differences and rehabilitation progress, ensuring individualized treatment adaptation and long-term stability. Multi-channel EEG data transmission is primarily achieved via Wi-Fi communication.

[0033] Oral signals are collected via a wearable oral device 12. The wearable device 12 is worn during eating to monitor and upload oral signals in real time. The wearable oral device 12 includes, but is not limited to, a combination of a tongue pressure sensor array, a tooth occlusal force sensor, and an intraoral temperature and humidity sensor. The tongue pressure sensors are distributed at multiple points on the tongue surface or palate to collect tongue pressure amplitude, pressure distribution, and rate of change in real time, capturing the key signal of the tongue pushing the food bolus when swallowing begins. The tooth occlusal force sensor is placed in the dental arch or denture base to detect changes in occlusal force during chewing and swallowing, distinguishing between unconscious chewing and effective swallowing actions, and reducing the false trigger rate. The intraoral temperature and humidity sensor monitors the oral environment to help determine whether the patient is eating / swallowing and filters signal interference in non-treatment scenarios. Communication is primarily via BLE 5.0+ to achieve low-latency, high-reliability wireless data transmission with the host, supporting real-time data streaming.

[0034] Electromyographic signals are acquired through an electromyographic signal acquisition device 13 with no less than two channels, acquiring electromyographic signals from facial muscles such as the masseter and buccinator, as well as the suprahyoid and infrahyoid muscle groups.

[0035] Signal processing for each mode: EEG signals: First, adaptive notch filtering (50Hz power frequency interference) is applied, followed by Laplace lead spatial filtering and CSP spatial filtering to remove eye movement, chewing, and neck muscle artifacts.

[0036] Oral signal: A 5th-order Butterworth low-pass filter (cutoff frequency 10Hz) was used to remove high-frequency noise, and the data amplitude was calibrated using a standard pressure source.

[0037] Electromyography (EMG) signals: Bandpass filtering (20-500Hz) was used to remove power frequency and baseline drift, and time-domain features such as root mean square (RMS) and integral EMG were extracted, which can be used for subsequent feedback evaluation.

[0038] In step S1 or step S2, EEG signals, oral signals and pharyngeal electromyography signals are collected synchronously to ensure that the above three modal signals have a unified timestamp; Specifically, synchronized data acquisition is achieved through a synchronizer. For example, the host computer software controls the synchronizer to issue serial port commands for tagging, or external light, sound, or switch signals are sent to the synchronizer for tagging. The synchronizer then broadcasts the time synchronization command to the EEG collector, tongue pressure sensor, and EMG collector via broadcast (e.g., using an independent 5.8GHz wireless transmission module with a latency of less than 1ms). Upon receiving the tagging signal, each collector and sensor synchronously marks the data in all uploaded data, enabling synchronized acquisition of various modal signals such as EEG, tongue pressure sensor (or teeth occlusion signal), and pharyngeal EMG signals by the host computer software, ensuring that all modal signals have a unified timestamp during acquisition. In step S1 or S2, the acquired EEG signals, oral cavity signals, and pharyngeal electromyography signals are unified to the same time resolution through downsampling or interpolation. For example, an integer multiple of the lowest sampling rate is used, or the frequency is unified to 200Hz to meet the low-frequency characteristics of swallowing movements and achieve unified time resolution. Then, the acquired modal signals are subjected to denoising and filtering, data segmentation and labeling, and dataset partitioning and layering.

[0039] In step S1 or step S2, the timing alignment of the acquired EEG signals, oral signals and pharyngeal electromyography signals is dynamically achieved through the TAD-CAT algorithm, or the timing alignment of the acquired EEG signals, oral signals and pharyngeal electromyography signals is achieved through the TAD-CAT algorithm and the Transformer timing attention mechanism.

[0040] Therefore, by coordinating hardware and algorithms, the timing alignment of each modal signal is achieved, thereby improving the accuracy of signal processing.

[0041] In step S2, 1D-CNN extracts local features of each modality signal, and 1D-CNN and Transformer establish long-range temporal dependencies between each modality signal. It combines at least the transient rhythm of EEG signals, the burst onset of EMG signals and the peak height of tongue pressure to learn the dependencies between different time steps and between different modality features, capture cross-modal delays, and then perform training.

[0042] During the training process in step S2, corresponding records are made based on the characteristics of the population, the performance of the model in different populations is observed, and domain adaptation or multi-task learning is performed. Among these, population characteristics include age, gender, and disease. Therefore, by adapting to the individual characteristics of the population, the accuracy and personalization of swallowing therapy can be improved.

[0043] In step S2, the time-series correlation model can calculate the correlation coefficient between the predicted delay and the actual delay; Specifically, this includes: calculating the Pearson correlation coefficient and root mean square error (RMSE) between the predicted waveform and the actual waveform; The temporal association model can use Dynamic Time Warped Distance (DTW) to evaluate shape similarity.

[0044] Therefore, monitoring and evaluating the model's learning and training is beneficial for subsequent error assessment, model optimization, and iterative updates.

[0045] In step S2, the following spatiotemporal sequence of pharyngeal movement delay and muscle activation is generated and output.

[0046] Throat movement delay: a scalar value representing the latency from neural command (EEG signal characteristics) to muscle response (tongue pressure / EMG activation).

[0047] Spatiotemporal sequence of muscle activation: The distribution of muscle activation intensity over a continuous period of time (such as tongue pressure waveform, pharyngeal muscle EMG envelope) can be output in the form of time series, or the key activation moments (such as pharyngeal initiation point, peak time, etc.) can be predicted.

[0048] Therefore, by capturing the long-term dependence of cross-modal signals through the Transformer self-attention mechanism, the weights of swallowing intention-related features are strengthened, and finally, feature vectors that fuse signals from various modalities are generated to produce spatiotemporal sequences of pharyngeal movement delay and muscle activation. Furthermore, hierarchical training and validation are performed based on population characteristics such as age and gender to obtain a digital representation from neural control to muscle execution, providing reliable parameters for physiological analysis and clinical control.

[0049] In step S3, the food properties include: elastic modulus, adhesive force, and rheological constitutive parameters; The quantitative parameters of swallowing efficiency include: pharyngeal contraction rate, food transport rate, and food residue.

[0050] This leads to the formation of a functional prediction platform based on "food-muscle group-efficacy".

[0051] Oropharyngeal biomechanical simulation models include oropharyngeal physiological and anatomical models, food physical property input models, numerical simulation models, swallowing efficacy quantification and verification models, and hierarchical and generalization optimization models.

[0052] Oropharyngeal physiological and anatomical model: Based on CT and / or MRI images of the human oropharynx, the three-dimensional contours of key structures such as the pharynx, tongue, soft palate, and esophageal inlet are extracted using medical image segmentation software to build a basic model.

[0053] Food property input model: Transforms food property parameters into a simulation-recognizable material model to match the mechanical behavior of the swallowing process.

[0054] Numerical simulation model: Based on the simulation framework, it integrates geometry, mechanics, and input parameters to realize the dynamic simulation of the swallowing process.

[0055] Quantitative and Validation Model of Swallowing Efficacy: Core efficacy parameters are extracted from simulation results to establish a quantitative system. These include parameters such as pharyngeal contraction rate, food transport rate, and amount of food residue. These indicators are measured using clinical methods such as swallowing imaging during actual swallowing, and compared with simulation results to correct model parameters.

[0056] Hierarchical and generalized optimization model: Combining age and / or gender stratification of the temporal motion algorithm, the population adaptability of the simulation model is optimized, and finally the parameters output by this model are used in the interactive system of the swallowing disorder treatment device.

[0057] In step S3, the maximum mean difference (MMD) distance between the real-time features and the initial model features is calculated to determine whether it is the first time the feature has been used or whether the feature has drifted. The MMD distance threshold is 0.15. If it is the first time to use or the feature is drifting, a meta-learning architecture based on Prototypical Network is adopted. Using individual effective data, such as 30 to 50 sets of individual effective data, only the parameters of the upper attention layer and classifier of Transformer are finely adjusted to achieve rapid adaptation to individual features. If the features do not drift, the previously stored individual parameters are directly called.

[0058] This improves the accuracy of output parameters, enabling the creation of personalized treatment plans.

[0059] Step SF1 is also included between steps S2 and S3; Step SF1: Verification of the validity of the swallowing intention: Based on the triggering of swallowing intention, the effectiveness of swallowing intention is verified by dual threshold rules and temporal constraints; Among them, the double threshold rule is: the amplitude of the oral signal is greater than or equal to the preset physiological threshold, and the probability of intention of the EEG signal is greater than or equal to 0.85; the preset physiological threshold is 1.5 times the individual baseline value; Timing constraint: EEG signals are valid within 200-500ms after oral signal triggering; If the dual threshold and timing constraints are met, it is determined to be a valid intent; otherwise, it is determined to be a false trigger or no intent, and the process returns to step S1: signal acquisition.

[0060] Therefore, the effectiveness of swallowing intent is verified by a dual-condition approach of double threshold rules and temporal constraints, thereby improving accuracy.

[0061] The swallowing intention can be triggered by the swallowing determination algorithm and the decoding of EEG signals. In step S1, when the signals of each modality are collected, the EEG signals are decoded and analyzed, and the swallowing intention is identified in conjunction with the swallowing determination algorithm.

[0062] In step S4, while outputting the neuromuscular electrical stimulation (NMES) sequence, the electromyographic signals of the swallowing muscle group after stimulation are simultaneously acquired, and the activation response of the pharyngeal muscles is recorded.

[0063] The low-frequency electrical stimulation output device is used to generate and output neuromuscular electrical stimulation (NMES) sequences. The output electrodes are distributed on the skin of the cheeks and neck, and electrical stimulation is performed on the tongue-pharyngeal-laryngeal muscle groups, thereby enhancing proprioceptive input and inducing muscle contraction, and bidirectionally promoting the functional remodeling and coordinated recovery of peripheral sensorimotor pathways and central control networks.

[0064] The method further includes: step S5; S5: Dynamically optimize model parameters: By comparing the feedback electromyographic signals, the expected contraction intensity, and the deviation from the timing, the error value is calculated. If the error is ≤5%, save the current parameters; If the error is greater than 5%, initiate adaptive updates by filtering data groups through a sliding window mechanism, such as generating pseudo-labels and filtering confidence scores based on 20 data groups in a sliding window, and adjusting model weights and stimulus parameters.

[0065] Thus, by monitoring physiological parameters, evaluating exercise effects, and collecting patient feedback, the execution results and efficacy data are fed back to drive the algorithm model to be optimized and iterated, forming a closed loop.

[0066] Therefore, this approach integrates multimodal signal data from oral cavity signals, electroencephalogram (EEG) signals, food characteristics, and demographic and pathological characteristics. Through iterative correction of patient data from different etiological types, it generates individualized prescriptions containing key parameters such as temporal characteristics, stimulus intensity, and location, thereby improving the effectiveness of swallowing therapy.

[0067] refer to Figure 4 A multimodal collaborative swallowing simulation electrical stimulation system based on a brain-computer interface, used to perform the above method, the system comprising: Signal acquisition module 1 is used to acquire multimodal signals such as electroencephalogram (EEG) signals, oral cavity signals, and electromyogram (EMG) signals; The signal acquisition module includes an electroencephalogram (EEG) signal acquisition device 11, an oral wearable device 12, and an electromyogram (EMG) signal acquisition device 13, etc. Temporal correlation module 2 is used to establish a temporal correlation model and generate and output the spatiotemporal sequences of pharyngeal movement delay and muscle activation. Swallowing intent verification module 3 is used to verify the validity of swallowing intent; Swallowing simulation module 4 is used to establish an oropharyngeal biomechanical simulation model. It simulates the food swallowing process based on food properties, pharyngeal movement delay, and spatiotemporal sequence of muscle activation, and generates and outputs quantified parameters of swallowing efficacy and temporal stimulation commands. The electrical stimulation output module 5 generates and outputs a neuromuscular electrical stimulation (NMES) sequence based on the pharyngeal effect energy quantification parameters and the timing stimulation command; wherein, the electrical stimulation output module includes a low-frequency electrical stimulator output device; Feedback optimization module 6 optimizes model parameters based on the feedback electromyographic signals.

[0068] It also includes: a swallowing trigger determination module 7 and an EEG signal decoding module 8; Swallowing trigger determination module 7 is used to determine the trigger of swallowing intention; EEG signal decoding module 8 is used for decoding and analyzing EEG signals.

[0069] The above description only discloses some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the invention.

Claims

1. A multimodal collaborative swallowing simulation electrical stimulation method using a brain-computer interface, characterized in that, Includes the following steps: S1: Signal Acquisition Three modalities of signals were collected: electroencephalogram (EEG), oral cavity signals, and pharyngeal electromyography (EMG). S2: Establishment of the temporal correlation model: A temporal correlation model is established between the collected EEG signals, oral signals, and pharyngeal electromyography signals using a 1D-CNN+Transformer architecture; this model promotes a nonlinear mapping relationship between the above three modal signals and the activity of the pharyngeal muscle groups; and generates and outputs the spatiotemporal sequences of pharyngeal movement delay and muscle activation. S3: Swallowing simulation: Establish an oropharyngeal biomechanical simulation model to simulate the food swallowing process based on food properties, pharyngeal movement delay, and spatiotemporal sequence of muscle activation, and generate and output quantifiable parameters of swallowing efficacy and temporal stimulation commands. S4: Output of the neuromuscular electrical stimulation (NMES) sequence: Based on the pharyngeal effect energy quantification parameters and timing stimulation instructions, a neuromuscular electrical stimulation (NMES) sequence is generated and output.

2. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interfaces according to claim 1, characterized in that, In step S1, the oral cavity signal includes a tongue pressure signal; Or the oral signals may include tongue pressure signals, tooth biting force signals, and oral temperature and humidity signals.

3. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interfaces according to claim 1, characterized in that, In step S1 or step S2, EEG signals, oral signals and pharyngeal electromyography signals are collected synchronously to ensure that the above three modal signals have a unified timestamp; By downsampling or interpolation, the acquired EEG signals, oral cavity signals, and pharyngeal electromyography signals are unified to the same temporal resolution; The timing alignment of acquired EEG, oral cavity, and pharyngeal electromyography signals can be dynamically achieved through the TAD-CAT algorithm, or by combining the TAD-CAT algorithm with the Transformer timing attention mechanism.

4. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interfaces according to claim 1, characterized in that, In step S2, 1D-CNN extracts local features of each modality signal, and 1D-CNN+Transformer establishes long-range temporal dependencies between each modality signal. It combines at least the transient rhythm of EEG signals, the burst onset of EMG signals and the peak height of tongue pressure to learn the dependencies between different time steps and between different modality features, capture cross-modal delays, and then performs training.

5. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interfaces according to claim 4, characterized in that, During the training process in step S2, corresponding records are made based on the characteristics of the population, the performance of the model in different populations is observed, and domain adaptation or multi-task learning is performed. Among these, population characteristics include age, gender, and disease; The time-series correlation model can calculate the correlation coefficient between predicted delay and actual delay; Specifically, this includes: calculating the Pearson correlation coefficient and root mean square error (RMSE) between the predicted waveform and the actual waveform; The temporal association model can use Dynamic Time Warped Distance (DTW) to evaluate shape similarity.

6. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interfaces according to claim 1, characterized in that, In step S3, the food properties include: elastic modulus, adhesive force, and rheological constitutive parameters; The quantitative parameters of swallowing efficiency include: pharyngeal contraction rate, food transport rate, and food residue.

7. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interface according to claim 1, characterized in that, In step S3, the maximum mean difference (MMD) distance between the real-time features and the initial model features is calculated to determine whether it is the first time the feature has been used or whether the feature has drifted. The threshold for the maximum mean difference (MMD) distance is 0.

15. If it is the first time using the technology or if there is feature drift, a meta-learning architecture based on Prototypical Network is adopted, which uses individual effective data to fine-tune the parameters of the upper attention layer and classifier of Transformer. If the features do not drift, the previously stored individual parameters are directly called.

8. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interface according to claim 1, characterized in that, Step SF1 is also included between steps S2 and S3; Step SF1: Verification of the validity of the swallowing intention: Based on the triggering of swallowing intention, the effectiveness of swallowing intention is verified by dual threshold rules and temporal constraints; Among them, the double threshold rule is: the amplitude of the oral signal is greater than or equal to the preset physiological threshold, and the probability of intention of the EEG signal is greater than or equal to 0.85; the preset physiological threshold is 1.5 times the individual baseline value; Timing constraint: EEG signals are valid within 200-500ms after oral signal triggering; If the dual threshold and timing constraints are met, it is determined to be a valid intent; otherwise, it is determined to be a false trigger or no intent, and the process returns to step S1: signal acquisition.

9. The multimodal collaborative swallowing simulation electrical stimulation method for brain-computer interface according to claim 1, characterized in that, In step S4, while outputting the neuromuscular electrical stimulation (NMES) sequence, electromyographic signals of the swallowing muscle group are simultaneously acquired after stimulation. The method further includes: step S5; S5: Dynamically optimize model parameters: By comparing the feedback electromyographic signals, the expected contraction intensity, and the deviation from the timing, the error value is calculated. If the error is ≤5%, save the current parameters; If the error is greater than 5%, adaptive updates are initiated, filtering data sets through a sliding window mechanism and adjusting model weights and stimulus parameters.

10. A multimodal collaborative swallowing simulation electrical stimulation system for brain-computer interfaces, characterized in that, include: The signal acquisition module is used to acquire electroencephalogram (EEG) signals, oral cavity signals, and electromyogram (EMG) signals. The temporal correlation module is used to establish a temporal correlation model and generate and output the spatiotemporal sequences of pharyngeal movement delay and muscle activation. The swallowing intent verification module is used to verify the validity of swallowing intent; The swallowing simulation module is used to establish an oropharyngeal biomechanical simulation model. It simulates the food swallowing process based on the food properties, pharyngeal movement delay, and spatiotemporal sequence of muscle activation, and generates and outputs quantified parameters of swallowing efficacy and temporal stimulation commands. The electrical stimulation output module generates and outputs a neuromuscular electrical stimulation (NMES) sequence based on the pharyngeal effect energy quantification parameters and timing stimulation instructions. The feedback optimization module optimizes the model parameters based on the feedback electromyographic signals.

Images