Swallowing interaction system and method based on multi-modal collaborative fusion

Abstract

The application discloses a swallowing interaction system and method based on multi-modal collaborative fusion. The system comprises: an interaction task paradigm guiding module, which is used for constructing an interaction task paradigm and outputting personalized guiding content to induce a swallowing intention; a multi-modal signal acquisition module, which synchronously acquires electroencephalogram, electromyogram and video signals of a user when the user performs the interaction task; a multi-modal data processing module, which is used for feature extraction, construction of a fusion feature vector through cross-modal feature engineering and a gated attention mechanism; an intention decoding module, which outputs an intention category based on the fusion feature; and an intelligent decision module, which dynamically adjusts the guiding content in real time according to an intention recognition result, forming a closed-loop interaction. The application overcomes the recognition limitation of a single mode through multi-source information fusion, realizes high-precision matching of central nervous intention and peripheral motor behavior, and effectively improves the intelligent level, safety and user participation of swallowing interaction training.

Description

Technical Field

[0001] This invention relates to the field of bio-information signal processing and intelligent human-computer interaction technology, specifically to a swallowing interaction system and method based on multimodal collaborative fusion. Background Technology

[0002] Swallowing is a complex physiological process involving precise control of the central nervous system and high coordination of peripheral muscle groups. Existing swallowing aids and intervention programs still face the following pressing technical bottlenecks in practical applications:

[0003] 1. Open-loop disconnect between macro-level decision-making and micro-level execution:

[0004] Current intervention models primarily rely on static plans developed by operators based on offline assessment scales or personal experience, with the execution device merely serving as a passive output of the pre-set program. This model leads to an information gap between "macro-level plan decision-making" and "micro-level execution," as the device cannot perceive the user's real-time neurophysiological state during training (such as EEG fluctuations and EMG intensity), making it difficult to dynamically optimize parameters based on the user's instantaneous fatigue or attention level, and lacking a refined closed-loop control mechanism.

[0005] 2. Lack of accurate differentiation between proactive intentions and reactive responses:

[0006] Traditional non-medullary electrical stimulation (NMES) or simple biofeedback techniques typically rely on fixed time series or single amplitude thresholds for triggering. Due to a lack of in-depth exploration of the correlation between "central intention" and "peripheral execution," existing technologies cannot effectively distinguish whether a user's action stems from conscious "active swallowing" or a "passive response" induced by external physical stimuli. This lack of discernment results in a lack of targeted intervention, hindering the precise induction of neuroplasticity.

[0007] 3. Static limitations of training parameters and bottlenecks in interactive training:

[0008] Existing interactive training systems mostly employ fixed temporal parameters, stimulus intensity, and guidance frequency. However, a user's physiological functions and motor control abilities evolve dynamically over time. Fixed parameter settings cannot adapt to the user's evolving needs from initial awareness to later performance improvement, resulting in a mismatch between training load and actual function. When a user enters a plateau in functional recovery, the system cannot dynamically optimize the interaction strategy through adaptive learning algorithms, thus limiting further improvements in training efficiency.

[0009] Therefore, how to construct an intelligent closed-loop system that can deeply integrate multimodal physiological signals, accurately decode active movement intentions, and achieve adaptive evolution of training parameters has become a core technical challenge in the field of human-computer interaction assisted training. Summary of the Invention

[0010] The technical objective of this invention is to address the above-mentioned technical problems existing in current swallowing interaction systems by providing a swallowing interaction system and method based on multimodal collaborative fusion.

[0011] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution.

[0012] In a first aspect, embodiments of the present invention provide a swallowing interaction system based on multimodal collaborative fusion, including: an interaction task paradigm guidance module, used to construct an interaction task paradigm and output personalized guidance content to the user as a neural and physiological stimulus source during the guidance phase, so as to guide the user to swallow and perform the interaction task within a specific time window;

[0013] The multimodal signal acquisition module is used to acquire EEG signals, EMG signals, and video signals in real time when the user performs swallowing interaction tasks;

[0014] The multimodal data processing module is used to synchronize and extract features from the acquired signals to obtain EEG feature vectors, EMG feature vectors, and video feature vectors; using cross-modal feature engineering and gating attention mechanisms, the feature vectors of each modality are fused to obtain a fused feature vector;

[0015] The intent decoding module is used to output the corresponding swallowing intent category based on the fused feature vector;

[0016] The intelligent decision-making module is used to dynamically adjust the guidance content in the interactive task paradigm guidance module in real time based on the swallowing intention category.

[0017] In a second aspect, embodiments of the present invention provide a swallowing interaction method based on multimodal collaborative fusion, which is applied to the swallowing interaction system based on multimodal collaborative fusion provided in the first aspect, including: constructing an interaction task paradigm, and outputting personalized guidance content to the user as a neural and physiological stimulus source during the guidance phase, so as to guide the user to perform a swallowing interaction task within a specific time window;

[0018] Real-time acquisition of EEG, EMG, and video signals from users while performing swallowing interaction tasks;

[0019] The acquired signals are synchronized and feature extracted to obtain EEG feature vectors, EMG feature vectors, and video feature vectors; cross-modal feature engineering and gating attention mechanisms are used to fuse the feature vectors of each modality to obtain a fused feature vector;

[0020] Based on the fused feature vector, the corresponding swallowing intention category is output;

[0021] Based on the swallowing intention category, the guidance content in the interactive task paradigm guidance module is dynamically adjusted in real time.

[0022] It should be understood that the summary section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0023] Compared with existing technologies, the beneficial technical effects achieved by this invention are as follows: This invention significantly improves the accuracy of swallowing intention recognition and the intelligence level of swallowing interaction training through multimodal collaborative fusion technology. Specifically, it achieves high robustness and high accuracy in intention recognition: Through cross-modal feature engineering and gated attention mechanisms, the system can deeply mine the spatiotemporal correlation features between EEG motor preparation potentials, electromyographic burst intensity, and video laryngeal movement trajectories. Compared with single-modality approaches, this invention effectively overcomes the interference of environmental noise and individual physiological differences, achieving accurate decoding of the entire chain from central intention to peripheral execution, greatly reducing the false trigger rate of intention recognition. Attached Figure Description

[0024] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this application in any way. Furthermore, the shapes and scales of the components in the drawings are merely illustrative to aid in understanding this application and do not specifically limit the shapes and scales of the components. Those skilled in the art, guided by the teachings of this application, can select various possible shapes and scales to implement this application according to specific circumstances. In the drawings:

[0025] Figure 1 A schematic diagram of the swallowing interaction system framework based on multimodal collaborative fusion provided for the embodiments;

[0026] Figure 2 A schematic diagram of the swallowing interaction method based on multimodal collaborative fusion provided for the embodiment;

[0027] Figure 3 This is a schematic diagram illustrating the process of generating the fused feature vector in the embodiment.

[0028] Figure 4 This is a schematic diagram of the electromyography feature extraction process in the example. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0030] It should be fully understood that the user's EEG signals, multimodal brain imaging data, etc. involved in this invention are all information and data authorized by the user or fully authorized by all parties. The use of user information should follow the privacy policies and practices of the industry that are generally considered to meet or exceed the requirements for maintaining user privacy. The collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation access points for users to choose to authorize or refuse.

[0031] Swallowing interaction systems based on multimodal collaborative fusion, such as Figure 1 As shown, it includes an interactive task paradigm guidance module, a multimodal signal acquisition module, a multimodal data processing module, an intent decoding module, and an intelligent decision-making module.

[0032] The interactive task paradigm guidance module is used to construct the interactive task paradigm and output personalized guidance content to the user during the guidance phase as a source of neural and physiological stimulation to guide the user to perform the swallowing interactive task within a specific time window.

[0033] The multimodal signal acquisition module is used to acquire EEG signals, EMG signals, and video signals in real time when the user performs swallowing interaction tasks.

[0034] The multimodal data processing module is used to synchronize and extract features from the acquired signals to obtain EEG feature vectors, EMG feature vectors, and video feature vectors. Using cross-modal feature engineering and gating attention mechanisms, the feature vectors of each modality are fused to obtain a fused feature vector.

[0035] The intent decoding module is used to output the corresponding swallowing intent category based on the fused feature vector and the swallowing state.

[0036] The intelligent decision-making module is used to dynamically adjust the guidance content in the interactive task paradigm guidance module in real time based on the swallowing intention category.

[0037] Based on the same inventive concept as the swallowing interaction system based on multimodal collaborative fusion in the above embodiments, this application also provides a swallowing interaction method based on multimodal collaborative fusion, which is applied to the above swallowing interaction system. The method includes: constructing an interaction task paradigm and outputting personalized guidance content to the user as a neural and physiological stimulus source during the guidance phase, so as to guide the user to perform swallowing interaction tasks within a specific time window.

[0038] Real-time acquisition of EEG, EMG, and video signals from users while performing swallowing interaction tasks;

[0039] The acquired signals are synchronized and feature extracted to obtain EEG feature vectors, EMG feature vectors, and video feature vectors; cross-modal feature engineering and gating attention mechanisms are used to fuse the feature vectors of each modality to obtain a fused feature vector;

[0040] Based on the fused feature vector, the corresponding swallowing intention category is output according to the swallowing state;

[0041] Based on the swallowing intention category, the guidance content in the interactive task paradigm guidance module is dynamically adjusted in real time.

[0042] In this embodiment, the framework of the interactive task paradigm includes a guidance phase. The guidance phase requires guiding content (such as a combination of visual and auditory elements) to induce the user to actively swallow.

[0043] Optionally, the interactive task paradigm also includes a feedback phase and a rest phase. The feedback phase uses devices of different modalities to provide feedback on the execution results of the current interactive task, helping users to improve or encouraging them to actively engage in swallowing interaction training. A random event interval (i.e., a rest phase) can be set between two swallowing interaction tasks to prevent users from preparing for the swallowing interaction task in advance, thereby prematurely inducing changes in brain electrical activity.

[0044] In a specific embodiment, the guidance content may include the style of guidance elements, the type of elements, or sound effects. Surveys can be conducted to collect information during the user's first swallowing interaction training, and the generated content can be shown to the user on the spot. The generated guidance elements can be retained as a reference based on subjective satisfaction. Subsequently, various details can be modified to generate guidance content with the same main theme during real-time interactive training.

[0045] In some embodiments, in personalized content satisfaction surveys, more accurate scoring can be achieved through EEG emotion monitoring, sampling common scoring and accuracy rates, and a comprehensive and optimal satisfaction model with generalizability. After the user selects the personalized guidance content, simulated swallowing interaction training is conducted, and the user's EEG signals are collected for analysis and evaluation to ensure the authenticity of the selection.

[0046] As an example, during the preparation phase of the interactive system, the interactive task paradigm guidance module uses user-defined voice effects (such as male or female voices) to provide preparation prompts, guiding the user to fix their head onto the designated bracket. Then, a straw containing a swallowing induction device is placed in the user's mouth. The swallowing induction device induces the user to swallow by spewing liquid.

[0047] In one embodiment, before the start of the guidance phase, the interactive task paradigm guidance module prompts the user to prepare by providing guidance content including sound and video animation elements, and then enters the guidance phase after waiting for a first preset duration (e.g., 2 seconds).

[0048] The guidance phase includes prompting the user through an audiovisual module (optionally, the visual effect is the type of food the user likes) to guide the user to imagine swallowing. The EEG collection time starts from the moment the guidance prompt is issued and ends when the imagined swallowing interactive task of the second preset duration (e.g., 3 seconds) is completed.

[0049] In some embodiments, during the swallowing interaction guidance process, the interactive task paradigm guidance module can use attention to determine whether the user is fully prepared to enter the guided collection process.

[0050] As an example, attention metrics can be monitored using the power ratios of brain functional bands, which have lower computational complexity, thus improving real-time performance.

[0051] Optionally, the guidance phase can proceed only after the post-guidance attention index exceeds a threshold and remains above a third preset duration (e.g., 1 second). Furthermore, during the guidance phase, P300 evoked potentials can be used to further enhance attention assessment by inducing steady-state visual evoked potentials by altering the flashing frequency of the throat region in the animation.

[0052] In some embodiments, the feedback phase begins after the guidance phase. If the purpose of the swallowing interaction guidance phase is primarily to collect signal data of the user performing the swallowing interaction task, then the feedback phase directly processes the correct result by triggering the swallowing induction device in the physical feedback module to spray a small dose (less than the dose threshold) of liquid (which can be a user-customized beverage or yogurt, etc.) to induce the user's swallowing reflex.

[0053] In some embodiments, the physical feedback module also includes an electrical stimulation device. For users with severe swallowing difficulties, electrical stimulation can be added to assist swallowing by attaching the electrodes of the electrical stimulation device to the swallowing-related muscle groups. The swallowing process mainly involves the activation of the laryngeal muscles to swallow food. During interactive tasks, the electrical stimulation device assists in the swallowing task through auxiliary neural electrical stimulation. Furthermore, in closed-loop control, the interactive training effect is enhanced through a linkage closed-loop control with EEG-EMG-electrical stimulation.

[0054] The entire interactive task paradigm process is divided into a training phase and a prediction phase, with the training phase primarily used to collect data for modeling. In some embodiments, the system also includes a cloud platform for synchronously storing and updating user personalized preference data and historical interaction data; the cloud platform performs big data analysis based on historical interaction state data to iteratively optimize the weights of the gating attention mechanism in the intent decoding module, and, combined with personalized preference data, issues customized interaction schemes.

[0055] Customized training can be performed using the cloud platform. The default training method uses a preset number of interactions (e.g., 15 times), and swallowing-inducing feedback is performed with each interaction.

[0056] In some embodiments, the interactive task paradigm guidance module uses generative AI combined with user preference data stored on a cloud platform, or utilizes scales to assess personalized user data, such as... Figure 2 As shown, pre-producing or generating guided content in real time can reduce the complexity of the development process and adapt to a wide variety of user groups. The example enhances the initiative and engagement of the interactive training process. By utilizing generative AI combined with visual mirror stimulation and steady-state visual evoked potentials (SSVEP) technology, the system transforms the tedious swallowing practice into a highly immersive interactive task. This proactive evoked mechanism effectively activates the mirror neuron system, enhances neural remodeling effects, and significantly improves user training compliance and psychological satisfaction.

[0057] The prediction phase is also performed a preset number of times via the cloud platform, but the overall paradigm time is kept within a fixed range, such as 10 minutes (maximum time), to prevent user resistance. In some embodiments, each interactive training session provides corresponding feedback based on the predicted imagination result (e.g., feedback is given if the imagination is correct, but no feedback is given if the imagination is incorrect). In addition, in some embodiments, the task interaction system also adds a compensation mechanism to prevent users from making repeated mistakes when they are in a poor state, thus preventing them from developing resistance to the interactive task.

[0058] In this embodiment, the multimodal signal acquisition module specifically includes:

[0059] Surface electromyography (EMG) signal acquisition unit: Using an EMG acquisition device with a sampling rate of 1024Hz, electrode pads are attached to the neck to acquire surface EMG changes when swallowing is successfully induced. Wavelet denoising is used to retain data in the 20-500Hz frequency band, and adaptive thresholding is performed. The EMG signals are used to assess the effectiveness of swallowing by observing the contraction and relaxation states of muscles around the swallowing site, and can also serve as a primary basis for judgment in closed-loop control.

[0060] Video signal acquisition unit: Acquires video signals from the user's face and neck using a high-definition 1080P camera with a 30Hz sampling rate. The facial visual signals recorded by the camera are used to monitor the user's facial expressions (natural, tense / uncomfortable) during swallowing in real time, and the visual characteristics of laryngeal movements are cross-validated with electromyographic signals to improve robustness.

[0061] EEG signal acquisition unit: The EEG equipment uses a standard 32-lead EEG setup with a sampling rate of 512Hz and employs wet electrodes to reduce external interference. Power frequency interference is removed using a 48-52Hz notch filter, and effective frequency bands are preserved using a 4-30Hz bandpass filter. Independent component analysis is then used to remove motion artifacts (blinking, chewing). EEG signals are used as the primary detection method to identify "swallowing imagery" and "active / passive intentions." Furthermore, during monitoring, emotionally induced states can be used to assist in observing swallowing moments and trigger key closed-loop control mechanisms.

[0062] In some embodiments, for ease of interactive training, the EMG and EEG acquisition devices employ a wireless acquisition mode. Video is also transmitted via RTSP streaming, for which an edge-cloud collaborative approach is used for multimodal data synchronization.

[0063] In some embodiments, the multimodal data processing module first sets up a master clock source at the edge device gateway and sends synchronization time packets to the multimodal devices (EMG acquisition devices, cameras, EEG devices). Upon receiving these packets, each modal device calculates its time difference with the master clock and establishes a short-term mapping relationship. Then, when transmitting data, each modal device sends its local time to the edge device gateway using the corresponding mapped timestamp. Finally, the edge device gateway establishes a cache using the data from each modality and interpolates the overall caches across different modalities over a certain period to align the times.

[0064] In this embodiment, the multimodal data processing module extracts features from the acquired signals, such as... Figure 3 As shown, it includes: 1) EEG feature extraction and EEG feature vector:

[0065] Temporal features include swallowing-related event-related potential (ERP) features, as well as temporal statistical features such as the Hjorth parameter to represent signal complexity and rate of change; mean μ and variance σ² to represent baseline stability and signal strength, where: μ = Σx / n; σ² = Σ(x-μ)² / n; n represents the total number of samples, and x represents the sample point.

[0066] Frequency domain characteristics: Since swallowing imagery mainly involves activation of the motor cortex, it primarily includes the alpha band (8~13Hz), beta band (13~30Hz), and attention-based monitoring, including the theta band (4~8Hz). Short-time Fourier transform (STFT) is used to obtain the power energy of different frequency bands.

[0067] Spatial characteristics: The swallowing state and the resting state are distinguished by CSP (common spatial pattern).

[0068] EEG feature vector: The EEG feature vector F_eeg is obtained by concatenating the features in the time domain, frequency domain, and spatial domain.

[0069] 2) Electromyographic feature extraction and electromyographic feature vector, such as Figure 4 As shown:

[0070] To characterize the temporal coordination of swallowing movements, electromyography (EMG) features are extracted using the DTW (Dynamic Time Adjustment) algorithm. The EMG sequence of each swallow is compared with a standard "ideal swallowing template," and the DTW distance is used as a feature to directly measure the completeness and coordination of motor execution.

[0071] In this embodiment, electromyography feature extraction includes the following steps:

[0072] Template Establishment: By selecting multiple healthy users, swallowing data was collected and recorded using a swallowing paradigm to achieve the best performance. Electromyography (EMG) signals from multiple channels (such as the submental muscles and thyrohyoid muscles) were selected, and a complete swallowing cycle (from oral preparation to the end of the pharyngeal phase) was captured. The root mean square (RMS) sequence of the EMG amplitude was extracted, representing the change in the activation intensity of the muscle over time. After time and amplitude normalization, for each channel m, at the normalized time point, the RMS values ​​of all repeated swallowing samples (i.e., all examples) from all healthy users were averaged to obtain the standard swallowing template sequence (dimension: M×L) for that channel: T_std[m]={v m,1 ,v m,2 ,...,v m,L}, where L represents the time dimension.

[0073] Real-time data acquisition: For a single real-time swallow to be evaluated, RMS sequences are acquired from M channels. Since the swallowing duration may vary, its original length N needs to be retained. The real-time data for channel m is denoted as: S_real[m] = {w m,1 ,w m,2 ,...,w m,N}

[0074] Timing alignment and distance calculation:

[0075] Constructing the local distance matrix: First, calculate the local distance d(i,j) between the two sequences (the standard swallowing template sequence and the root mean square sequence of real-time acquired electromyography amplitudes) at each time point pair (i,j), forming an N x L matrix D. The local distance d(i,j) is the sum of the Euclidean distances of all channels at that point pair. , This represents the i-th real-time data from channel m. The standard swallowing template for the i-th time dimension of channel m.

[0076] In this embodiment, this method ensures that the matching is cross-channel collaborative, that is, the matching is of the collaborative pattern of the entire muscle group, rather than a single muscle.

[0077] Finding the optimal curved path: We need to find a path P = {p1, p2, ..., p...} from (1,1) to (N,L). K}, where K represents the total number of elements in the set, p k =(i k ,j k );p k Representing the k-th key point, i k Let j represent the x-coordinate of the k-th key in the image coordinate system. k This represents the ordinate of the k-th key in the image coordinate system; each p k The path P is designed to start from the starting point and end at the ending point; the path must progress over time, and can only move to adjacent grid points at a time. This ensures that the temporal order is not disrupted.

[0078] Dynamic programming solution: The recurrence relation is C(i,j)=d(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)}. The optimal path to (i,j) can only extend from its three adjacent points: (i-1,j) to its left, (i,j-1) below it, or (i-1,j-1) to its lower left, choosing the direction with the minimum cumulative distance.

[0079] Calculate the DTW distance: The sum of local distances along the optimal path P is the final DTW distance D_dtw. It measures the minimum "cost" required to "stretch" or "compress" the user's real-time muscle co-activation pattern to a standard pattern. The smaller the D_dtw value, the closer the user's swallowing pattern is to a standard, coordinated pattern.

[0080] Phase feature extraction: Using the alignment path calculated by DTW, the duration features of each physiological phase of swallowing can be accurately extracted.

[0081] Phase boundary mapping: On the standard template T_std, we have defined the boundary time points of key phases based on physiological knowledge (e.g., j_oral_end represents the end of the oral preparation phase, and j_pharyngeal_start represents the beginning of the pharyngeal phase).

[0082] Finding the corresponding point: On the optimal path P, find all real-time sequence time points {i|(i,j_oral_end) are in or near P} corresponding to the standard template time point j_oral_end. Taking the median or average of these i, the actual time T'_oral_end of the end of the oral preparation period in this swallow can be estimated. Similarly, the swallowing start time T'_swallow_start, swallowing start T_prep, peak time T_peak, and swallowing end T_end time points can be obtained.

[0083] Calculate the duration of each phase: Oral preparation phase duration T_oral_prep = T'_oral_end - T'_swallow_start; Pharyngeal phase duration T_pha = T'_pharyngeal_end - T'_pharyngeal_start; T'_pharyngeal is the start time of the swallowing contraction within the swallowing cycle, T'_pharyngeal_start is the completion time of the swallowing contraction within the swallowing cycle; Relaxation phase duration T_rel = T'_relend - T'rel_start; T'_relend is the end time of the relaxation phase, T'rel_start is the start time of the relaxation phase.

[0084] Electromyographic feature vector:

[0085] F_emg=[D_dtw,T_oral_prep,T_pha,T_rel,RMS_peak,(RMS_left / RMS_right),...]; RMS_peak is the peak value of the real-time acquired EMG amplitude, RMS_left is the real-time acquired root mean square of the left muscle group EMG amplitude, and RMS_right is the real-time acquired root mean square of the right muscle group EMG amplitude.

[0086] 3) Video feature extraction and video feature vectors:

[0087] Visual features can be analyzed in detail using the openFace vision toolkit, and then different facial features can be numbered according to the definition of the Facial Action Coding System (FACS).

[0088] Facial features: including emotion-related and mouth-related features such as (jawline, upper lip, lower lip, left eye, right eye, left eyebrow, right eyebrow, etc.).

[0089] Throat features: The movement state of the mandible is calculated by judging the mandibular opening and closing angle and the mandibular movement trajectory as described above.

[0090] Video feature vector: The video feature vector F_vis is obtained by concatenating facial features and throat features.

[0091] Optionally, the multimodal data processing module performs feature dimensionality reduction on the multimodal features.

[0092] In some embodiments, when performing feature selection, the multimodal data processing module uses a mutual information-based feature selection algorithm for the high-dimensional feature vector of each modality to select the top N features most relevant to the swallowing intention category, forming dimensionality-reduced feature vectors F_eeg_sel, F_emg_sel, and F_vis_sel. During feature dimensionality reduction, principal component analysis is performed on the feature vector of each modality, retaining principal components with a cumulative contribution rate exceeding 95%, thus compressing the feature dimension of each modality to a fixed and lower dimension.

[0093] In some embodiments, the multimodal data processing module utilizes cross-modal feature engineering and gated attention mechanisms to fuse feature vectors from various modalities to construct a fused feature vector. This includes: performing temporal alignment and correlation analysis on the power ratio of specific bands in the motor cortex from EEG features, the duration of swallowing activation phases from EMG features, and the motion trajectory of key points in the larynx from video features within a preset swallowing interaction task time window to generate a cross-modal collaborative feature vector that includes representation of intent-initiation synchronicity, execution-performance synchronicity, and global synergy; inputting the cross-modal collaborative feature vector into a gated attention network to output dynamic importance weights for the three modalities; abstracting the feature vectors of the three modalities independently for each modality through intramodal encoding to obtain corresponding latent vectors; and weighting and summing the latent vectors and dynamic importance weights to obtain the fused feature vector.

[0094] Please continue reading Figure 3 In this embodiment, cross-modal feature engineering uses temporal anchors to establish a joint feature of intent-initiation-execution-performance. Electromyography (EMG) has already acquired the temporal process features of swallowing, including the swallowing initiation state, peak muscle activity, and swallowing termination time.

[0095] As an example, cross-modal feature engineering includes:

[0096] 1. Intention-Initiation Synchronization: EEG Energy Ratio R_prep: This represents the ratio of the average power of the beta band (13-30Hz) in the motor cortex (leads C3 / C4) within a fixed time window (e.g., -300ms to -100ms) before swallowing initiation to the average power of the beta band during the resting period (before the task). A ratio >1 indicates that "event-related desynchronization" has occurred in the motor cortex before the movement is initiated, suggesting active motor preparation. The higher the ratio, the stronger the active intention.

[0097] Initiation synchronicity S_sync_prep: Then, within a time window near the swallowing initiation (e.g., -50ms to +50ms), the negative peak of the motor cortical potentials (MRCP) is calculated. The closer this peak is to the swallowing moment and the larger its amplitude, the better the synchronicity between neural drive and muscle initiation. This can be simply quantified as initiation synchronicity S_sync_prep = 1 / |T_mrcp_peak - T_prep|, where T_mrcp_peak is the MRCP peak time. This directly quantifies the time-locked relationship between "thinking" and "doing". T_prep refers to the physical starting point identified from the multimodal signals when the user begins to perform the swallowing action.

[0098] 2. Execution-Performance Synchronization: Force-Vision Peak Time Difference ΔT_peak: Extract the time T_vis_peak when the laryngeal key point (such as the thyroid cartilage) reaches its highest position from the video features, and obtain ΔT_peak = T_vis_peak - T_peak; T_vis_peak is the peak moment of the visual displacement of the larynx, and T_peak is the peak moment of the muscle tension.

[0099] In efficient swallowing, the peak of muscle tone should be almost synchronized with the peak of the visual displacement of the larynx (ΔT_peak≈0). An excessively large value may indicate low biomechanical efficiency (such as delayed larynx elevation due to uncoordinated muscle contraction).

[0100] The facial expression score C_expr includes the average activation intensity I_d of negative facial expression action units (such as AU4 frowning, AU7 eyelid tension) and the average activation intensity I_e of neutral or positive facial expression action units (such as AU12 upturned corners of the mouth) during the same time period. Therefore, C_expr = I_e / (I_d + 1).

[0101] 3. Global Coordination: Correlation Coefficient R_global: Within the entire swallowing task time window, the Pearson correlation coefficient R_global is calculated from the EEG motor cortex β-wave energy sequence and the EMG RMS sequence. The higher the value, the more consistent the signal change trends of the nerves and muscles, indicating that this was a highly coordinated and neurally driven active swallowing.

[0102] Cross-modal collaborative feature vector: F_cross=[R_prep,S_sync_prep,ΔT_peak,C_expr,R_global].

[0103] In this embodiment, the input layer of the intent decoding module takes into account the feature vectors of the three modalities [F_eeg, F_ems, F_vis], and performs feature abstraction independently for each modality through intramodal encoding to obtain the corresponding latent vectors [Z_eeg, Z_emg, Z_vis].

[0104] In some embodiments, fully connected networks or convolutional neural network layers corresponding to their respective modalities are used as encoders to map heterogeneous EEG feature vectors, EMG feature vectors, and video feature vectors to a latent space of a unified dimension to obtain the corresponding latent vectors.

[0105] In this embodiment, the second layer of the intent decoding module uses a gated attention network; the cross-modal collaborative feature vector F_cross is input into the gated attention network, and the dynamic importance weights of the three modalities [G_eeg,G_emg,G_vis]=GateNet(F_cross) are output.

[0106] GateNet dynamically adjusts the contribution of basic features from each modality based on high-level synergistic relationships (e.g., stronger causal chains rely more on EEG and EMG; inconsistent visual performance reduces video weight). Finally, weighted fusion and decision-making are performed. The weighted fusion feature vector is calculated as follows:

[0107] Z_fused=G_eeg*Z_eeg+G_emg*Z_emg+G_vis*Z_vis.

[0108] In this embodiment, the intent decoding module is specifically used to concatenate the fused feature vector Z_fused with the cross-modal collaborative feature vector F_cross, and then input it into a pre-trained classification model (such as a fully connected layer + Softmax). The classification model learns the components representing electromyographic intensity and the components representing the activation intensity of the motor cortex in the concatenated vector, and maps them in combination with a multidimensional adaptive threshold, thereby outputting the corresponding swallowing intent category, such as obtaining three swallowing intents: strong swallowing intent, weak swallowing intent, and no swallowing intent.

[0109] The classification model extracts the root mean square (RMS) envelope of the electromyographic (EMG) signals on the surface of the submental muscles and captures their peak components within the swallowing task window. Optionally, a weak intent is indicated when the peak component of the EMG signal (muscle tone) is greater than 0 but less than threshold A, and a strong intent is indicated when the muscle tone is greater than threshold B. A and B can be set as needed; this is just an example.

[0110] In this embodiment, the intelligent decision-making module also performs state assessments based on the characteristics of each modality, such as EEG assessment: attention index, cognitive load, and strength of swallowing intention; EMG assessment: activation sequence, intensity, and symmetry of swallowing muscle groups; and visual assessment: completeness of movement and facial expression (happy / sad).

[0111] In some embodiments, the intelligent decision-making module monitors engagement levels to adjust the difficulty of the guidance content.

[0112] As an example, different brain function indicators are acquired by utilizing the energy relationships of frequency band information related to brain function: α (8~13Hz), β (13~20Hz), δ (1~4Hz), and θ (4~8Hz). The timing of user entry into the guidance and the feedback effect are determined based on the quantification level of participation. A higher quantification result indicates stronger user attention and a stronger understanding of the guidance animation during the current guidance phase. Data collection begins after a set threshold (e.g., 1 second) is exceeded. Furthermore, the incentive value can be adjusted during the feedback phase based on the current quantification level of participation.

[0113] As an example, methods for determining engagement metrics include:

[0114] The active attention value S_attention is determined by the formula: S_attention = P_β / (P_α + P_θ);

[0115] P_β represents the power in the β band, P_α represents the power in the α band, and P_θ represents the power in the θ band.

[0116] Determine the comprehensive cognitive load S_cognitive: S_cognitive = P_β / (P_α + P_θ + P_δ); P_δ is the power of the δ frequency band.

[0117] Determine the α asymmetry S_alpha in the prefrontal cortex: S_alpha = P_αr - P_αl, where P_αr is the sum of α energies in the right prefrontal cortex channel and P_αl is the sum of α energies in the left prefrontal cortex channel.

[0118] The different features are normalized and then weighted and fused to form the engagement index S_engage:

[0119] S_engage=a*S_attention+b*S_cognitive+c*S_alpha;

[0120] Where a, b, and c are weighting coefficients.

[0121] In this embodiment, the system also includes a physical feedback module, which includes an electrical stimulation device and a swallowing induction device. The intelligent decision-making module coordinates the physical feedback module during the feedback phase according to the swallowing intention category: if a strong swallowing intention is identified, the swallowing induction device is triggered to spray a preset dose of liquid; if a weak swallowing intention is identified and the participation index is higher than a preset threshold, the electrical stimulation device is simultaneously triggered to provide auxiliary electrical stimulation to the user's pharyngeal muscles.

[0122] In this embodiment, the intelligent decision-making module also monitors fatigue levels to dynamically adjust the duration of rest periods in order to achieve physiological function compensation.

[0123] Fatigue levels are monitored through a combination of EEG and visual feature analysis. After a period of training, users may experience fatigue due to repetitive tasks or prolonged mental concentration. At this point, fatigue indicators are quantified to allow for appropriate adjustments to rest periods. Furthermore, reducing the intensity or effect of stimuli during the decision-making process can alleviate fatigue.

[0124] As an example, methods for determining fatigue indexes include: determining fatigue level S_θ = P_θ / P_β;

[0125] Determine the slow wave ratio: S_slow=(P_θ+P_δ) / (P_α+P_β);

[0126] Eye-closed angle monitoring: S_close_angle = S_open_angle - S_current_angle; where S_open_angle is the eye opening and closing angle in the normal state, and S_current_angle is the opening and closing angle in the current frame.

[0127] The different features are normalized and then weighted and fused to form the fatigue index S_tired:

[0128] S_tired = a*S_θ + b*S_slow + c*S_close_angle; a, b, c are the weight coefficients of each feature.

[0129] In some embodiments, the intelligent decision-making module is also used for performance evaluation.

[0130] Electromyography (EMG) is used to assess the executive function of peripheral muscles, thereby evaluating executive ability, quantifying movement quality, and determining the stage of interactive training. EMG detection is initiated and quantified upon entering the execution stage to reflect the level of executive function.

[0131] The performance intensity index S_strenth: S_strenth = RMS_burst / RMS_base; reflects the muscle contraction intensity of a single swallowing action. A ratio >1 indicates increased intensity; <1 indicates insufficient intensity or fatigue. RMS_burst refers to the root mean square value of the electromyographic signal within the effective time window of the swallowing action, and RMS_base refers to the root mean square value of the electromyographic signal before entering the guidance phase.

[0132] The performance efficiency metric S_effect: S_effect = RMS_raise; reflects the speed at which a muscle goes from activation to peak performance, i.e., neuromuscular performance efficiency. This value should be increased during interactive training. RMS_raise represents the rate of increase of the root mean square (RMS) value of the electromyographic signal during the initial stage of movement (i.e., the slope of the signal envelope).

[0133] If the execution efficiency index is lower than the preset threshold, it indicates that the user is exerting force slowly. The system may judge this as "weak execution" and thus increase the depth of electrical stimulation in the physical feedback module to assist the muscle in quickly completing the contraction.

[0134] In this embodiment, the intelligent decision-making module makes decisions in the task based on the quantification results of the real-time status.

[0135] 1) High engagement, low performance efficiency: When the swallowing intention is less than the set threshold A, but the engagement is greater than the set threshold B, it indicates that the user has good engagement, but the swallowing process is not smooth, and fatigue may be the dominant factor. In this case, appropriate auxiliary stimuli should be added to help the user with interactive training, and the task difficulty should be reduced while the rest time should be increased appropriately. A and B are learnable and adaptive parameters; the preset values ​​after user questionnaire evaluation are used for the first training session.

[0136] 2) High engagement metrics and high execution efficiency metrics: At this point, the user is in a good state, and the difficulty can be gradually increased in the next training task. If the user can maintain a good state for a relatively long time under interactive training with certain parameters, the current parameters can be optimized for model parameters.

[0137] 3) Low engagement metrics, high execution efficiency metrics: At this stage, the user may have good interactive training results and can passively perform swallowing tasks. At this point, their interactive training level can be adjusted to proceed to the next stage of treatment.

[0138] 4) Low participation and low execution efficiency indicators: Users may have resistance. The swallowing interaction training mode and other parameters can be dynamically changed by asking users about their symptoms.

[0139] The embodiment constructs an efficient adaptive perception-feedback closed loop: by evaluating the user's participation, fatigue, and execution indicators in real time, the system can dynamically adjust the guidance difficulty and rest duration of the interaction paradigm. This personalized adjustment mechanism ensures both the effective stimulation intensity of swallowing interaction training and prevents neural fatigue caused by overtraining, significantly shortening the user's swallowing interaction training cycle.

[0140] In some embodiments, the system also includes a safety monitoring submodule for extracting coughing action characteristics from video signals and abnormal impedance fluctuations from electromyographic signals in real time; when a preset aspiration hazard is detected, the safety monitoring submodule immediately shuts down the physical feedback module and issues an alarm.

[0141] In this embodiment, the intelligent decision-making module can preset some state limits or thresholds to assess and decide on the safety status of the interactive training process. Epilepsy monitoring and potential choking hazards during swallowing are used as indicators. Multimodal data is used to extract corresponding features. For example, during choking, there will be abnormal peak changes in the pharynx, and short-term high-frequency coughing symptoms will appear in the visual modality, while the EEG will also reflect negative characteristic signals. At this point, the paradigm task, hardware feedback, and alarm processing should be stopped immediately.

[0142] The implementation provides multiple safety safeguards and quantitative assessments: the built-in safety monitoring submodule, through dual verification of video and electromyography, can identify aspiration risks in milliseconds and immediately cut off physical feedback, ensuring the safety of training in home or unsupervised environments. Simultaneously, the system-generated digital performance report provides clinicians with objective and quantitative assessment data, making swallowing interaction training visual and traceable.

[0143] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, or a tablet computer, or any combination of these devices.

[0144] The swallowing interaction system and method based on multimodal collaborative fusion provided in this application have been described in detail above. Specific examples have been used in this paper to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the concept of this application and should not be construed as a limitation on the scope of protection of this application.

Claims

1. A swallowing interaction system based on multimodal collaborative fusion, characterized in that, include: The interactive task paradigm guidance module is used to construct interactive task paradigms and output personalized guidance content to users during the guidance phase as a source of neural and physiological stimulation to guide users to perform swallowing interactive tasks within a specific time window. The multimodal signal acquisition module is used to acquire EEG signals, EMG signals, and video signals in real time when the user performs swallowing interaction tasks; The multimodal data processing module is used to synchronize and extract features from the acquired signals to obtain EEG feature vectors, EMG feature vectors, and video feature vectors; using cross-modal feature engineering and gating attention mechanisms, the feature vectors of each modality are fused to obtain a fused feature vector; The intent decoding module is used to output the corresponding swallowing intent category based on the fused feature vector; The intelligent decision-making module is used to dynamically adjust the guidance content in the interactive task paradigm guidance module in real time based on the swallowing intention category.

2. The swallowing interaction system based on multimodal collaborative fusion according to claim 1, characterized in that, By utilizing cross-modal feature engineering and gated attention mechanisms, feature vectors from various modalities are fused to construct a fused feature vector, including: By performing temporal alignment and correlation analysis on the power ratio of specific bands in the motor cortex in EEG features, the duration of swallowing activation phase in EMG features, and the movement trajectory of key points in the larynx in video features within a preset swallowing interaction task time window, cross-modal collaborative feature vectors including representation intention-initiation synchronicity, execution-performance synchronicity, and global synergy are generated. The cross-modal collaborative feature vector is input into a gated attention network, which outputs dynamic importance weights for the three modalities. The feature vectors of the three modalities are used to perform feature abstraction for each modality independently through intramodal encoding to obtain the corresponding latent vectors. The latent vectors and dynamic importance weights are weighted and summed to obtain the fused feature vector.

3. The swallowing interaction system based on multimodal collaborative fusion according to claim 2, characterized in that, The intent decoding module is specifically used to concatenate the fused feature vector with the cross-modal collaborative feature vector and input it into the pre-trained classification model. The classification model learns the components representing electromyographic intensity and the components representing the activation intensity of the motor cortex in the concatenated vector and maps them together with a multidimensional adaptive threshold to output the corresponding swallowing intent category.

4. The swallowing interaction system based on multimodal collaborative fusion according to claim 2, characterized in that, When performing feature abstraction, the multimodal data processing module uses fully connected networks or convolutional neural network layers corresponding to each modality as encoders to map heterogeneous EEG feature vectors, EMG feature vectors, and video feature vectors to a unified-dimensional latent space to obtain the corresponding latent vectors.

5. The swallowing interaction system based on multimodal collaborative fusion according to claim 1, characterized in that, The interactive task paradigm also includes a rest phase; The intelligent decision-making module is also used to extract the slow wave ratio in the electroencephalogram (EEG) signal and eye features in the video signal to obtain a fatigue index, which is used to dynamically adjust the duration of the rest phase to achieve physiological function compensation.

6. The swallowing interaction system based on multimodal collaborative fusion according to claim 1, characterized in that, The interactive task paradigm also includes a feedback phase; the intelligent decision-making module is also used to extract the power ratio of specific bands in the EEG signal to obtain an engagement index, which is used to adjust the difficulty of the guidance content. The system also includes a physical feedback module, which includes an electrical stimulation device and a swallowing induction device. The intelligent decision-making module coordinates the physical feedback module during the feedback phase according to the swallowing intention category: if a strong swallowing intention is identified, the swallowing induction device is triggered to spray a preset dose of liquid; if a weak swallowing intention is identified and the participation index is higher than a preset threshold, the electrical stimulation device is simultaneously triggered to provide auxiliary electrical stimulation to the user's pharyngeal muscles.

7. The swallowing interaction system based on multimodal collaborative fusion according to claim 6, characterized in that, The system also includes a safety monitoring submodule, which is used to extract coughing behavior characteristics from the video signal and abnormal impedance fluctuations from the electromyographic signal in real time; when a preset aspiration hazard is detected, the safety monitoring submodule immediately shuts down the physical feedback module and issues an alarm.

8. The swallowing interaction system based on multimodal collaborative fusion according to claim 1, characterized in that, The system also includes a cloud platform for synchronously storing and updating user personalized preference data and historical interaction data; the cloud platform performs big data analysis based on historical interaction status data to iteratively optimize the weight of the gating attention mechanism in the intent decoding module, and, in conjunction with personalized preference data, issues customized interaction schemes.

9. The swallowing interaction system based on multimodal collaborative fusion according to claim 8, characterized in that, When outputting guidance content, the interactive task paradigm guidance module uses a generative AI model combined with user preference data stored on the cloud platform to generate visual animation styles and triggering audio in real time. Before the guidance phase begins, it monitors attention indicators in real time. When the attention indicators exceed a preset threshold for a preset time, it enters the guidance collection phase.

10. A swallowing interaction guidance and feedback method based on multimodal collaborative fusion, using the system described in any one of claims 1-9, characterized in that, Includes the following steps: An interactive task paradigm was constructed, and personalized guidance content was output to users during the guidance phase as a source of neural and physiological stimulation to guide users to perform swallowing interactive tasks within a specific time window. Real-time acquisition of EEG, EMG, and video signals from users while performing swallowing interaction tasks; The acquired signals are synchronized and feature extracted to obtain EEG feature vectors, EMG feature vectors, and video feature vectors; cross-modal feature engineering and gating attention mechanisms are used to fuse the feature vectors of each modality to obtain a fused feature vector; Based on the fused feature vector, the corresponding swallowing intention category is output; Based on the swallowing intention category, the guidance content in the interactive task paradigm guidance module is dynamically adjusted in real time.

Images