Method, device, storage medium and electronic device for recognizing oral action

Abstract

The application relates to a method and device for identifying oral actions, a storage medium and an electronic device, the method comprising: acquiring an electromyogram signal to perform short-time Fourier transform, acquiring electromyogram low-frequency features and electromyogram high-frequency features, constructing an electromyogram feature matrix, acquiring an electroencephalogram signal to perform continuous wavelet transform, acquiring electroencephalogram time-frequency band features, constructing an electroencephalogram feature matrix, mapping the feature matrices into the same feature space, obtaining electroencephalogram transformed features and electromyogram transformed features, performing mean zero and variance normalization operations on the spliced results, and obtaining a standardized feature matrix; based on the standardized feature matrix, a linear mapping matrix and a lightweight self-attention function, an adaptive weighted feature vector is obtained; the distances between the adaptive weighted feature vector and prototype feature vectors corresponding to various oral action categories pre-labeled are calculated, and the oral action category of the adaptive weighted feature vector is determined based on the calculated distances. The communication efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and in particular to a method, apparatus, storage medium, and electronic device for recognizing oral movements. Background Technology

[0002] Language plays a vital role in human social interaction. However, a large number of patients worldwide suffer from impaired or even complete loss of speech function due to vocal cord damage, neurodegenerative diseases (such as amyotrophic lateral sclerosis ALS), stroke, traumatic brain injury, and other reasons. This severely limits their emotional expression and social participation after losing their normal ability to speak, significantly impacting their quality of life.

[0003] Currently, assistive communication methods for patients with severe speech disorders mainly include sign language, writing boards, eye-tracking systems, and electromyography (EMG)-based interfaces. While sign language and writing boards can facilitate communication to some extent, they place high demands on the patient's physical function, resulting in slow communication speed and limited expression. Due to these high physical function requirements, misrecognition is common, making them unsuitable for communication in complex daily situations, leading to low communication efficiency and accuracy. Eye-tracking systems monitor gaze direction for information input, but the equipment is expensive, cumbersome to operate, has a slow response time, and causes eye fatigue with prolonged use, resulting in a poor user experience. Eye fatigue can also lead to information input errors, affecting communication efficiency and accuracy. EMG interfaces assist communication by detecting electrical signals from facial or neck muscles. While they are highly responsive, muscle condition is easily affected by fatigue, sweat, and electrode displacement, leading to insufficient stability and accuracy in assistive communication, resulting in low communication efficiency. Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, storage medium and electronic device for recognizing oral movements.

[0005] Specifically, the present invention is achieved through the following technical solution:

[0006] According to a first aspect of the present invention, a method for recognizing oral movements is provided, the method comprising:

[0007] Electromyography (EMG) and electroencephalography (EEG) signals of the target object are acquired through a pre-set brain-computer interface acquisition device. Continuous wavelet transform is performed on the EEG signals to obtain EEG time-frequency band features. Short-time Fourier transform is performed on the EMG signals to obtain EMG low-frequency and high-frequency features characterizing muscle activity.

[0008] Based on the EEG time-frequency band features, an EEG feature matrix is ​​constructed, and based on the EMG low-frequency and high-frequency features, an EMG feature matrix is ​​constructed. Using a cross-modal feature mapping mechanism, the EEG feature matrix and the EMG feature matrix are mapped to the same feature space to obtain EEG transformation features and EMG transformation features.

[0009] Based on a pre-set feature-level fusion strategy, the EEG transformation features and EMG transformation features are spliced ​​together to obtain a preliminary fusion feature matrix. The preliminary fusion feature matrix is ​​then subjected to mean zeroing and variance normalization operations to obtain a standardized feature matrix.

[0010] Based on the standardized feature matrix and the pre-set linear mapping matrix, query vector features, key vector features and value vector features are obtained. Based on the query vector features, key vector features and value vector features and the pre-set lightweight self-attention function, an adaptive weighted feature vector is obtained.

[0011] The distance between the adaptive weighted feature vector and the prototype feature vector corresponding to each oral action category pre-labeled by the deep neural network encoder is calculated, and the oral action category of the adaptive weighted feature vector is determined based on the calculated distance.

[0012] The method for recognizing oral movements in this technical solution acquires electromyography (EMG) and electroencephalography (EEG) signals of the target object through a pre-set brain-computer interface acquisition device. It performs continuous wavelet transform on the EEG signals to obtain EEG time-frequency band features, and performs short-time Fourier transform on the EMG signals to obtain low-frequency and high-frequency EMG features characterizing muscle activity. Based on the EEG time-frequency band features, an EEG feature matrix is ​​constructed, and based on the low-frequency and high-frequency EMG features, an EMG feature matrix is ​​constructed. Using a cross-modal feature mapping mechanism, the EEG feature matrix and the EMG feature matrix are mapped to the same feature space to obtain EEG transformation features and EMG transformation features. Based on pre-set features... A multi-modal fusion strategy is employed, concatenating the EEG and EMG transformation features to obtain a preliminary fusion feature matrix. This matrix is ​​then normalized by zeroing the mean and normalizing the variance to obtain a standardized feature matrix. Based on this standardized feature matrix and a pre-set linear mapping matrix, query vector features, key vector features, and value vector features are obtained. Based on these features and a pre-set lightweight self-attention function, an adaptive weighted feature vector is acquired. The distance between this adaptive weighted feature vector and the prototype feature vector corresponding to each oral movement category pre-labeled by the deep neural network encoder is calculated. The oral movement category of the adaptive weighted feature vector is determined based on the calculated distance. Thus, through multimodal feature extraction and deep fusion of EEG and EMG signals, highly discriminative fusion features are obtained. Oral movement recognition is performed based on the deep time-frequency features (adaptive weighted feature vector) and prototype feature vectors obtained from the fusion features, improving the ability to distinguish complex movements and effectively enhancing assisted communication efficiency.

[0013] According to a second aspect of the present invention, an apparatus for recognizing oral cavity movements is provided, the apparatus comprising:

[0014] The brain-computer interface signal acquisition and processing module is used to acquire electromyography (EMG) signals and electroencephalography (EEG) signals of the target object, perform continuous wavelet transform on the EEG signals to obtain EEG time-frequency band features, and perform short-time Fourier transform on the EMG signals to obtain EMG low-frequency features and EMG high-frequency features characterizing muscle activity.

[0015] The feature transformation module is used to construct an EEG feature matrix based on the EEG time-frequency band features, and to construct an EMG feature matrix based on the EMG low-frequency features and EMG high-frequency features. Using a cross-modal feature mapping mechanism, the EEG feature matrix and the EMG feature matrix are mapped to the same feature space respectively to obtain EEG transformation features and EMG transformation features.

[0016] The feature splicing module is used to splice the EEG transformation features and EMG transformation features based on a pre-set feature-level fusion strategy to obtain a preliminary fusion feature matrix. The preliminary fusion feature matrix is ​​then subjected to mean zeroing and variance normalization operations to obtain a standardized feature matrix.

[0017] The feature fusion module is used to obtain query vector features, key vector features and value vector features based on the standardized feature matrix and the pre-set linear mapping matrix, and to obtain an adaptive weighted feature vector based on the query vector features, key vector features and value vector features and the pre-set lightweight self-attention function.

[0018] The category recognition module is used to calculate the distance between the adaptive weighted feature vector and the prototype feature vector corresponding to each oral action category pre-labeled by the deep neural network encoder, and to determine the oral action category of the adaptive weighted feature vector based on the calculated distance.

[0019] According to a third aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for recognizing oral movements in any possible implementation of the first aspect.

[0020] According to a fourth aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for recognizing oral movements in any possible implementation of the first aspect. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a method for recognizing oral movements provided in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of feature fusion in a method for recognizing oral movements provided in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of signal processing in a method for recognizing oral movements provided in an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a device for recognizing oral movements provided in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Among the related technologies, the auxiliary communication methods for patients with severe speech disorders based on sign language, writing boards, eye-tracking systems, and electromyography interfaces are all prone to misidentification, have low stability and accuracy, and poor efficiency.

[0030] With the development of brain-computer interface (BCI) technology, electroencephalography (EEG), as a non-invasive, portable technique capable of recording brain neural activity in real time, has attracted widespread attention. EEG signals can capture high-level information such as motor intentions and cognitive activities, theoretically offering possibilities for silent speech recognition. However, EEG signals themselves are characterized by low signal-to-noise ratios, susceptibility to electromagnetic interference and physiological artifacts, and single-channel EEG signals are insufficient to accurately reflect complex oral motor patterns. Furthermore, EEG signals lack the spatial resolution to finely distinguish the neural activity corresponding to micro-movements of the tongue and lips, thus limiting the accuracy of oral motor recognition based on EEG signals.

[0031] Electromyography (EMG) signals can directly reflect the electrical activity caused by muscle contraction, especially in the speech-related muscle groups such as the tongue, cheek, and masseter muscles, where they can highly sensitively capture subtle changes in movement. EMG signals have high local movement discrimination, making them suitable for detecting details of oral motor movements. However, EMG signals only reflect the execution of the end-effector's movements and not the intention behind the movement. They are easily affected by factors such as muscle fatigue, changes in skin resistance, and electrode position drift, leading to decreased recognition stability. Furthermore, in scenarios involving complex or continuous movements, EMG signals are prone to signal aliasing, increasing the difficulty of decoding.

[0032] EEG signals provide high-level cognitive intent information, while EMG signals provide details of end-effector execution. These two signals complement each other at the physiological level; therefore, fusing EEG and EMG signals is an important direction for improving the performance of silent oral motor action recognition. However, current methods for fusing EEG and EMG signals to improve oral motor action recognition have many limitations. For example, related technologies cascade the two signals, failing to fully exploit the spatiotemporal correlation features between them, resulting in limited fusion effects. Furthermore, fusion methods rely on large-scale training data, making them unsuitable for clinical applications with limited data and significant individual differences. In addition, the lack of lightweight model structure design and few-sample adaptive learning mechanisms leads to complex deployment, high training costs, and poor user experience, hindering practical application.

[0033] This embodiment proposes an innovative oral action recognition method that, under limited sample conditions, fully utilizes the complementary characteristics of EEG and EMG signals. Through multimodal feature extraction and deep fusion of EEG and EMG signals, highly discriminative fusion features are obtained. A deep learning algorithm is employed to efficiently model the deep time-frequency features obtained from the fusion features, enhancing the ability to distinguish complex actions. This allows for rapid personalized adaptation through incremental learning with few samples, reducing reliance on large-scale training data. This method meets the needs of patients with severe speech disorders for natural, stable, and efficient silent communication in daily life, thereby improving assisted communication efficiency and addressing the limitations of existing oral action recognition methods in terms of accuracy, efficiency, feature modeling ability, and personalized adaptability.

[0034] See Figure 1 This invention provides a method for recognizing oral movements, which may include the following steps:

[0035] S101. Acquire electromyography (EMG) signals and electroencephalography (EEG) signals of the target object through a pre-set brain-computer interface acquisition device, perform continuous wavelet transform on the EEG signals to obtain EEG time-frequency band features, and perform short-time Fourier transform on the EMG signals to obtain EMG low-frequency features and EMG high-frequency features characterizing muscle activity.

[0036] In this embodiment, the electroencephalogram (EEG) signal is the electrical signal of the target object's brain, and the electromyography (EMG) signal is the electrical signal of the target object's oral muscles. The EEG and EMG signals can be synchronously acquired through a brain-computer interface system. As an optional embodiment, the target object includes, but is not limited to: patients with vocal cord injury, patients with neurodegenerative diseases (such as amyotrophic lateral sclerosis (ALS), stroke patients, and patients with traumatic brain injury.

[0037] In this embodiment, as an optional implementation, a surface electromyography (sEMG) signal acquisition system including a dry electrode surface electromyography sensor is used to acquire electromyography signals, and the acquisition of electromyography signals of the target object includes:

[0038] Electromyography (EMG) signals are collected using dry electrode surface electromyography sensors deployed on pre-defined facial muscle groups involved in oral cavity movements.

[0039] In this embodiment, the dry electrode surface electromyography sensor has good signal stability and reusability, and is compatible with a wireless data transmission module, making it suitable for integration into wearable devices.

[0040] In this embodiment, as an optional implementation, the preset facial muscle groups include, but are not limited to: the zygomaticus major muscle group, the masseter muscle group, the mental muscle group, and the orbicularis oris muscle group. The electrode patch of the dry electrode surface electromyography sensor maintains good adhesion to the skin, and conductive paste or conductive gel is used to enhance conductivity and reduce impedance.

[0041] In this embodiment, as an optional embodiment, the surface electromyography (EMG) sensors on the dry electrode are arranged along the direction of muscle fibers, with a spacing of no less than 2 cm between each channel to avoid crosstalk; the synchronous acquisition reference ground electrode of the surface EMG sensor on the dry electrode is attached to the neck or earlobe as a potential reference point; the sampling frequency is set to ≥1000Hz to ensure that high-frequency EMG signal components are captured; wireless transmission is performed using a Bluetooth or WiFi module.

[0042] In this embodiment, the electroencephalogram (EEG) and electromyogram (EMG) signals involved in oral action recognition are preprocessed. As an optional embodiment, the signals are combined with their respective physiological characteristics to perform frequency domain decomposition and feature extraction, so as to extract action-related features with strong discriminativeness to the greatest extent.

[0043] In this embodiment, for the electroencephalogram (EEG) signal, the Continuous Wavelet Transform (CWT) is used to perform multi-scale time-frequency analysis on the acquired raw EEG signal in order to capture the dynamic features of local changes at different time scales or time steps.

[0044] In this embodiment, as an optional implementation, continuous wavelet transform is performed on the electroencephalogram (EEG) signal to obtain EEG time-frequency band features, including:

[0045] The EEG signal was subjected to continuous wavelet transform to obtain the EEG time-frequency signal;

[0046] Based on the frequency scale corresponding to the pre-set extraction band, feature extraction is performed on the EEG time-frequency signal to obtain the EEG time-frequency band features corresponding to each extraction band.

[0047] In this embodiment, the acquired time-series-based raw EEG signal is denoted as x(t). After CWT transformation, the resulting EEG time-frequency signal Seeg(t,f) is defined as follows:

[0048]

[0049] in, It is the scaling factor, which is the conjugate form of the wavelet function of f, where f is the frequency scale.

[0050] In this embodiment, as an optional implementation, the pre-set extraction bands include, but are not limited to: the delta band (corresponding frequency scale: 0.5–4 Hz), the theta band (corresponding frequency scale: 4–8 Hz), the alpha band (corresponding frequency scale: 8–13 Hz), the beta band (corresponding frequency scale: 13–30 Hz), and the gamma band (corresponding frequency scale: 30–100 Hz). By traversing different frequency scales f of the EEG time-frequency signal, key neural activity frequency band features such as the delta band (0.5–4 Hz), the theta band (4–8 Hz), the alpha band (8–13 Hz), the beta band (13–30 Hz), and the gamma band (30–100 Hz) are extracted respectively. As another optional implementation, a bandpass filter bank can also be used to decompose the EEG signal to obtain the EEG time-frequency band features.

[0051] In this embodiment, a Short-Time Fourier Transform (STFT) is performed on the EMG signal to capture the local features of muscle activity in the time-frequency domain. As an optional embodiment, given the original EMG signal x(t), the STFT transform yields the electromyographic time-frequency signal Semg(t). This can be expressed as:

[0052]

[0053] Where w(·) is the sliding window function and ω is the angular frequency.

[0054] In this embodiment, by analyzing and extracting the low-frequency (0–25 Hz) and high-frequency (>25 Hz) components of the electromyography (EMG) time-frequency signal, we can obtain the slow movement initiation features based on low frequency, i.e., the low-frequency EMG features, and the fast and subtle movement execution features based on high frequency, i.e., the high-frequency EMG features, thereby providing a basis for subsequent fine modeling of EMG signals.

[0055] S102. Based on the EEG time-frequency band features, construct an EEG feature matrix, and based on the EMG low-frequency features and EMG high-frequency features, construct an EMG feature matrix. Using a cross-modal feature mapping mechanism, map the EEG feature matrix and the EMG feature matrix to the same feature space to obtain EEG transformation features and EMG transformation features.

[0056] In this embodiment, after the EEG signal undergoes continuous wavelet transform or bandpass filter bank decomposition, three typical EEG time-frequency band features (δ: 0.5–4 Hz, θ: 4–8 Hz, α: 8–13 Hz) are extracted at each time step (time window). Based on the EEG time-frequency band features of the three consecutive time steps, an EEG feature matrix is ​​constructed. As an optional embodiment, the constructed EEG feature matrix is ​​as follows:

[0057]

[0058] Among them, f δ (t) represents the feature value extracted at time t in the δ band, i.e., the EEG time-frequency band feature corresponding to the δ band, including but not limited to the energy or amplitude mean. As an optional embodiment, if the number of selected bands (feature dimension) is 3 and the number of time steps is 3, then the dimension of the EEG feature matrix is ​​3×3. In this embodiment, the construction of the electromyography feature matrix is ​​similar to the construction of the EEG feature matrix.

[0059] In this embodiment, time-frequency decomposition, feature extraction, and cross-modal statistical modeling provide a high-quality, structured feature foundation for subsequent unified feature space construction and deep fusion learning.

[0060] In this embodiment, after completing the preliminary feature extraction and cross-modal collaborative modeling of electroencephalogram (EEG) and electromyogram (EMG) signals, in order to further improve the discrimination ability in silent oral action recognition, a method based on fusion feature construction and unified spatial representation is used to achieve deep coupling and spatial alignment of different modal information, thereby enhancing the separability of action categories in the feature space.

[0061] In this embodiment, as an optional implementation, a cross-modal feature mapping mechanism is used to map the feature matrix and the electromyography feature matrix to the same feature space, respectively, to obtain EEG transformation features and electromyography transformation features, including:

[0062] The EEG feature matrix is ​​mapped using a pre-set first mapping function to obtain EEG transformation features;

[0063] The electromyography feature matrix is ​​mapped using a pre-set second mapping function to obtain electromyography transformation features. The first mapping function and the second mapping function are mapping functions in the same feature space.

[0064] In this embodiment, the extracted EEG feature matrix Xeeg and EMG feature matrix Xemg are mapped to the same feature space through a cross-modal feature mapping mechanism, facilitating subsequent fusion and unified processing. As an optional embodiment, a first mapping function zeeg(·) and a second mapping function zemg(·) are defined to perform linear transformations on the EEG feature matrix Xeeg and the EMG feature matrix Xemg, respectively. The EEG transformation features and EMG transformation features after linear transformation are expressed as follows:

[0065]

[0066] Among them, W eeg W emg Let b be the feature transformation matrix of each modality (EEG signal and EMG signal). eeg b emg For the corresponding bias term, i.e., W eeg W is the feature transformation matrix corresponding to the electroencephalogram (EEG feature matrix). emg Let b be the feature transformation matrix corresponding to the electromyography signal. eeg b is the bias term corresponding to the EEG signal. emg This is the bias term corresponding to the electromyography signal.

[0067] In this embodiment, the feature transformation matrix W corresponding to the EEG feature matrix eeg This can be learned in a data-driven manner through the embedding layer in a deep learning network. As an optional embodiment, this is achieved by setting the learning parameter matrix and bias terms of the neural network. For example, a set of learnable parameter matrices can be set in the embedding layer of the neural network. and bias terms Based on the learning parameter matrix and bias terms, a linear transformation is performed on the EEG feature matrix at the embedding layer to obtain the shared space feature matrix:

[0068]

[0069] in, d′ is the shared space feature matrix after projection (linear transformation), and d′ is the pre-set uniform embedding dimension.

[0070] In this embodiment, the weights (learnable parameter matrix) W of the embedding layer eeg and bias term b eegBy jointly optimizing the shared space feature matrix obtained through linear transformation with the loss function of the classification or recognition task during the entire deep network training process, the final feature transformation matrix is ​​ensured to possess the following characteristics: 1. Preserving the discriminative nature of the original EEG features; 2. Aligning the embedding vectors of EMG modalities to achieve consistency across modal spaces; 3. Supporting collaborative modeling of subsequent structures such as attention fusion, contrastive learning, or two-stream networks. This effectively avoids the limitations of manual design or fixed projection, ensuring that the feature transformation matrix W... eeg It possesses task relevance, data adaptability, and model collaborative optimization capabilities, making it a key module for cross-modal feature alignment and fusion.

[0071] In this embodiment, linear transformation using a mapping function can ensure that features from different sources tend to be consistent in terms of dimensions, scale, and statistical properties, thus creating conditions for subsequent fusion operations.

[0072] S103. Based on a pre-set feature-level fusion strategy, the EEG transformation features and EMG transformation features are spliced ​​together to obtain a preliminary fusion feature matrix. The preliminary fusion feature matrix is ​​then subjected to mean zeroing and variance normalization operations to obtain a standardized feature matrix.

[0073] In this embodiment, after feature mapping is completed, a feature-level fusion strategy is adopted to integrate the EEG change features Z. eeg With electromyographic transformation characteristics Z emg By concatenating the features along the feature dimension, a preliminary fused feature matrix Z is obtained. fusion :

[0074]

[0075] Where [·;·] represents a concatenation operation along the feature dimension.

[0076] In this embodiment, feature-level fusion can preserve the unique descriptive capabilities of both EEG and EMG signals, while also introducing a joint expression of both signals. This allows the initial fusion feature matrix to simultaneously capture action information at both the brain intention and muscle execution levels.

[0077] Figure 2 This is a schematic diagram of feature fusion in a method for recognizing oral movements provided in an embodiment of the present invention. Figure 2 As shown, the dimension of the EEG feature matrix is ​​32×10, the corresponding dimension of the feature transformation matrix is ​​64×32, and the dimension of the EEG transformation feature after linear transformation is 64; the dimension of the EMG feature matrix is ​​32×8, the corresponding dimension of the feature transformation matrix is ​​64×32, and the dimension of the EEG transformation feature after linear transformation is 64.

[0078] In this embodiment, as an optional embodiment, the method further includes:

[0079] Based on a pre-set covariance matrix construction strategy, the EEG feature matrix, and the EMG feature matrix, a joint covariance matrix feature is obtained. The joint covariance matrix feature, EEG transformation feature, and EMG transformation feature are then concatenated to obtain the preliminary fusion feature matrix.

[0080] In this embodiment, after extracting the EEG time-frequency band features from the EEG signal and the EMG low-frequency and high-frequency EMG features from the EMG signal, cross-modal statistical properties can be used to model and uncover the deep dependency relationship between the two signals. As an optional embodiment, let the EEG feature matrix constructed based on the EEG time-frequency band features be: The EMG feature matrix constructed based on low-frequency and high-frequency electromyographic features is as follows: , where d e d m Let T be the feature dimension and T be the number of time steps. Define the feature C of the joint covariance matrix. joint as follows:

[0081]

[0082] in, This represents the mean eigenvector of each modality. For example, for the EEG feature matrix, This represents the mean of each eigenvector in the EEG feature matrix, with the superscript indicating the transpose.

[0083] In this embodiment, by obtaining the joint covariance matrix features, the statistical correlation between the EEG feature matrix and the EMG feature matrix can be obtained, especially the joint change trend of the two modes during the dynamic process; at the same time, the degree of coupling between cross modes can be quantified, and the joint features of coordinated change can be extracted.

[0084] In this embodiment, by utilizing the joint covariance matrix feature, it is possible not only to describe the changing characteristics of the internal structure of EEG and EMG signals, but also to effectively capture the interactive coupling modes between different modes (two types of signals).

[0085] In this embodiment, to further reveal the synchronicity between cross-modal features and improve the accuracy of subsequent identification, as another optional embodiment, the method further includes:

[0086] Based on a pre-set correlation matrix construction strategy, the EEG feature matrix, and the EMG feature matrix, cross-correlation matrix features are obtained, and the cross-correlation matrix features, EEG transformation features, and EMG transformation features are spliced ​​together to obtain the preliminary fusion feature matrix.

[0087] In this embodiment, as an optional implementation, the cross-correlation matrix feature R cross The definition is as follows:

[0088]

[0089] Where cov(·,·) is the covariance function, σ(·) is the standard deviation, and X eeg (i,:) represents the EEG feature vector (EEG time-frequency band feature) extracted at the i-th time step (window), X emg (j,:) represents the EMG feature vector extracted at the j-th time step. As an optional embodiment, i≠j, modal synchronization at different times can be obtained.

[0090] In this embodiment, the cross-correlation matrix feature possesses temporal extensibility, making it suitable for analyzing the coupling relationship between non-strictly time-aligned EMG and EEG signals. It can measure the statistical correlation between EEG and EMG features at different time steps and reveal the dynamic coupling pattern between brain commands and muscle responses during oral movements. For example, if the EEG signal at time t is strongly correlated with the EMG signal at time t+1, it reflects the sequential transmission order of neuromuscular activation and can be used to identify the temporal dependence of different movement patterns.

[0091] In this embodiment, the dynamic synergistic relationship between EEG and EMG signals during the execution of an action can be quantified by using the cross-correlation matrix features, thereby further enhancing the ability to model the brain-electromyography synergistic mechanism during action pattern recognition.

[0092] In this embodiment, to further improve the stability and consistency of the initial fused feature matrix, a standardization process is introduced in the fused feature space, that is, mean normalization and variance normalization are performed on the initial fused feature matrix (each fused feature vector). As an optional embodiment, the standardized feature matrix after performing mean normalization and variance normalization is expressed as follows:

[0093]

[0094] Where μ(·) represents the mean, σ(·) represents the standard deviation, and Z0... fusion (i,:) represents the initial fusion feature matrix Z. fusion The fused feature vector corresponding to the i-th row (i-th time step).

[0095] In this embodiment, by performing mean-zeroing and variance-normalization operations (standardization operations), the distribution shift of different modal features due to different dimensions can be effectively suppressed, further improving the gradient stability and optimization convergence speed in the deep modeling process, while enhancing the discriminativeness of action categories in the feature space.

[0096] In this embodiment, by using a fusion feature construction and unified representation method, the natural alignment and seamless fusion of brain nerve signals and muscle movement signals in the same feature space can be achieved, laying the foundation for subsequent complex temporal relationship modeling and high-precision oral action recognition.

[0097] S104. Based on the standardized feature matrix and the pre-set linear mapping matrix, obtain query vector features, key vector features and value vector features. Based on the query vector features, key vector features and value vector features and the pre-set lightweight self-attention function, obtain an adaptive weighted feature vector.

[0098] In this embodiment, local convolutional features are obtained based on the standardized feature matrix, and convolutional features are obtained based on the local convolutional features. Different pre-set linear mapping matrices are used to map the convolutional features to obtain query vector features, key vector features, and value vector features.

[0099] In this embodiment, after the fusion features are constructed and a preliminary unified representation is achieved, in order to further improve the modeling ability of the complex dynamic process of silent oral movements, this embodiment proposes a deep feature modeling mechanism that combines multi-scale temporal convolution with a lightweight Transformer structure. This mechanism aims to simultaneously capture the short-term dynamic patterns of local action features and the long-term dependencies in the overall action sequence, thereby comprehensively improving the expressive power and discriminative performance of the action recognition system.

[0100] After completing the cross-modal feature concatenation, this embodiment feeds the fused features into a feature-level fusion layer, which is located after the encoder backbone and before the PrototypeClassifier. The fusion layer first calculates the correlation weights between the EEG and EMG channels using multi-head self-attention, and then performs channel weighting to generate weighted fused features. To avoid parameter inflation, a lightweight attention factor is introduced, which performs learnable scaling only on one-quarter of the channels, while the remaining channels share weights, thereby reducing computational cost and mitigating the risk of overfitting.

[0101] In the training process, a prototype network-based meta-learning framework is adopted: first, the labeled support set is fed into the encoder-fusion layer-attention layer to calculate the prototype vector for each oral action category; then, the query sample is mapped to the same feature space, and the Euclidean distance with the prototype vector is calculated and the category probability is obtained through softmax. To enhance robustness to small samples, prototype classification loss and contrastive loss are introduced into the loss function (positive sample pairs are brought closer, negative sample pairs are moved further apart), and the weight coefficients are dynamically adjusted every 50 epochs to achieve fast convergence and robust generalization.

[0102] In this embodiment, the fused standardized feature matrix is... Multi-Scale 1D Convolution is introduced for local modeling. As an optional embodiment, by setting convolution kernels with different receptive field sizes in parallel, local dynamic changes (local convolution features) at different time scales during oral cavity movements can be perceived simultaneously.

[0103] In this embodiment, the receptive field of the convolution kernel is set to 2K+1, where K takes the values ​​1, 2, and 3. Local convolution features are obtained using the following formula:

[0104]

[0105] Among them, W k，i b represents the kernel weights corresponding to the k-th convolution scale. k σ is the bias term, and σ(·) is the activation function.

[0106] In this embodiment, multi-scale parallel convolution can effectively cover rapid micro-movements in oral cavity movements, such as short tongue tip movements, as well as longer continuous movements, such as slow mouth opening and closing movements, thereby enhancing the diversity and integrity of local dynamic change features.

[0107] In this embodiment, after extracting local convolutional features, a lightweight self-attention mechanism is further introduced. This mechanism can directly capture the interaction between any two time steps without relying on sequence recursion, thus offering higher parallelism and modeling freedom. As an optional embodiment, let the convolutional output features (convolutional features formed based on local convolutional features) be... By using different linear mapping matrices, the convolution output features are mapped to obtain the query vector Q, key vector K, and value vector V:

[0108]

[0109] Among them, W Q W K W V A pre-set trainable mapping matrix.

[0110] In this embodiment, the local convolution feature H k (t) are the atomic units or local codes that constitute the convolution output feature H, that is:

[0111] H=[H1(t),H2(t),…,H k (t)].

[0112] In this embodiment, the feature vectors of all modalities are fused and convolutioned to generate a full sequence representation (convolution output feature) H. As an optional embodiment, the convolution output feature is a semantically enhanced version obtained by extracting, transforming and fusing each local convolution feature.

[0113] In this embodiment, as an optional implementation, attention weights are calculated by scaling the dot product, and an adaptive weighted feature vector is obtained:

[0114]

[0115] Where, d k The dimension of the key vector is used to prevent the gradient from vanishing due to excessively large dot product values. softmax() is a lightweight self-attention function.

[0116] In this embodiment, the lightweight self-attention function based on the self-attention mechanism allows for the autonomous and selective focusing on key action segments during the encoding process, based on the correlation of features at each time step in the action sequence, which greatly improves the accuracy and robustness of complex action pattern recognition.

[0117] S105. Calculate the distance between the adaptive weighted feature vector and the prototype feature vector corresponding to each oral action category pre-labeled by the deep neural network encoder, and determine the oral action category of the adaptive weighted feature vector based on the calculated distance.

[0118] In this embodiment, as an optional embodiment, after obtaining the adaptive weighted feature vector and before calculating the distance between the adaptive weighted feature vector and the prototype feature vector corresponding to each pre-labeled oral action category, the method further includes:

[0119] Based on a pre-set temporal attention function, the adaptive weighted features are transformed to obtain temporal attention features. Based on a pre-set frequency domain attention function, the adaptive weighted features are transformed to obtain frequency domain attention features. The temporal attention features and frequency domain attention features are concatenated to replace the adaptive weighted feature vector used to calculate the distance.

[0120] Figure 3 This is a schematic diagram of signal processing in a method for recognizing oral movements provided in an embodiment of the present invention. Figure 3 As shown, feature extraction, feature concatenation, lightweight Transformer processing, and time-frequency attention processing are performed on the collected electromyography (EMG) and electroencephalography (EEG) signals to obtain the output vector (adaptive weighted feature vector).

[0121] In this embodiment, a time-frequency attention function is used to transform the adaptive weighted features. This time-frequency attention function includes both a time-domain attention function and a frequency-domain attention function. Taking the frequency domain as an example, by introducing a frequency-aware attention function into the frequency feature domain for transformation, the main frequency components are extracted to obtain frequency-domain attention features. Based on the time-frequency attention function, it is possible to consider changes in both time and frequency domain features during action execution, thereby forming a more complete and robust fusion feature representation.

[0122] In this embodiment, the temporal attention feature Z time With frequency domain attention feature Z freq Perform feature-level concatenation and fusion to obtain the concatenated features, represented as follows:

[0123]

[0124] Among them, [Z time ;Z freq ] indicates characteristic-level concatenation, W f b f To fuse the linear transformation parameters.

[0125] In this embodiment, based on Attention(Q, K, V), features in the time dimension are obtained to obtain temporal attention features, and features in the frequency dimension are obtained to obtain frequency attention features.

[0126] In this embodiment, temporal attention features characterize the importance distribution of action signals in the time dynamic change dimension (temporal attention), and frequency domain attention features characterize the salience features of action signals in frequency response and oscillation patterns (frequency domain attention). Both are high-order abstract semantic features, which are transformed into a final fused expression (feature splicing) through linear transformation. This is uniformly mapped into a deep fusion space, thereby achieving comprehensive modeling across scales and semantic layers to construct the final classification decision features.

[0127] In this embodiment, feature adaptation is performed based on a prototypical network. As an optional embodiment, the prototypical feature vector is a pre-acquired set of labeled samples, let the set of labeled samples be... , where x n Denotes the sample, y n Representing oral action categories, for each oral action category c, the prototype feature vector (Prototype) p c Defined as the mean of all sample features for this oral action category:

[0128]

[0129] Among them, S c f is a subset of all samples under oral action category c. θ (·) indicates a pre-set deep feature encoder.

[0130] In this embodiment, the oral cavity action category is determined based on typical oral articulation patterns, including but not limited to:

[0131] 1. Lip-closing gestures: such as m, b, p;

[0132] 2. Tongue tip protrusion type: such as t, d;

[0133] 3. Tongue elevation type: such as k, g;

[0134] 4. Gingival contact type: such as S, Z;

[0135] 5. Lip-teeth contact type: such as f, v;

[0136] 6. Open-mouth sounds: such as a, e, o;

[0137] 7. Special configurations: such as ʃ etc.

[0138] In this embodiment, the classification decision based on the nearest neighbor principle is achieved by calculating the distance between the feature vector of the sample to be identified (adaptive weighted feature vector) and the prototype feature vector corresponding to each oral action category, thereby determining the oral action category of the adaptive weighted feature vector. As an optional embodiment, the oral action category of the feature vector of the sample to be identified is determined using the following formula:

[0139]

[0140] Where d(·,·) represents the Euclidean distance function.

[0141] In this embodiment, to improve the robustness of decision-making during the inference stage, a confidence estimation mechanism is further introduced. Let d be the minimum distance between the obtained predicted categories. min The reliability of action recognition is evaluated based on the following confidence score function:

[0142]

[0143] Where α>0 is a hyperparameter for adjusting confidence sensitivity.

[0144] In this embodiment, a confidence value close to 1 indicates high reliability and a reliable prediction category. When the confidence value is low (e.g., below a preset confidence threshold), a rejection mechanism can be triggered to avoid the risk of misoperation caused by low-confidence prediction.

[0145] In this embodiment, the complementary characteristics of EEG and EMG signals are fully utilized. Through multimodal feature extraction and deep fusion of EEG and EMG signals, the needs of patients with severe speech disorders for natural, stable and efficient silent communication in daily life can be met under very limited sample conditions, effectively improving the efficiency of assisted communication.

[0146] In this embodiment, as an optional embodiment, the method further includes:

[0147] The system queries a pre-set mapping dictionary of oral action categories and communication instructions to obtain the corresponding communication instructions for the specific oral action category in order to communicate with the target object.

[0148] In this embodiment, the recognition results are converted into patient communication commands through a preset mapping dictionary. As an optional embodiment, the mapping dictionary may include, for example, oral action category A corresponding to the communication command "yes", oral action category B corresponding to the communication command "no", oral action category C corresponding to the communication command "pain", etc.

[0149] In this embodiment, as an optional implementation, the prototype feature vectors corresponding to each pre-labeled oral action category are obtained, including:

[0150] A11. Acquire electromyography (EMG) and electroencephalography (EEG) sample signals of each sample subject based on oral movements. Based on one or more interactions with each sample subject, determine the oral movement category corresponding to each EMG and EEG sample signal. Label the EMG and EEG sample signals. The sample subjects include: patients with vocal cord injury, patients with neurodegenerative diseases, stroke patients, and patients with traumatic brain injury.

[0151] In this embodiment, EEG sample signals and EMG sample signals are acquired simultaneously, and preprocessing of the EEG sample signals and EMG sample signals can be performed, such as noise reduction and time alignment.

[0152] A12. Construct sample pairs based on the electromyography sample signals and electroencephalography sample signals, perform continuous wavelet transform on the electroencephalography sample signals in the sample pairs to obtain the time-frequency band sample features of the electroencephalography, and perform short-time Fourier transform on the electromyography sample signals in the sample pairs to obtain the low-frequency sample features and high-frequency sample features of electromyography that characterize muscle activity.

[0153] A13. Based on the EEG time-frequency band sample features, construct an EEG sample feature matrix; and based on the EMG low-frequency sample features and EMG high-frequency sample features, construct an EMG sample feature matrix. Map the EEG sample feature matrix and the EMG sample feature matrix to the same feature space to obtain EEG transformation sample features and EMG transformation sample features.

[0154] A14. Based on the feature-level fusion strategy, the EEG transformation sample features and EMG transformation sample features are spliced ​​together to obtain a preliminary fusion sample feature matrix. The preliminary fusion sample feature matrix is ​​then subjected to mean zeroing and variance normalization operations to obtain a standardized sample feature sequence.

[0155] A15. Based on the standardized sample feature matrix and the linear mapping matrix, query vector sample features, key vector sample features and value vector sample features are obtained. Based on the query vector sample features, key vector sample features and value vector sample features and the lightweight self-attention function, prototype feature vector is obtained, and the deep neural network encoder including the correspondence between the determined oral action category and the prototype feature vector is constructed.

[0156] In this embodiment, as an optional implementation, the prototype feature vector is obtained based on the query vector sample features, key vector sample features, value vector sample features, and the lightweight self-attention function, including:

[0157] Based on the query vector sample features, key vector sample features, and value vector sample features, as well as the lightweight self-attention function, an adaptive weighted sample feature vector is obtained, which is a prototype feature vector.

[0158] In this embodiment, as another optional embodiment, based on the query vector sample features, key vector sample features, and value vector sample features, as well as the lightweight self-attention function, a prototype feature vector is obtained, including:

[0159] Based on the query vector sample features, key vector sample features, and value vector sample features, as well as the lightweight self-attention function, an adaptive weighted sample feature vector is obtained.

[0160] Based on a pre-set temporal attention function, the adaptive weighted sample features are transformed to obtain temporal attention sample features. Based on a pre-set frequency domain attention function, the adaptive weighted sample features are transformed to obtain frequency domain attention sample features. The temporal attention sample features and frequency domain attention sample features are concatenated to replace the adaptive weighted sample feature vector used for distance calculation.

[0161] In this embodiment, multi-dimensional semantic features are extracted using temporal and frequency domain attention mechanisms. Through deep feature fusion, the comprehensive representation capabilities of the fused features (concatenated features) in multiple dimensions, such as temporal dynamics, frequency response, and cross-scale variations, are significantly enhanced. Simultaneously, through deep fusion modeling using multi-scale convolution and lightweight self-attention mechanisms, a global understanding and fine-grained discrimination of silent oral action sequences can be achieved on the basis of a unified feature space, laying a solid foundation for subsequent few-shot personalized learning and high-precision classification and recognition.

[0162] In this embodiment, as an optional implementation, the deep neural network encoder, which includes the correspondence between determined oral action categories and the prototype feature vector, comprises:

[0163] The prototype feature vector is input into the initial encoder of the deep neural network to obtain the predicted category;

[0164] Based on the oral action category corresponding to the predicted category and the prototype feature vector, the model parameters of the deep neural network initial encoder are adjusted using backpropagation, and the deep neural network initial encoder is trained.

[0165] Calculate the overall loss of the initial encoder of the trained deep neural network. If the overall loss is less than a preset threshold, the deep neural network encoder is obtained. In the deep neural network encoder, the correspondence between the labeled oral action categories and the prototype feature vector is stored.

[0166] In this embodiment, as an optional implementation, the predicted category is the predicted probability of each category. Let the query sample x be... query The deep encoding features (prototype feature vectors) are f θ (xquery), the prototype vectors for each category are p c Then the predicted class probability distribution P(y=c|x) query The expression using softmax normalization is as follows:

[0167]

[0168] Where d(·,·) represents the Euclidean distance.

[0169] In this embodiment, as an optional implementation, the overall loss includes, but is not limited to, contrastive loss, prototype network loss, and regularization loss. The contrastive loss characterizes the distance between samples of the same class, the prototype network loss characterizes the classification accuracy, and the regularization loss characterizes the generalization ability.

[0170] In this embodiment, contrastive learning is used as a pre-training method for the feature space to measure the similarity between samples. This serves as a training supervision signal, guiding the deep neural network encoder to optimize the embedding space structure. This aims to bring similar samples closer together and push dissimilar samples further apart, thereby enhancing the clustering of identical oral action samples and the separation between different action samples in the feature embedding space. As an optional embodiment, let the EEG transformation sample features and EMG transformation sample features corresponding to two time steps be zi and zj, respectively. If these two features belong to the same oral action category, they are defined as a positive sample pair; if they belong to different oral action categories, they are defined as a negative sample pair. The contrastive learning objective is to minimize the Euclidean distance between positive sample pairs and maximize the distance between negative sample pairs, thus minimizing the contrastive loss. The contrastive loss is defined as:

[0171]

[0172] Among them, y ij ∈{0,1} represents the sample pair label (oral action category), m is the preset boundary distance hyperparameter, and ||·||2 is the Euclidean norm. Contrast loss can effectively alleviate the problem of feature space convergence difficulty caused by sparse class samples and large intra-sample variation, and effectively improve the class separation ability under small sample conditions.

[0173] In this embodiment, by calculating the contrastive loss between the two actions—for example, calculating the loss for actions belonging to the same oral action category and the loss for actions not belonging to the same oral action category—it is determined whether they belong to the same oral action category and can be used as a labeled sample set in subsequent processing. Thus, through contrastive learning optimization, a feature space with good class-discriminating boundaries can be autonomously learned under unsupervised or semi-supervised conditions. This lays the foundation for accurate classification with few samples, enabling rapid modeling and discrimination of class boundaries under extremely limited sample conditions, and demonstrating good adaptability to the individualized action patterns of new users.

[0174] In this embodiment, to further improve the stability of few-sample adaptation, a class center regularization strategy is introduced during training. This strategy minimizes the distance between sample features and their corresponding class prototypes, suppressing overfitting risk, enhancing the model's generalization ability, encouraging sample features to converge towards their corresponding class prototypes, and preventing class boundary ambiguity caused by data sparsity. The regularization loss is defined as follows:

[0175]

[0176] Where, p yn For sample xn The prototype vector of the category. This regularization loss term continuously converges to the feature distribution during training, stabilizing the system's adaptation performance when faced with individual differences.

[0177] The corresponding prototype classification loss is defined as the negative log-likelihood loss:

[0178]

[0179] In this embodiment, the negative log-likelihood loss L is optimized. proto This can help the system form clear and structured category boundaries in the feature space, thereby improving classification accuracy and inference stability.

[0180] Ultimately, the overall loss L in this embodiment is... total It is a weighted combination of the above three sub-loss terms, in the following form:

[0181]

[0182] Wherein, λ1, λ2, and λ3 are the weighted hyperparameters corresponding to each sub-loss term, which are dynamically adjusted through cross-validation to achieve the optimal training effect under different application requirements (such as prioritizing classification accuracy or prioritizing rapid adaptation).

[0183] In this embodiment, through the overall loss design and optimization mechanism, the feature space structure can be effectively optimized while ensuring classification accuracy, thereby enhancing the robustness and individualized adaptability of the model. This provides a strong guarantee for efficient deployment and application in actual clinical scenarios, and can comprehensively optimize the training effect of the silent oral action recognition system, thereby maximizing the discriminative ability, robustness and individualized generalization ability.

[0184] In this embodiment, considering the significant individual differences and difficulties in obtaining training data for patients with severe speech disorders in practical applications, this embodiment proposes a targeted few-sample incremental learning strategy based on deep feature modeling. This strategy aims to achieve rapid adaptation using a very small number of user-specific samples, thereby improving the personalized practicality and generalization ability of the deep neural network encoder. Therefore, as an optional embodiment, the method further includes:

[0185] Based on the EEG and EMG sample features, sample pairs are constructed.

[0186] Calculate the contrast loss between the sample pair and the prototype feature vectors corresponding to each pre-labeled oral action category. If the contrast loss is less than a pre-set contrast loss threshold, label the sample pair as a positive sample pair and its corresponding oral action category. Based on the positive sample pair, obtain the corresponding prototype feature vector and store it in the labeled sample set. The labeled sample set stores the prototype feature vectors corresponding to each labeled oral action category.

[0187] In this embodiment, a few-sample incremental learning strategy is employed to rapidly adapt the system to a minimal amount of individual data. This significantly improves the accuracy and stability of silent oral motor recognition across different patient groups and motor styles. It achieves a complete closed loop from raw signal input to action category output, balancing inference speed, classification accuracy, and decision reliability. This provides a practical, natural, and efficient auxiliary communication tool for patients with severe speech disorders, offering significant application advantages. Specifically, it has the following beneficial technical effects:

[0188] First, in the signal feature extraction stage, this embodiment adopts multi-scale time-frequency decomposition and cross-modal statistical characteristic modeling based on the physical characteristics of EEG and EMG signals. It fully explores the local dynamic changes and cross-modal collaborative response information related to actions. By constructing joint covariance matrix features and cross-correlation matrix features and building a unified feature space, it improves the deep correlation modeling capability between different modalities and strengthens the ability to recognize silent oral actions. It can accurately distinguish complex fine-grained oral actions and accurately recognize complex silent oral actions, thereby improving the naturalness and expressive richness of silent communication and laying the foundation for high-quality expression of fused features.

[0189] Secondly, in the stage of feature construction and unified representation, this embodiment proposes a feature-level standardization and spatial alignment mechanism, which unifies the differences in dimensions, scale and distribution characteristics of different modal features, effectively reduces the risk of internal inconsistency caused by modality fusion, and significantly improves the stability and convergence efficiency of subsequent deep modeling and classification reasoning.

[0190] Next, in terms of deep feature modeling, this embodiment innovatively combines multi-scale temporal convolution with a lightweight Transformer structure, which can capture local fine-grained dynamic changes and model long-term global dependencies. At the same time, the time-frequency domain dual-channel attention mechanism enhances the system's comprehensive perception of complex dynamic characteristics during action execution, greatly improving the model's expressive power and recognition robustness. It can adapt to patients with multiple languages, action patterns and different pathological states, and has good versatility and expansion potential.

[0191] Furthermore, addressing the issues of significant individual differences and scarce training samples, this embodiment designs a few-sample incremental learning strategy. It optimizes the feature space structure through comparative learning and achieves rapid personalized adaptation based on a prototype network. This enables rapid individual adaptation with minimal user guidance data, feature space optimization, and deep neural network encoder fine-tuning with minimal user data. This significantly shortens training and deployment time, thereby substantially reducing the sample requirements and training time for new user deployment. It improves the scalability and practicality in actual clinical applications, enabling rapid personalized deployment and effectively enhancing the accuracy and robustness of action recognition for different users.

[0192] Finally, in the classification reasoning stage, this embodiment introduces a classification decision mechanism based on prototype matching and supplements it with confidence assessment to improve the decision stability and noise resistance in continuous silent action recognition scenarios, effectively addressing issues such as action ambiguity and signal fluctuations in actual use.

[0193] In summary, the intelligent oral action recognition method proposed in this embodiment has achieved systematic breakthroughs in feature modeling depth, fusion mechanism design, personalized adaptability, and inference system stability. It can effectively make up for the shortcomings of existing single-modal methods such as low recognition rate, poor adaptability, and complex deployment, and provides a practical, convenient and natural new way of assistive communication for patients with severe speech disorders, effectively improving the efficiency of assistive communication.

[0194] Based on the same inventive concept, such as Figure 4 As shown, this embodiment of the invention also provides a device for recognizing oral cavity movements, the device comprising:

[0195] The brain-computer interface signal acquisition and processing module 401 is used to acquire electromyography (EMG) signals and electroencephalography (EEG) signals of the target object, perform continuous wavelet transform on the EEG signals to acquire EEG time-frequency band features, and perform short-time Fourier transform on the EMG signals to acquire EMG low-frequency features and EMG high-frequency features characterizing muscle activity.

[0196] In this embodiment, as an optional embodiment, the brain-computer interface signal acquisition and processing module 401 is specifically used for:

[0197] Electromyography (EMG) signals are collected using dry electrode surface electromyography sensors deployed on pre-defined facial muscle groups involved in oral cavity movements.

[0198] In this embodiment, as another optional embodiment, the brain-computer interface signal acquisition and processing module 401 is further used for:

[0199] The EEG signal was subjected to continuous wavelet transform to obtain the EEG time-frequency signal;

[0200] Based on the frequency scale corresponding to the pre-set extraction band, feature extraction is performed on the EEG time-frequency signal to obtain the EEG time-frequency band features corresponding to each extraction band.

[0201] The feature transformation module 402 is used to construct an EEG feature matrix based on the EEG time-frequency band features, and to construct an EMG feature matrix based on the EMG low-frequency features and EMG high-frequency features. Using a cross-modal feature mapping mechanism, the EEG feature matrix and the EMG feature matrix are mapped to the same feature space respectively to obtain EEG transformation features and EMG transformation features.

[0202] In this embodiment, as an optional embodiment, the feature transformation module 402 is specifically used for:

[0203] The EEG feature matrix is ​​mapped using a pre-set first mapping function to obtain EEG transformation features;

[0204] The electromyography feature matrix is ​​mapped using a pre-set second mapping function to obtain electromyography transformation features. The first mapping function and the second mapping function are mapping functions in the same feature space.

[0205] The feature splicing module 403 is used to splice the EEG transformation features and EMG transformation features based on a pre-set feature-level fusion strategy to obtain a preliminary fusion feature matrix, and to perform mean zeroing and variance normalization operations on the preliminary fusion feature matrix to obtain a standardized feature matrix.

[0206] The feature fusion module 404 is used to obtain query vector features, key vector features and value vector features based on the standardized feature matrix and the pre-set linear mapping matrix, and to obtain an adaptive weighted feature vector based on the query vector features, key vector features and value vector features and the pre-set lightweight self-attention function.

[0207] The category recognition module 405 is used to calculate the distance between the adaptive weighted feature vector and the prototype feature vector corresponding to each oral action category pre-labeled by the deep neural network encoder, and to determine the oral action category of the adaptive weighted feature vector based on the calculated distance.

[0208] In this embodiment, as an optional embodiment, the device further includes:

[0209] The joint covariance matrix feature construction module (not shown in the figure) is used to obtain joint covariance matrix features based on a pre-set covariance matrix construction strategy, the EEG feature matrix and the EMG feature matrix, and to splice the joint covariance matrix features, EEG transformation features and EMG transformation features to obtain the preliminary fusion feature matrix.

[0210] In this embodiment, as another optional embodiment, the device further includes:

[0211] The cross-correlation matrix feature construction module is used to obtain cross-correlation matrix features based on a pre-set correlation matrix construction strategy, the EEG feature matrix, and the EMG feature matrix, and to splice the cross-correlation matrix features, EEG transformation features, and EMG transformation features to obtain the preliminary fusion feature matrix.

[0212] In this embodiment, as another optional embodiment, the device further includes:

[0213] The model training module is used to acquire electromyography (EMG) and electroencephalography (EEG) sample signals of each sample subject based on oral movements. Based on one or more interactions with each sample subject, the module determines the oral movement category corresponding to each EMG and EEG sample signal and labels the EMG and EEG sample signals. The sample subjects include: patients with vocal cord injury, patients with neurodegenerative diseases, stroke patients, and patients with traumatic brain injury.

[0214] Based on the electromyography (EMG) sample signals and electroencephalography (EEG) sample signals, a sample pair is constructed. Continuous wavelet transform is performed on the EEG sample signals in the sample pair to obtain EEG time-frequency band sample features. In addition, short-time Fourier transform is performed on the EMG sample signals in the sample pair to obtain EMG low-frequency sample features and EMG high-frequency sample features characterizing muscle activity.

[0215] Based on the EEG time-frequency band sample features, an EEG sample feature matrix is ​​constructed, and based on the EMG low-frequency sample features and EMG high-frequency sample features, an EMG sample feature matrix is ​​constructed. The EEG sample feature matrix and the EMG sample feature matrix are mapped to the same feature space to obtain EEG transformation sample features and EMG transformation sample features.

[0216] Based on the feature-level fusion strategy, the EEG transformation sample features and EMG transformation sample features are spliced ​​together to obtain a preliminary fused sample feature matrix. The mean is zeroed and the variance is normalized on the preliminary fused sample feature matrix to obtain a standardized sample feature sequence.

[0217] Based on the standardized sample feature matrix and the linear mapping matrix, query vector sample features, key vector sample features and value vector sample features are obtained. Based on the query vector sample features, key vector sample features and value vector sample features and the lightweight self-attention function, prototype feature vectors are obtained, and the deep neural network encoder including the correspondence between the determined oral action categories and the prototype feature vectors is constructed.

[0218] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for recognizing oral movements in any of the above possible implementations.

[0219] Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0220] Based on the same inventive concept, see [link to inventive concept] Figure 5 This invention also provides an electronic device, including a memory 101 (e.g., non-volatile memory), a processor 102, and a computer program stored on the memory 101 and executable on the processor 102. When the processor 102 executes the program, it implements the steps of the method for recognizing oral movements in any of the above possible implementations, which can be equivalent to the aforementioned device for recognizing oral movements. Of course, the processor can also be used to process other data or perform calculations. This electronic device can be a PC, server, terminal, or other similar device.

[0221] like Figure 5 As shown, the electronic device may also include: memory 103, network interface 104, and internal bus 105. In addition to these components, other hardware may also be included, which will not be described in detail here.

[0222] It should be noted that the above-mentioned device for recognizing oral movements can be implemented by software. As a logical device, it is formed by the processor 102 of the electronic device in which it is located reading the computer program instructions stored in the non-volatile memory into the memory 103 for execution.

[0223] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0224] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by special-purpose logic circuitry—such as FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), and the device can also be implemented as special-purpose logic circuitry.

[0225] Computers suitable for executing computer programs include, for example, general-purpose or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0226] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0227] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0228] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0229] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0230] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0231] The above are merely specific embodiments of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method of recognizing an oral action, characterized by, The method comprises the following steps: obtaining electromyogram signals and electroencephalogram signals of a target object through a pre-set brain-computer interface acquisition device, performing continuous wavelet transform on the electroencephalogram signals to obtain electroencephalogram time-frequency band features, and performing short-time Fourier transform on the electromyogram signals to obtain electromyogram low-frequency features and electromyogram high-frequency features representing muscle activity; based on the electroencephalogram time-frequency band features, constructing an electroencephalogram feature matrix, and based on the electromyogram low-frequency features and electromyogram high-frequency features, constructing an electromyogram feature matrix, and using a cross-modal feature mapping mechanism to map the electroencephalogram feature matrix and the electromyogram feature matrix into the same feature space respectively to obtain electroencephalogram transformed features and electromyogram transformed features; based on a pre-set feature-level fusion strategy, splicing the electroencephalogram transformed features and the electromyogram transformed features to obtain a preliminary fusion feature matrix, performing mean zeroing and variance normalization operations on the preliminary fusion feature matrix to obtain a standardized feature matrix; based on the standardized feature matrix and a pre-set linear mapping matrix, obtaining query vector features, key vector features and value vector features, and based on the query vector features, the key vector features and the value vector features and a pre-set lightweight self-attention function, obtaining an adaptive weighted feature vector; transforming the adaptive weighted feature according to a pre-set time domain attention function to obtain a time domain attention feature, transforming the adaptive weighted feature according to a pre-set frequency domain attention function to obtain a frequency domain attention feature, and splicing the time domain attention feature and the frequency domain attention feature to obtain an adaptive weighted feature vector used for calculating a distance; calculating distances between the adaptive weighted feature vector and prototype feature vectors corresponding to each oral action category pre-labeled by a deep neural network encoder respectively, and determining an oral action category of the adaptive weighted feature vector based on the calculated distances.

2. The method of recognizing oral action according to claim 1, wherein, The method further comprises the following steps: using dry electrode surface electromyography sensors arranged on pre-set facial muscle groups participating in oral action to collect electromyogram signals.

3. The method of recognizing oral action according to claim 1, wherein, The method further comprises the following steps: performing continuous wavelet transform on the electroencephalogram signals to obtain electroencephalogram time-frequency signals; extracting features from the electroencephalogram time-frequency signals according to pre-set frequency scales corresponding to each extracted band to obtain electroencephalogram time-frequency band features corresponding to each extracted band respectively.

4. The method of claim 1, wherein, The method further comprises the following steps: using a pre-set first mapping function to map the electroencephalogram feature matrix to obtain electroencephalogram transformed features; using a pre-set second mapping function to map the electromyogram feature matrix to obtain electromyogram transformed features, the first mapping function and the second mapping function being mapping functions of the same feature space.

5. The method of recognizing an oral action according to any one of claims 1 to 4, characterized in that, The method further comprises the following steps: The joint covariance matrix feature is obtained based on a preset covariance matrix construction strategy, the electroencephalogram feature matrix and the electromyogram feature matrix.

6. The method of recognizing mouth action according to any one of claims 1 to 4, characterized in that, The method further includes: The cross-correlation matrix feature is obtained based on a preset correlation matrix construction strategy, the electroencephalogram feature matrix and the electromyogram feature matrix, and the preliminary fusion feature matrix is obtained by splicing the cross-correlation matrix feature, the electroencephalogram transformation feature and the electromyogram transformation feature.

7. The method of recognizing oral actions according to any one of claims 1 to 4, characterized in that, The prototype feature vector corresponding to each oral action category is obtained by pre-labeling, including: The electromyogram sample signal and the electroencephalogram sample signal of each sample object based on oral action collection are obtained, and the oral action category corresponding to each electromyogram sample signal and electroencephalogram sample signal is determined according to one or more communications with the sample object, and the electromyogram sample signal and the electroencephalogram sample signal are labeled, and the sample object includes a vocal cord injury patient, a neurodegenerative disease patient, a stroke patient and a traumatic brain injury patient; Based on the electromyogram sample signal and the electroencephalogram sample signal, a sample pair is constructed, the electroencephalogram sample signal in the sample pair is subjected to continuous wavelet transformation to obtain an electroencephalogram time-frequency band sample feature, and the electromyogram sample signal in the sample pair is subjected to short-time Fourier transformation to obtain electromyogram low-frequency sample features and electromyogram high-frequency sample features representing muscle activity; Based on the electroencephalogram time-frequency band sample feature, an electroencephalogram sample feature matrix is constructed, and based on the electromyogram low-frequency sample features and the electromyogram high-frequency sample features, an electromyogram sample feature matrix is constructed, and the electroencephalogram sample feature matrix and the electromyogram sample feature matrix are respectively mapped into the same feature space to obtain electroencephalogram transformation sample features and electromyogram transformation sample features; Based on the feature-level fusion strategy, the electroencephalogram transformation sample features and the electromyogram transformation sample features are spliced to obtain a preliminary fusion sample feature matrix, and mean zero and variance normalization operations are performed on the preliminary fusion sample feature matrix to obtain a standardized sample feature sequence; Based on the standardized sample feature matrix and the linear mapping matrix, a query vector sample feature, a key vector sample feature and a value vector sample feature are obtained, a prototype feature vector is obtained based on the query vector sample feature, the key vector sample feature and the value vector sample feature, and the lightweight self-attention function, and the deep neural network encoder including the corresponding relationship between the determined oral action category and the prototype feature vector is constructed.

8. An apparatus for recognizing an oral action, characterized by, The device for recognizing oral action includes: An electroencephalogram signal and an electromyogram signal of a target object are obtained, the electroencephalogram signal is subjected to continuous wavelet transformation to obtain electroencephalogram time-frequency band features, and the electromyogram signal is subjected to short-time Fourier transformation to obtain electromyogram low-frequency features and electromyogram high-frequency features representing muscle activity. The feature transformation module is configured to construct an electroencephalogram feature matrix based on the electroencephalogram time-frequency band features, construct an electromyogram feature matrix based on the electromyogram low-frequency features and electromyogram high-frequency features, map the electroencephalogram feature matrix and the electromyogram feature matrix into the same feature space by using a cross-modal feature mapping mechanism, and obtain electroencephalogram transformed features and electromyogram transformed features; The feature splicing module is configured to splice the electroencephalogram transformed features and the electromyogram transformed features based on a pre-set feature-level fusion strategy, obtain a preliminary fusion feature matrix, and perform mean zeroing and variance normalization operations on the preliminary fusion feature matrix to obtain a standardized feature matrix; The feature fusion module is configured to obtain query vector features, key vector features, and value vector features based on the standardized feature matrix and a pre-set linear mapping matrix, obtain an adaptive weighted feature vector based on the query vector features, the key vector features, the value vector features, and a pre-set lightweight self-attention function; The feature fusion module is configured to obtain a time domain attention feature by transforming the adaptive weighted feature according to a pre-set time domain attention function, obtain a frequency domain attention feature by transforming the adaptive weighted feature according to a pre-set frequency domain attention function, and splice the time domain attention feature and the frequency domain attention feature to obtain an adaptive weighted feature vector used for calculating a distance; The class recognition module is configured to calculate distances between the adaptive weighted feature vector and prototype feature vectors corresponding to various oral action classes pre-labeled by a deep neural network encoder, respectively, and determine an oral action class of the adaptive weighted feature vector based on the calculated distances.

9. A storage medium, characterized by A program or instruction is stored on a storage medium, and the program or instruction is run by a processor to implement the steps of the method for recognizing oral actions according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method for recognizing oral actions according to any one of claims 1 to 7 when executing the program.

Images