Language decoding method for electroencephalogram signals, program product and related device

Abstract

The application relates to a language decoding method for an electroencephalogram signal, a program product and related equipment. The method comprises the following steps: inputting a target electroencephalogram signal into a trained speech detection model to output a pronunciation time period of the target electroencephalogram signal; cutting the target electroencephalogram signal according to the pronunciation time period to obtain a candidate pronunciation segment; screening a target pronunciation segment from the candidate pronunciation segment, wherein the target pronunciation segment is derived from a candidate channel and has a pronunciation frequency within a candidate frequency band, and extracting a frequency domain feature of the target pronunciation segment; extracting a time domain feature according to an original voltage signal; inputting the frequency domain feature into a frequency domain branch of a trained pinyin decoding model and inputting the time domain feature into a time domain branch of the pinyin decoding model to output an initial probability and a final probability corresponding to the target electroencephalogram signal; and converting the initial probability and the final probability into a Chinese character sequence through a trained language model. The application can improve the language decoding accuracy, reduce the calculation complexity and shorten the delay.

Description

Technical Field

[0001] This application relates to the field of brain-computer interface technology, and in particular to a language decoding method, program product and related equipment for electroencephalogram (EEG) signals. Background Technology

[0002] With the development of brain-computer interface (BCI) technology, it has greatly aided research into assistive communication systems for patients with language disorders, brain-controlled input devices and prosthetic control interfaces, as well as the mechanisms of language production and perception in neuroscience experiments. Language ability is a core component of human cognitive function. For example, severe motor impairments caused by various diseases such as stroke, amyotrophic lateral sclerosis (ALS), and locked-in syndrome can lead to the loss of language expression ability, severely impacting patients' quality of life and social participation. BCI technology, by decoding brain signals and directly reconstructing language output, provides a potential pathway for the "thought-language" transformation for these patients.

[0003] Currently, language decoding methods based on EEG signals are mainly implemented through three approaches: directly using raw ECoG (Electrocorticography) signals, which are then filtered and fed into a traditional classifier; using ECoG signals in a specific high-gamma band for Chinese language decoding; and using sEEG stereoscopic EEG signals, which are pre-trained or processed end-to-end before being classified for language decoding.

[0004] The drawback of using this method for language decoding of EEG signals lies in the blind selection of signal features. Traditional methods often use full-channel, full-band ECoG signals as input without feature selection specific to the language task. Traditional methods use 64-channel, full-band ECoG signals for speech decoding, resulting in a feature dimension as high as 64 × 100 = 6400 dimensions. This leads to a huge computational burden and severe overfitting. Experiments have shown that neural activity in a large number of channels is irrelevant to the language task. Blindly selecting signal features introduces significant noise and increases computational complexity. Summary of the Invention

[0005] Therefore, it is necessary to provide a language decoding method, program product, and related equipment for EEG signals that can improve the accuracy of language decoding while reducing computational complexity, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for language decoding of electroencephalogram (EEG) signals, including:

[0007] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0008] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0009] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0010] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0011] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0012] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0013] In one embodiment, the step of determining the candidate channel includes:

[0014] Acquire sample EEG signals through full-channel acquisition, and extract the time-frequency amplitude sequences of the vocal segment and the time-frequency amplitude sequences of the silent segment from the sample EEG signals based on the annotation of the sample EEG signals;

[0015] Calculate the similarity value between the time-frequency amplitude sequence of the pronunciation segment and the time-frequency amplitude sequence of the silence segment in different channels;

[0016] The channel corresponding to the time-frequency amplitude sequence of the pronunciation segment with a similarity value lower than the preset value is determined as the candidate channel.

[0017] In one embodiment, calculating the similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silence segment in different channels includes:

[0018] The similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels is calculated using the following formula:

[0019]

[0020] in, Indicates the number of frequency bands. Indicates the total number of channels. The 'j' indicates the vocal channel, and the 'j' indicates the silence channel. Indicates frequency, Indicates the vocal tract The frequencies of the mute channel j and the mute channel j are the same. Similarity measure at different times;

[0021] The similarity measure between the vocal channel and the silent channel is determined using the following formula. :

[0022]

[0023] in, This indicates the result after normalization. Take the maximum value;

[0024] The cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels. Normalization is performed using the following formula:

[0025]

[0026] in, This represents the cyclic cross-correlation value. Indicates the time lag;

[0027] The cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels is calculated using the following formula:

[0028]

[0029] in, The time lag is represented by T, the total number of time segments, and t, which represents the t-th time segment. This represents the cyclic cross-correlation function. Indicates the vocal tract. Indicates frequency, Indicates the mute channel. Represents the time series of the spectral spectrum of the articulation segment; This represents the time series of the spectrum of the silent segment.

[0030] In one embodiment, the step of determining the candidate frequency band includes:

[0031] Extract the frequencies carried in the time series of the phonological segment spectrum of the sample EEG signal transmitted through the candidate channel, and obtain the pinyin category of the phonological segment spectrum time series that has been pre-annotated;

[0032] Calculate the Pearson correlation coefficient between the time series spectra of the pronunciation segment that are of the same pinyin category and frequency but originate from different candidate channels;

[0033] Based on the calculated Pearson correlation coefficient, calculate the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin category is the same;

[0034] The average of the intra-class coherence scores for all different pinyin categories is used to obtain the final intra-class coherence score between the candidate channel and the corresponding frequency.

[0035] Construct a global redundancy matrix between the candidate channel and the frequencies carried in the spectral time series of the speech segment;

[0036] Calculate the redundancy distance between two different feature units in the global redundancy matrix that have different candidate channels but the same frequency;

[0037] Based on this redundancy distance, calculate the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channels;

[0038] The frequency set corresponding to the candidate channels whose final score of intra-class synergy is greater than the first threshold and whose average redundancy distance is greater than the second threshold is determined as the candidate frequency band.

[0039] In one embodiment, calculating the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin categories are the same includes:

[0040] The intra-class synergy score is calculated using the following formula:

[0041]

[0042] in, Indicates the category of Pinyin. Indicates the pinyin category as The number of samples, Represents the set of candidate channels. Indicates different candidate channels. Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment This represents the calculated Pearson correlation coefficient;

[0043] The step of calculating the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channels based on the redundancy distance includes:

[0044] The average redundancy distance is calculated using the following formula:

[0045]

[0046]

[0047] in, , Represents the set of candidate channels. Indicates the number of frequency bands. The candidate channel is represented. Indicates frequency, Indicates candidate channel as , frequency is The characteristic unit, Indicates candidate channel as , frequency is The characteristic unit, express , The Pearson correlation coefficient between two feature units.

[0048] In one embodiment, the speech detection model includes a trained LSTM encoder, the frequency domain branch of the pinyin decoding model includes the LSTM encoder, and the method further includes:

[0049] The frequency domain features are extracted from the target speech segment using the trained LSTM encoder.

[0050] The frequency domain feature and the time domain feature are concatenated in the channel dimension through the feature fusion layer of the Pinyin decoding model to obtain the fused feature;

[0051] The initial consonant classification head of the Pinyin decoding model outputs the fused feature to obtain the probability of the initial consonant, and the final vowel classification head of the Pinyin decoding model outputs the fused feature to obtain the probability of the final vowel.

[0052] Secondly, this application also provides a language decoding device for electroencephalogram (EEG) signals, comprising:

[0053] The time period detection module is used to acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal.

[0054] The trimming module is used to trim the target EEG signal according to the pronunciation time period to obtain at least one candidate pronunciation segment;

[0055] The frequency domain feature extraction module is used to select candidate pronunciation segments from the at least one candidate pronunciation segments whose corresponding target EEG signals originate from candidate channels and whose pronunciation frequencies are within candidate frequency bands, as target pronunciation segments, and to extract frequency domain features based on the target pronunciation segments;

[0056] The time-domain feature extraction module is used to acquire the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal.

[0057] The probability output module is used to input the frequency domain feature into the frequency domain branch of the trained Pinyin decoding model, input the time domain feature into the time domain branch of the Pinyin decoding model, and output the initial consonant probability and final vowel probability corresponding to the target EEG signal.

[0058] The conversion module is used to convert the initial consonant and final vowel probabilities into Chinese character sequences using the trained language model.

[0059] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0060] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0061] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0062] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0063] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0064] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0065] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0066] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0067] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0068] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0069] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0070] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0071] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0072] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0073] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0074] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0075] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0076] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0077] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0078] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0079] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0080] The aforementioned language decoding methods, software products, and related equipment for EEG signals, after cropping candidate pronunciation segments from the target EEG signal to be decoded, select candidate pronunciation segments from these segments that correspond to the target EEG signal originating from candidate channels and whose pronunciation frequencies fall within candidate frequency bands. This initial screening of the target EEG signal, rather than using all channels, preserves brain regions sensitive to language tasks. Since language-related brain regions only cover a limited area of ​​the cerebral cortex, while the electrode arrays corresponding to each channel for acquiring the target EEG signal typically cover a large area of ​​the cortical surface, including sensorimotor areas and visual areas unrelated to language, this approach is beneficial. This application performs an initial screening of the target EEG signal through candidate channels, which can eliminate a large number of invalid channels, reduce noise interference, and improve decoding efficiency and accuracy. On the other hand, this application performs a second screening of candidate pronunciation segments by utilizing the condition that the pronunciation frequency is within the candidate frequency band. This can select the frequency components with the most language discrimination power from many candidate pronunciation segments. It can fuse and analyze language information with different levels (such as phoneme recognition level, prosodic processing level, vocabulary retrieval level, etc.) that may be complementary carried by different frequency bands. At the same time, it can further verify the candidate pronunciation segments obtained in the previous steps, further improve the accuracy of language decoding, and shorten the decoding delay. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a flowchart illustrating a language decoding method for electroencephalogram (EEG) signals in one embodiment.

[0083] Figure 2 This is a flowchart illustrating the steps for determining candidate channels in one embodiment;

[0084] Figure 3 This is a flowchart illustrating the steps for determining candidate frequency bands in one embodiment;

[0085] Figure 4 This is a schematic diagram of the architecture of a Pinyin decoding model in one embodiment;

[0086] Figure 5 This is a structural block diagram of a language decoding device for electroencephalogram signals in one embodiment;

[0087] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0089] It should be noted that the terms "first," "second," etc., used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more.

[0090] In one embodiment, such as Figure 1 As shown, a method for language decoding of electroencephalogram (EEG) signals is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes steps 100 to 600. Wherein:

[0091] Step 100: Obtain the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal.

[0092] In this embodiment, an ECoG electrode array can be used to acquire the target EEG signal and the sample EEG signals mentioned later. This ECoG electrode array uses a 10-10 international standard system to cover the language-related cortex of the left hemisphere, including the inferior frontal gyrus, superior temporal gyrus, middle temporal gyrus, and motor cortex. The electrode material is PtIr platinum-iridium alloy, the electrode spacing is 10 mm, and the exposed diameter is 4 mm. This embodiment uses a 32-channel ECoG device and a 1 kHz sampling rate signal as an example to describe in detail the implementation details of the technical solution of this invention. The parameters can be adjusted according to the actual equipment configuration and application scenario.

[0093] Preferably, after acquiring the EEG signal, the method further includes preprocessing the EEG signal to obtain the target EEG signal. Further, the preprocessing process includes:

[0094] Bandpass filtering: A fourth-order Butterworth filter is used, with a frequency range of 0.5Hz-500Hz, to retain the frequency band related to neural activity while suppressing power line interference (50Hz) and high-frequency noise;

[0095] Power frequency notch filtering: For 50Hz and 100Hz power frequencies and their harmonics, a second-order notch filter is used for deep suppression, with a notch depth greater than 40dB.

[0096] Re-reference: The Common Average Reference (CAR) method is used to subtract the average value of all channels from the signal of each channel to eliminate global noise;

[0097] Baseline correction: For each trial EEG signal, the mean and standard deviation of the previous 3-second resting period signal are standardized to eliminate signal drift between individuals and between trial EEG signals;

[0098] Artifact removal: Calculate the signal amplitude and gradient of each channel. If they exceed the mean ± 3 times the standard deviation, mark the channel as abnormal and perform interpolation repair. If the proportion of artifacts in the entire EEG signal exceeds 20%, remove the EEG signal of that trial.

[0099] Furthermore, the speech detection model used in this embodiment is the VAD (Voice Activity Detection) model. This speech detection model, from the input layer to the output layer, includes an input layer, an LSTM encoder, a global max pooling layer, a fully connected layer, and an output layer. Wherein:

[0100] The input layer is used to receive variable-length temporal features. Each temporal sequence T varies depending on the duration of the speech, typically between 100ms and 800ms. Here, T represents the number of time steps, indicating how many frames / times a segment of input has, and representing a variable sequence length.

[0101] The LSTM encoder consists of 64 hidden units with an input dimension of 56. This layer is used to capture temporal dependencies and learn inter-frame contextual information. The gating mechanism of the LSTM encoder is as follows:

[0102]

[0103] in, express Input at any time, Indicates a hidden state. This represents the hidden state of the LSTM at time t-1. Indicates cell state, This represents the cell state at time t-1. This represents the sigmoid function. This represents element-wise multiplication. This represents the output of the LSTM input gate at time t. This represents the output of the LSTM forget gate at time t. This represents the output of the LSTM output gate at time t. This represents the candidate cell state of the LSTM at time t. , , , Let represent the bias vectors of the input gate, forget gate, output gate, and candidate state in the LSTM encoder, respectively. , , and These represent the inputs in the LSTM encoder. The weight matrices are sequentially arranged from the input gate, forget gate, output gate, and candidate state, where tanh represents the hyperbolic tangent activation function. , , , These represent the cyclic weight matrices of the input gate, forget gate, output gate, and candidate state in the LSTM encoder, respectively. In this embodiment, they are... , , , Defined as the previous hidden state The weights of each gate or candidate state;

[0104] The global max pooling layer is used to perform max pooling on the output of the LSTM encoder in the time dimension. This operation extracts the most significant temporal features and is invariant to variable-length inputs.

[0105] The fully connected layer in this embodiment includes 32 neurons, and the activation function can be ReLU (Linear Rectification function).

[0106] The output layer of this speech detection model is a single neuron, using the sigmoid function as the activation function, and the output is the pronunciation probability.

[0107] When training this speech detection model, positive samples from the pronunciation segment and negative samples from the silence segment are selected. The training process of this speech detection model can be represented as follows:

[0108]

[0109] in, Indicates sample The time-frequency feature tensor, These are binary labels, where 1 represents a sound frame and 0 represents a non-sound frame. Indicates the total number of samples. Indicates the sample The output.

[0110] In this model, each sample is composed of equal-length segments extracted from the vocal segment and the preceding and following silent segments to form a positive and negative sample pair, ensuring temporal alignment. The loss function of this speech detection model can be the Binary Cross-Entropy loss function. The training process ends when the loss function converges. The target EEG signal to be decoded is then input into the trained speech detection model, which can output the vocalization time segment of the target EEG signal. Specifically, the target EEG signal stream to be decoded from the continuous ECoG cortical EEG is sent into the VAD speech detection model to detect the start and end points of speech activity in real time and output one or more vocalization time segments.

[0111] Step 200: Based on the pronunciation time period, the target EEG signal is cropped to obtain at least one candidate pronunciation segment.

[0112] In this embodiment, this step is used to cut out at least one candidate pronunciation segment from the original target EEG signal according to the time boundary output by the VAD speech detection model, and to generate a frequency domain input based on the candidate pronunciation segment.

[0113] Step 300: Select candidate pronunciation segments from the at least one candidate pronunciation segments whose corresponding target EEG signals originate from candidate channels and whose pronunciation frequencies are within candidate frequency bands, and use them as target pronunciation segments. Extract frequency domain features based on the target pronunciation segments.

[0114] Preferably, before the step of extracting frequency domain features based on the target pronunciation segment, the method further includes feature enhancement of the target EEG signal, specifically including:

[0115] The candidate speech segments obtained by cropping are subjected to STFT (short-time Fourier transform) to obtain intermediate time-frequency features;

[0116] Calculate the logarithmic power spectrum of the intermediate time-frequency features, and enhance the features of the candidate speech segments based on the calculated logarithmic power spectrum.

[0117] The intermediate time-frequency characteristic can be expressed as:

[0118]

[0119] in, Indicates the channel index. This represents the frequency index, and t represents the time frame index. Indicates the time lag as Candidate pronunciation segments.

[0120] Furthermore, in this embodiment, the logarithmic power spectrum of the intermediate time-frequency feature is calculated using the following formula:

[0121]

[0122] in, , represents the constant to prevent the logarithm from reaching zero, and k represents the channel. t represents frequency, and t represents time.

[0123] It is understandable that, before training the speech detection model using sample EEG signals in the early stages, the aforementioned feature enhancement methods can be used to enhance the features of the sample EEG signals, and then the enhanced data can be used to train the speech detection model to help the model converge as quickly as possible and improve training speed and recognition accuracy.

[0124] It is understood that this embodiment does not perform frequency domain feature extraction on all the pronunciation segments obtained in step 200. Instead, it filters the candidate pronunciation segments by using candidate channels and candidate frequency bands. By filtering the candidate pronunciation segments by candidate channels, it selects EEG segments that can reflect the expression of language brain regions and removes data that is not related to language. Then, it performs a second screening by using the condition that the pronunciation frequency is within the candidate frequency band. It fuses and analyzes the language information that may be complementary and have different levels (such as phoneme recognition level, prosodic processing level, vocabulary retrieval level) carried by different frequency bands. At the same time, it can further verify the candidate pronunciation segments obtained in the previous steps and remove invalid data, thereby improving the accuracy of language decoding and shortening the decoding delay.

[0125] Step 400: Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal.

[0126] Preferably, in this embodiment, after directly capturing the original voltage signal of the candidate channel, the sampling rate is downsampled to a sampling rate aligned with the frequency domain frame, preferably from 1kHz to 100Hz. Then, the sampling result is normalized to the training set statistics. In this embodiment, the sampling rate is reduced to be aligned with the frequency domain frame to facilitate subsequent fusion and splicing with frequency domain features.

[0127] Step 500: Input the frequency domain feature into the frequency domain branch of the trained Pinyin decoding model, input the time domain feature into the time domain branch of the Pinyin decoding model, and output the initial consonant probability and final vowel probability corresponding to the target EEG signal.

[0128] The training of the pinyin decoding model includes collecting sample EEG signals from the subjects. In this embodiment, the scenario for collecting sample EEG signals from the subjects is as follows: each subject completes two experiments, and each experiment includes multiple sample EEG signals. The specific process includes:

[0129] Pre-resting period: 3 seconds, subjects relax, computer device screen displays "+", avoid verbal imagery;

[0130] Prompt period: 0.5 seconds, the target pinyin (e.g., "hao") is displayed on the screen;

[0131] Pronunciation period: 2 seconds, the subject silently reads the prompted pinyin, internally without making a sound;

[0132] Post-resting period: 3 seconds, the screen displays "+", and the subject relaxes;

[0133] The total acquisition time for each sample's EEG signal was approximately 8.5 seconds, with a rest period of approximately 5 minutes between different sampling sessions.

[0134] When training this pinyin decoding model, each sample pronunciation segment needs to be labeled and recorded as a triple, for example:

[0135]

[0136] in, and These represent sample pronunciation segments. The start and end times, in seconds. This indicates the pinyin annotation. This embodiment uses a dual-head annotation strategy for initials and finals, splitting the complete pinyin into two parts: initials and finals.

[0137] The initial consonant annotation includes 23 types of initial consonants and those without initial consonants. There are 24 classes in total:

[0138] ;

[0139] The final vowel annotation includes 24 categories of final vowels:

[0140] ;

[0141] Taking the pinyin "hao" as an example, it is marked as the initial consonant "h" and the final vowel "ao"; taking the pinyin "a" as an example, it is marked as the initial consonant. With the final "a".

[0142] Figure 4 This is a schematic diagram of the architecture of a Pinyin decoding model in one embodiment, such as... Figure 4 As shown, the Pinyin decoding model includes a frequency domain branch, a time domain branch, a feature fusion layer, a global average pooling layer, a consonant classification head, and a vowel classification head. The frequency domain branch includes the LSTM encoder included in the trained VAD speech detection model.

[0143] The initial consonant classification head sequentially passes the feature tensor output from the global average pooling layer through its first Dense fully connected layer, Dropout random deactivation layer, second Dense fully connected layer, and softmax logistic regression layer to output the probability distribution of the initial consonants. The first and second Dense fully connected layers have different input dimensions: the first has a feature dimension of 128, while the second has an input dimension of 24, representing the classification of the 23 initial consonants plus no initial consonant. It can be represented as:

[0144] (24 categories).

[0145] Preferably, the sub-loss of the initial consonant classification head is the cross-entropy loss function. Optional:

[0146]

[0147] in, Indicates the first N represents the total number of samples, and c represents the category of initial consonants. Indicates sample The true initial consonant label, Indicates sample The probability of predicting the initial consonant.

[0148] The network structure of this vowel classification head is the same as that of the initial consonant classification head, also including a first Dense fully connected layer, a Dropout random deactivation layer, a second Dense fully connected layer, and a softmax logistic regression layer, which are used to output the probability distribution of the vowels. The difference is that the input dimension of the second Dense fully connected layer of this vowel classification head is 24, representing the classification of 24 vowels, and the vowel categories. It can be represented as:

[0149] ;

[0150] Among them, the categories of finals It contains a total of 24 categories.

[0151] Similarly, the sub-loss of this vowel classification head—the cross-entropy loss function. Optional:

[0152]

[0153] In this formula, Indicates the first There are N samples, where N represents the total number of samples and c represents the category of vowels. Indicates sample The true vowel label, Indicates sample The probability of predicting the vowel.

[0154] Furthermore, this pinyin decoding model employs a multi-task learning strategy. Preferably, it also includes a joint optimization loss for the classification of initials and finals, the loss function of which is as follows:

[0155]

[0156] in As a balance coefficient, in this embodiment, the preferred value is... , Indicates the loss of classification of vowels. Represents the classification loss of initial consonants, when , and When all convergences are achieved, the trained Pinyin decoding model is obtained.

[0157] The pinyin decoding model classification head in this embodiment adopts a dual-head architecture, which outputs the probability distribution of initials and finals respectively. It can be understood that the initials with the highest probability distribution and the finals with the highest probability distribution can be selected as the initial and final probabilities corresponding to the target EEG signal.

[0158] The dataset output by this Pinyin decoding model can be represented by tuples, for example:

[0159]

[0160] in This indicates the first transcribed segment from the time or frequency domain of the sound segment. One sample of EEG signals, This indicates the number of EEG signals to be decoded. It is a double-headed tag.

[0161] Preferably, the training set, validation set, and test set for training the Pinyin decoding model are divided in a 7:1:2 ratio to ensure that data from the same subject is not leaked across sets.

[0162] Step 600: Convert the initial consonant probability and the final vowel probability into a Chinese character sequence using the trained language model.

[0163] In this embodiment, an n-gram language model is used to convert the initial and final probabilities into Chinese character sequences. This step specifically includes pinyin sequence construction, language model scoring, and Chinese character conversion. Specifically:

[0164] Pinyin sequence construction: For multi-character pronunciation, the pinyin of individual characters is concatenated into a sequence. ;

[0165] Language model scoring: The likelihood probability of the pinyin sequence is calculated using the following formula:

[0166]

[0167] in, Indicates the first N represents the total number of characters. The output probability of the n-gram language model is derived from statistics of a large-scale Chinese corpus of more than 1 million characters, representing the probability.

[0168] Chinese character conversion: The pinyin sequence is converted into a Chinese character sequence. The Viterbi Algorithm is used to find the optimal Chinese character path, which is mathematically represented as follows:

[0169]

[0170] in Indicates the sequence of candidate Chinese characters. Indicates to Each word The likelihood probability of forming the complete sequence reaches its maximum value. This represents the sequence of Chinese characters obtained after the conversion;

[0171] Output formatting: Add punctuation and format the converted Chinese character sequence to output the final text.

[0172] The language decoding method for EEG signals proposed in this embodiment, after cropping candidate pronunciation segments from the target EEG signal to be decoded, selects candidate pronunciation segments from these segments that correspond to the target EEG signal originating from candidate channels and whose pronunciation frequencies fall within candidate frequency bands. These candidate pronunciation segments are then used as target pronunciation segments. This initial screening of the target EEG signal, rather than using all channels, preserves brain regions sensitive to language tasks. Since language-related brain regions only cover a limited area of ​​the cerebral cortex, while the electrode arrays corresponding to each channel for acquiring target EEG signals typically cover a large area of ​​the cortical surface, including sensorimotor areas and visual areas unrelated to language, this method... The initial screening of the target EEG signal through candidate channels can eliminate a large number of invalid channels, reduce noise interference, and improve decoding efficiency and accuracy. On the other hand, this application uses the condition that the pronunciation frequency is within the candidate frequency band to perform a second screening of candidate pronunciation segments. This can select the frequency components with the most language discrimination power from many candidate pronunciation segments. It can fuse and analyze language information with different levels (such as phoneme recognition level, prosodic processing level, vocabulary retrieval level, etc.) that may be complementary carried by different frequency bands. At the same time, it can further verify the candidate pronunciation segments obtained in the previous steps, further improve the accuracy of language decoding, and shorten the decoding delay.

[0173] In one embodiment, Figure 2 This is a flowchart illustrating the steps for determining candidate channels in one embodiment, as shown below. Figure 2 As shown, the steps for determining the candidate channel include steps 301 to 303.

[0174] Step 301: Acquire the sample EEG signal collected through all channels. Based on the annotation of the sample EEG signal, extract the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment from the sample EEG signal.

[0175] In this embodiment, the total number of channels is 32, corresponding to 32 ECoG electrode arrays. The time-frequency amplitude sequence of the sound segment can be marked with "1" and the time-frequency amplitude sequence of the silence segment can be marked with "0".

[0176] Step 302: Calculate the similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels.

[0177] In one embodiment, calculating the similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silence segment in different channels includes:

[0178] The similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels is calculated using the following formula:

[0179]

[0180] in, Indicates the number of frequency bands. Indicates the total number of channels. Indicates the vocal tract. Indicates the mute channel. Indicates frequency, Indicates the vocal tract and mute channel The frequencies are the same The similarity metric is used in this embodiment. Indicates the number of frequency bands. This represents the total number of channels. The range is [0,1]. The higher the value, the more similar the channel is to the silent segment during pronunciation, meaning the smaller the change in neural activity and the lower the discrimination ability for language tasks;

[0181] The similarity measure between the vocal channel and the silent channel is determined using the following formula. :

[0182]

[0183] in, This indicates the result after normalization. Take the maximum value;

[0184] The cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels. Normalization is performed using the following formula:

[0185]

[0186] in, This represents the cyclic cross-correlation value. Indicates the time lag;

[0187] The cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels is calculated using the following formula:

[0188]

[0189] in, The time lag is represented by T, the total number of time segments, and t, which represents the t-th time segment. This represents the cyclic cross-correlation function. Indicates the vocal tract. Indicates frequency, Indicates the mute channel. Represents the time series of the spectral spectrum of the articulation segment; This represents the time series of the spectrum of the silent segment.

[0190] Step 303: Determine the candidate channel for the time-frequency amplitude sequence of the pronunciation segment whose similarity value is lower than the preset value. The preset value can be adaptively determined by binary K-means clustering.

[0191] The final candidate channels can be determined by performing binary K-means clustering, specifically as follows:

[0192] ;

[0193] ;

[0194] ;

[0195] in, This represents the similarity values ​​corresponding to the 32 channels. This represents a cluster of low-similarity channels, corresponding to task-sensitive candidate channels. Indicates high-similarity channel clusters, This represents the set of candidate channels to be retained, corresponding to the task-insensitive channels that need to be removed. In this embodiment, Finally, the top 11 channels with low similarity were selected, covering Broca's area, Wernicke's area and the adjacent superior temporal gyrus region.

[0196] Since some channels represent language pronunciation, indicating that they correspond to the language brain region, while others represent other brain activity areas such as motor and visual areas, or even if a channel itself corresponds to a language brain region, it is not sensitive to language tasks, the purpose of this embodiment is to filter out channels that are sensitive to language tasks and those that are not relevant from the 32 channels. Because channels with the same pronunciation or silence have high similarity and cannot be used as a reference, if the time-frequency amplitude sequence of the pronunciation segment of a channel has a high similarity value to the time-frequency amplitude sequence of the silence segment of other channels, it indicates that the channel does not represent language or is not sensitive to language tasks, and this channel is removed. This embodiment achieves preliminary screening of training samples by filtering out channels that can represent and transmit language signals. Whether subsequently using the selected sample EEG signals to train the pinyin decoding model or to decode the target EEG signals, the accuracy of the model and decoding can be improved.

[0197] In one embodiment, Figure 3 This is a flowchart illustrating the steps for determining candidate frequency bands in one embodiment, as shown below. Figure 3 As shown, the steps for determining the candidate frequency band include steps 401 to 408.

[0198] Step 401: Extract the frequencies carried in the time series of the phonological segment spectrum of the sample EEG signal transmitted through the candidate channel, and obtain the pinyin category of the phonological segment spectrum time series that has been pre-annotated.

[0199] The process of labeling the spectral time series of each pronunciation segment with pinyin categories includes labeling the initials and finals of the pinyin separately. The specific labeling methods have been illustrated earlier and will not be repeated here.

[0200] Step 402: Calculate the Pearson correlation coefficient between the time series of the spectral spectrum of the pronunciation segment that are the same as the pinyin category and the frequency and come from different candidate channels.

[0201] in, Indicates samples of the same type and The Pearson correlation coefficient between them is calculated using the following methods:

[0202] Collect Pinyin Categories In all trials, the candidate channels are k and the frequency is Time series:

[0203]

[0204] in Indicates the pinyin category as The number of samples, Indicates the first Time series of a sample Indicates the first The candidate channel to which each sample belongs. Indicates the first The frequency of each sample This represents a real number signal of dimension T.

[0205] Calculate similar samples using the following formula. and Pearson correlation coefficient between the time series:

[0206]

[0207] in, Indicates the first The candidate channel to which each sample belongs. Indicates the first The candidate channel to which each sample belongs. This indicates the calculation of covariance. This represents the standard deviation of the set of subscripts. Indicates frequency.

[0208] Step 403: Based on the calculated Pearson correlation coefficient, calculate the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin category is the same.

[0209] In one embodiment, calculating the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin categories are the same includes:

[0210] The intra-class synergy score is calculated using the following formula:

[0211]

[0212] in, Indicates the category of Pinyin. Indicates the pinyin category as The number of samples, Represents the set of candidate channels. Indicates different candidate channels. Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment This represents the calculated Pearson correlation coefficient. It should be noted that... and The source is all raw brain electrical signals. The data type is a two-dimensional tensor of channel number * time length, while The data type is a three-dimensional tensor of channel * frequency component * time slice. This can be understood as a sample... This refers to a recording of an electroencephalogram (EEG) signal that spans several seconds.

[0213] The purpose of this calculation is to cover all Candidate channel set, With all samples The average of the Pearson correlation coefficients is used to obtain the categories. Next candidate channel k, frequency Intra-class synergy score.

[0214] This embodiment analyzes each pinyin category. ( (various pinyin combinations), each candidate channel Each frequency Calculate the intra-class collaboration score.

[0215] Step 404: Take the average of the intra-class coherence scores for all different pinyin categories to obtain the final intra-class coherence score between the candidate channel and the corresponding frequency.

[0216] Specifically, the final score for intra-category synergy can be calculated by averaging across all pinyin categories using the following formula:

[0217]

[0218] in, The value range is [0,1]. The higher the value, the more consistent the response of the channel-frequency combination is in the same type of pinyin samples, and the more likely it is to be a stable marker of language-related neural activity.

[0219] Step 405: Construct a global redundancy matrix between the candidate channel and the frequencies carried in the spectral time series of the speech segment.

[0220] It is understandable that the global redundancy matrix is ​​a feature cell of a two-dimensional matrix that includes channels and frequencies.

[0221] Step 406: Calculate the redundancy distance between two different feature units in the global redundancy matrix that have different candidate channels but the same frequency.

[0222] Specifically, it can be applied to any two feature units. and Define its redundancy distance:

[0223]

[0224] in This represents the two feature units calculated using the above formula. The corresponding Pearson correlation coefficient of the time series. Preferably, to simplify calculations, redundancy is calculated only between feature units in channels with the same frequency, i.e. hour Meaningful Time setting .

[0225] Step 407: Based on the redundancy distance, calculate the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channels.

[0226] Specifically, for each feature unit Calculate the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channel, wherein the step of calculating the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channel based on the redundancy distance includes:

[0227] The average redundancy distance is calculated using the following formula:

[0228]

[0229]

[0230] in, Represents the set of all candidate feature units. Represents the set of candidate channels. Indicates the number of frequency bands. The candidate channel is represented. Indicates frequency, Indicates candidate channel as , frequency is The characteristic unit, Indicates candidate channel as , frequency is The characteristic unit, express , The Pearson correlation coefficient between two feature units The value range is [0,2]. The higher the value, the less redundant the unit is with other feature units, that is, the greater the signal difference.

[0231] Step 408: Determine the set of frequencies corresponding to the candidate channels whose final score of intra-class synergy is greater than the first threshold and whose average redundancy distance is greater than the second threshold as the candidate frequency band.

[0232] In this embodiment, the candidate frequency band can be selected using two-dimensional K-means clustering. Specifically, each candidate feature unit... Mapped to two-dimensional feature vectors:

[0233]

[0234] in, This represents the final score for intra-class collaboration. This represents the average redundancy distance.

[0235] For all Perform binary K-means clustering (K=2):

[0236]

[0237] in, This represents the cluster with high intra-class cohesion and large self-redundancy distance. This represents other clusters with low intra-class cooperativeness. The cluster with high intra-class cooperativeness and a large self-redundancy distance is selected. As the final feature subset, its corresponding channel-frequency combination can be expressed as:

[0238]

[0239] In this embodiment, It contains 56 feature points, with frequency bands mainly concentrated in the ranges of 4-8Hz, 8-13Hz, 30-50Hz and 70-150Hz, which is consistent with the neural oscillation mechanism of language processing.

[0240] To verify the effectiveness of feature selection, visual analysis can be performed on the features before and after selection. Specifically, there are three analysis methods:

[0241] By analyzing the channel distribution map, we found that the channels were concentrated in the left hemisphere language network (Broca's area, Wernicke's area) and removed channels were mainly distributed in the sensorimotor area, parietal lobe and other non-linguistic areas.

[0242] Analysis was conducted using frequency contribution heatmaps. By plotting the frequency distribution heatmaps of 56 selected features, it was observed that the feature density was highest in the high-frequency gamma band (70-150Hz) and the low-frequency theta band (4-8Hz), consistent with existing findings in language neuroscience.

[0243] Through inter-class separation analysis, we found that calculating the inter-class Fisher discriminant ratio of features before and after filtering in various pinyin categories verifies that the filtered features do indeed improve inter-class separability.

[0244] In one embodiment, the speech detection model includes a trained LSTM encoder, the frequency domain branch of the pinyin decoding model includes the LSTM encoder, and the method further includes:

[0245] The frequency domain features are extracted from the target speech segment using the trained LSTM encoder.

[0246] The frequency domain feature and the time domain feature are concatenated in the channel dimension through the feature fusion layer of the Pinyin decoding model to obtain the fused feature;

[0247] The initial consonant classification head of the Pinyin decoding model outputs the fused feature to obtain the probability of the initial consonant, and the final vowel classification head of the Pinyin decoding model outputs the fused feature to obtain the probability of the final vowel.

[0248] During feature concatenation, if the time dimensions of the outputs of the two branches are not perfectly aligned, such as the frequency domain branch... Time-domain branch Features after feature fusion By directly applying global average pooling, a fixed-dimensional vector is output, avoiding time alignment issues.

[0249] The frequency domain branch of the Pinyin decoding model provided in this embodiment reuses the LSTM encoder of the pre-trained VAD speech detection model as a feature extractor, with the input being the filtered time-frequency features. Its advantage lies in the fact that since the VAD speech detection model has already taught the LSTM encoder to distinguish between phonological and non-phonological temporal patterns during the pre-training stage, these temporal patterns also have discriminative value for Pinyin decoding. Therefore, the decoding performance can be significantly improved through transfer learning.

[0250] To evaluate the cross-subject generalization ability of the language decoding method for EEG signals provided in this embodiment, the present invention also conducted LOSO (Leave-One-Subject-Out) cross-validation, where trials represent the number of samples, wherein:

[0251] Training set: 9 subjects × 400 trials = 3600 trials

[0252] Validation set: 9 subjects × 50 trials = 450 trials

[0253] Test set: 1 subject × 500 trials = 500 trials

[0254] During the data collection, 500 trials were collected from each subject. Under the LOSO settings, 500 trials from one subject were reserved as the test set, and from the remaining nine subjects, 400 trials were used for training and 50 trials were used for validation. The remaining trials were either discarded or used for model selection.

[0255] LOSO cross-validation was performed on each of the 10 participants, and the results are shown in Table 1 below:

[0256] Table 1. LOSO Cross-Validation Results

[0257]

[0258] This embodiment also conducted multiple ablation experiments to verify the contribution of each module of the Pinyin decoding model. The verification results are shown in Table 2 below:

[0259] Table 2. Schematic diagram of ablation verification results

[0260]

[0261] The experimental results in Table 2 show that:

[0262] 1. The dual-branch fusion improves performance by approximately 8-12 percentage points compared to the single-branch fusion, demonstrating the complementarity of time-domain and frequency-domain features;

[0263] 2. The two-head classification improved by about 5 percentage points compared with the single-head classification, proving that the decoupling of initials and finals is effective;

[0264] 3. Feature-based screening improved performance by approximately 15 percentage points compared to no screening, demonstrating the significant effectiveness of the two-stage screening mechanism.

[0265] Under the LOSO setting, the average pinyin accuracy rate of 10 subjects was 87.49%; under a fixed training / validation / test division, the complete two-branch model achieved a pinyin accuracy rate of 88.1%. The above generalization experiments among subjects demonstrate that the language decoding method for EEG signals proposed in this invention can maintain a pinyin decoding accuracy rate of over 80% across different subjects, proving the effectiveness and robustness of this application. Furthermore, with a larger number of subjects and fewer channels, the language decoding latency of this application is only 180ms, compared to the 290ms~420ms of existing technologies, exhibiting higher decoding accuracy and lower latency, representing a significant improvement.

[0266] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0267] Based on the same inventive concept, this application also provides a language decoding device for EEG signals to implement the language decoding method for EEG signals described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the language decoding device for EEG signals provided below can be found in the limitations of the language decoding method for EEG signals described above, and will not be repeated here.

[0268] In one exemplary embodiment, such as Figure 5 As shown, a language decoding device 100 for EEG signals is provided, including a time period detection module 11, a cropping module 12, a frequency domain feature extraction module 13, a time domain feature extraction module 14, a probability output module 15, and a conversion module 16.

[0269] The time period detection module 11 is used to acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal.

[0270] The trimming module 12 is used to trim the target EEG signal according to the pronunciation time period to obtain at least one candidate pronunciation segment.

[0271] The frequency domain feature extraction module 13 is used to select candidate pronunciation segments from the at least one candidate pronunciation segments whose corresponding target EEG signals originate from candidate channels and whose pronunciation frequencies are within candidate frequency bands, as target pronunciation segments, and extract frequency domain features based on the target pronunciation segments.

[0272] The time-domain feature extraction module 14 is used to acquire the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal.

[0273] The probability output module 15 is used to input the frequency domain feature into the frequency domain branch of the trained Pinyin decoding model, input the time domain feature into the time domain branch of the Pinyin decoding model, and output the initial consonant probability and final vowel probability corresponding to the target EEG signal.

[0274] The conversion module 16 is used to convert the initial consonant probability and the final vowel probability into a Chinese character sequence through the trained language model.

[0275] In one embodiment, the language decoding device 100 for the EEG signal further includes:

[0276] The sample signal acquisition module is used to acquire sample EEG signals collected through all channels. Based on the annotation of the sample EEG signals, the time-frequency amplitude sequences of the vocal segment and the time-frequency amplitude sequences of the silent segment from each channel are extracted from the sample EEG signals.

[0277] The first calculation module is used to calculate the similarity value between the time-frequency amplitude sequence of the pronunciation segment and the time-frequency amplitude sequence of the silence segment in different channels;

[0278] The candidate channel determination module is used to determine the channel corresponding to the time-frequency amplitude sequence of the pronunciation segment with a similarity value lower than a preset value.

[0279] In one embodiment, the first computing module specifically includes a first computing unit, a second computing unit, a third computing unit, and a fourth computing unit, specifically:

[0280] The first calculation unit is used to calculate the similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels using the following formula:

[0281]

[0282] in, Indicates the number of frequency bands. Indicates the total number of channels. The 'j' indicates the vocal channel, and the 'j' indicates the silence channel. Indicates frequency, Indicates the vocal tract The frequencies of the mute channel j and the mute channel j are the same. Similarity metric at different times.

[0283] The second calculation unit is used to determine the similarity measure between the pronunciation channel and the silence channel using the following formula. :

[0284]

[0285] in, This indicates the result after normalization. Take the maximum value.

[0286] The third calculation unit is used to calculate the cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silence segment in different channels. Normalization is performed using the following formula:

[0287]

[0288] in, This represents the cyclic cross-correlation value. Indicates the time lag.

[0289] The fourth calculation unit is used to calculate the cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silence segment in different channels using the following formula:

[0290]

[0291] in, The time lag is represented by T, the total number of time segments, and t, which represents the t-th time segment. This represents the cyclic cross-correlation function. Indicates the vocal tract. Indicates frequency, Indicates the mute channel. Represents the time series of the spectral spectrum of the articulation segment; This represents the time series of the spectrum of the silent segment.

[0292] In one embodiment, the language decoding device 100 for the EEG signal further includes a pinyin category acquisition module, a second calculation module, a third calculation module, a fourth calculation module, a matrix construction module, a fifth calculation module, a sixth calculation module, and a candidate frequency band determination module, specifically:

[0293] The Pinyin category acquisition module is used to extract the frequencies carried in the spectral time series of the sample EEG signal transmitted through the candidate channel and obtain the Pinyin category of the spectral time series ...

[0294] The second calculation module is used to calculate the Pearson correlation coefficient between the time series of the spectral spectrum of the pronunciation segment that are of the same pinyin category and frequency and originate from different candidate channels.

[0295] The third calculation module is used to calculate the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin category is the same, based on the calculated Pearson correlation coefficient.

[0296] The fourth calculation module is used to average the intra-class coherence scores of all different pinyin categories to obtain the final intra-class coherence score between the candidate channel and the corresponding frequency.

[0297] The matrix construction module is used to construct a global redundancy matrix between the candidate channel and the frequencies carried in the spectral time series of the speech segment.

[0298] The fifth calculation module is used to calculate the redundancy distance between two different feature units in the global redundancy matrix that have different candidate channels but the same frequency.

[0299] The sixth calculation module is used to calculate the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channels based on the redundancy distance.

[0300] The candidate frequency band determination module is used to determine the set of frequencies corresponding to candidate channels whose final score of intra-class cooperation is greater than a first threshold and whose average redundancy distance is greater than a second threshold as the candidate frequency band.

[0301] In one embodiment, the third calculation module is specifically used to calculate the intra-class synergy score using the following formula:

[0302]

[0303] in, Indicates the category of the pinyin. Indicates the pinyin category as The number of samples, Represents the set of candidate channels. Indicates different candidate channels. Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment This represents the calculated Pearson correlation coefficient.

[0304] The sixth calculation module is specifically used to calculate the average redundancy distance using the following formula:

[0305]

[0306]

[0307] in, , Represents the set of candidate channels. Indicates the number of frequency bands. The candidate channel is represented. Indicates frequency, Indicates candidate channel as , frequency is The characteristic unit, Indicates candidate channel as , frequency is The characteristic unit, express , The Pearson correlation coefficient between two feature units.

[0308] In one embodiment, the speech detection model includes a trained LSTM encoder, the frequency domain branch of the pinyin decoding model includes the LSTM encoder, and the language decoding device for the EEG signal further includes a time period detection module, a pruning module, a frequency domain feature extraction module, a time domain feature extraction module, a probability output module, and a conversion module. Specifically:

[0309] The time period detection module is used to acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal.

[0310] The trimming module is used to trim the target EEG signal according to the pronunciation time period to obtain at least one candidate pronunciation segment;

[0311] The frequency domain feature extraction module is used to select candidate pronunciation segments from the at least one candidate pronunciation segments whose corresponding target EEG signals originate from candidate channels and whose pronunciation frequencies are within candidate frequency bands, as target pronunciation segments, and to extract frequency domain features based on the target pronunciation segments;

[0312] The time-domain feature extraction module is used to acquire the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal.

[0313] The probability output module is used to input the frequency domain feature into the frequency domain branch of the trained Pinyin decoding model, input the time domain feature into the time domain branch of the Pinyin decoding model, and output the initial consonant probability and final vowel probability corresponding to the target EEG signal.

[0314] The conversion module is used to convert the initial consonant and final vowel probabilities into Chinese character sequences using the trained language model.

[0315] Each module in the aforementioned language decoding device for EEG signals can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0316] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a language decoding method for electroencephalogram (EEG) signals. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0317] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0318] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0319] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0320] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0321] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0322] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0323] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0324] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0325] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0326] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0327] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0328] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0329] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0330] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0331] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0332] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0333] Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal;

[0334] The target EEG signal is cropped based on the pronunciation time period to obtain at least one candidate pronunciation segment;

[0335] From the at least one candidate vocal segment, the candidate vocal segment whose corresponding target EEG signal originates from the candidate channel and whose vocal frequency is within the candidate frequency band is selected as the target vocal segment, and frequency domain features are extracted based on the target vocal segment;

[0336] Obtain the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal;

[0337] The frequency domain feature is input into the frequency domain branch of the trained Pinyin decoding model, and the time domain feature is input into the time domain branch of the Pinyin decoding model. The probability of the initial consonant and the probability of the final vowel corresponding to the target EEG signal are output.

[0338] The trained language model converts the initial consonant and final vowel probabilities into Chinese character sequences.

[0339] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0340] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0341] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0342] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for language decoding of electroencephalogram (EEG) signals, characterized in that, The method includes: Acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal; The target EEG signal is cropped according to the pronunciation time period to obtain at least one candidate pronunciation segment; Candidate pronunciation segments whose corresponding target EEG signals originate from candidate channels and whose pronunciation frequencies are within candidate frequency bands are selected from the at least one candidate pronunciation segments and used as target pronunciation segments. Frequency domain features are then extracted based on the target pronunciation segments. Obtain the original voltage signal of the candidate channel, and extract time-domain features based on the original voltage signal; The frequency domain features are input into the frequency domain branch of the trained Pinyin decoding model, and the time domain features are input into the time domain branch of the Pinyin decoding model. The probability of initial consonant and the probability of final vowel corresponding to the target EEG signal are output. The trained language model converts the initial consonant probabilities and final vowel probabilities into Chinese character sequences.

2. The method according to claim 1, characterized in that, The steps for determining the candidate channel include: Acquire sample EEG signals collected through all channels, and extract the time-frequency amplitude sequences of the vocal segment and the time-frequency amplitude sequences of the silent segment from the sample EEG signals based on the annotation of the sample EEG signals; Calculate the similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels; The candidate channels are determined by identifying the channels corresponding to the time-frequency amplitude sequences of the pronunciation segments with similarity values ​​lower than a preset value.

3. The method according to claim 2, characterized in that, The calculation of the similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels includes: The similarity value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels is calculated using the following formula: in, Indicates the number of frequency bands. Indicates the total number of channels. The 'j' indicates the vocal channel, and the 'j' indicates the silence channel. Indicates frequency, Indicates the vocal tract The frequencies of the mute channel j and the mute channel j are the same. Similarity measure at different times; The similarity measure between the vocal channel and the silent channel is determined by the following formula. : in, This indicates the result after normalization. Take the maximum value; The cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels. Normalization is performed using the following formula: in, This represents the cyclic cross-correlation value. Indicates the time lag; The cyclic cross-correlation value between the time-frequency amplitude sequence of the vocal segment and the time-frequency amplitude sequence of the silent segment in different channels is calculated using the following formula: in, The time lag is represented by T, the total number of time segments, and t, which represents the t-th time segment. This represents the cyclic cross-correlation function. Indicates the vocal tract. Indicates frequency, Indicates the mute channel. Represents the time series of the spectral spectrum of the articulation segment; This represents the time series of the spectrum of the silent segment.

4. The method according to claim 2, characterized in that, The steps for determining the candidate frequency band include: Extract the frequencies carried in the time series of the phonological segment spectrum of the sample EEG signal transmitted through the candidate channel, and obtain the pinyin category of the phonological segment spectral time series that has been pre-annotated; Calculate the Pearson correlation coefficient between the time series of the pronunciation segments whose pinyin category and frequency are the same and originate from different candidate channels; Based on the calculated Pearson correlation coefficient, calculate the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin categories are the same; The average of the intra-class coherence scores for all different pinyin categories is used to obtain the final intra-class coherence score between the candidate channel and the corresponding frequency. Construct a global redundancy matrix between the candidate channel and the frequencies carried in the spectral time series of the speech segment; Calculate the redundancy distance between two different feature units in the global redundancy matrix that have different candidate channels but the same frequency; Based on the redundancy distance, calculate the average redundancy distance between each feature unit and the feature units of all other candidate channels at the same frequency; The frequency set corresponding to the candidate channels whose final intra-class synergy score is greater than the first threshold and whose average redundancy distance is greater than the second threshold is determined as the candidate frequency band.

5. The method according to claim 4, characterized in that, The calculation of the intra-class coherence score between the candidate channel and the frequency of the corresponding pinyin category when the pinyin categories are the same includes: The intra-class synergy score is calculated using the following formula: in, Indicates the category of the pinyin. Indicates the pinyin category as The number of samples, Represents the set of candidate channels. Indicates different candidate channels. Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment Indicates the pinyin category as , frequency is And through the candidate channel The first transmission A time series of the spectral spectrum of each articulation segment This represents the calculated Pearson correlation coefficient; The step of calculating the average redundancy distance between each feature unit and all other feature units of the same frequency candidate channels based on the redundancy distance includes: The average redundancy distance is calculated using the following formula: in, , Represents the set of candidate channels. Indicates the number of frequency bands. The candidate channel is represented. Indicates frequency, Indicates candidate channel as , frequency is The characteristic unit, Indicates candidate channel as , frequency is The characteristic unit, express , The Pearson correlation coefficient between two feature units.

6. The method according to any one of claims 1 to 5, characterized in that, The speech detection model includes a trained LSTM encoder, the frequency domain branch of the pinyin decoding model includes the LSTM encoder, and the method further includes: The frequency domain features are extracted from the target speech segment by the trained LSTM encoder; The frequency domain features and the time domain features are concatenated along the channel dimension by the feature fusion layer of the Pinyin decoding model to obtain the fused features; The initial consonant classification head of the Pinyin decoding model is used to output the fused features to obtain the initial consonant probability, and the final vowel classification head of the Pinyin decoding model is used to output the fused features to obtain the final vowel probability.

7. A language decoding device for electroencephalogram (EEG) signals, characterized in that, include: The time period detection module is used to acquire the target EEG signal, input the target EEG signal into the trained speech detection model, and output the pronunciation time period of the target EEG signal. The trimming module is used to trim the target EEG signal according to the pronunciation time period to obtain at least one candidate pronunciation segment; The frequency domain feature extraction module is used to select candidate pronunciation segments from the at least one candidate pronunciation segments, whose corresponding target EEG signals originate from candidate channels and whose pronunciation frequencies are within candidate frequency bands, as target pronunciation segments, and to extract frequency domain features based on the target pronunciation segments; The time-domain feature extraction module is used to acquire the original voltage signal of the candidate channel and extract time-domain features based on the original voltage signal. The probability output module is used to input the frequency domain features into the frequency domain branch of the trained Pinyin decoding model, input the time domain features into the time domain branch of the Pinyin decoding model, and output the initial consonant probability and final vowel probability corresponding to the target EEG signal. The conversion module is used to convert the initial consonant probabilities and final vowel probabilities into Chinese character sequences using the trained language model.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the language decoding method for EEG signals as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the language decoding method for EEG signals as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the language decoding method for EEG signals according to any one of claims 1 to 6.

Images