Chinese character phrase and sentence decoding method based on stereotactic EEG signals

Abstract

This invention discloses a method for decoding Chinese character phrases and sentences based on stereotactic electroencephalogram (sEEG) signals. This method uses stereotactic electroencephalogram (sEEG) signals as system input to decode Chinese phonemes, achieving greater generalization and adaptability than syllables. After identifying the initials, finals, and tones that make up Chinese Pinyin (Pinyin has 23 initials and no initials, 24 finals, and five tones), this method performs a two-way correction process using a language model modeled by Bayesian probabilistic modeling and a large language model. The decoded and corrected phrases or sentences are then displayed in real time on a user interface.

Description

Technical Field

[0001] The present invention relates to the technical field of computer information processing, and in particular to a method for decoding Chinese characters, phrases and sentences based on stereotactic electroencephalogram signals. Background Art

[0002] Over the past few decades, signal processing and artificial intelligence algorithm technologies in computer information technology have evolved from serving mathematical science and industry to integrating with hardware scenarios in many aspects of social life, gradually empowering end-users to meet their needs more closely. Electrophysiology is a scientific computing field that has gradually leveraged computer technology and is experiencing increasing development. Electrophysiological signal analysis and its application research and development have evolved from early physical methods to simple equipment assistance, then to the use of computers to collect and process raw electrophysiological signals for analysis, and finally to the comprehensive application of computer information science, which utilizes high-performance information processing algorithms and mathematical tools for information analysis. The use of advanced algorithms, information processing technologies, and computing infrastructure for the processing, calculation, analysis, and application of physiological signals has become a key scientific development direction in this century. Furthermore, more advanced learning-based algorithmic models have enabled the efficient execution of complex electrophysiological signal analysis tasks, thus generating more substantial value for human society and providing a new technological foundation for the rehabilitation and treatment of neurodegenerative diseases. Summary of the Invention

[0003] The purpose of the present invention is to provide a Chinese character and phrase sentence decoding method based on stereotactic EEG signals to address the shortcomings of existing syllable decoding and cortical EEG signal decoding technology.

[0004] The objective of the present invention is achieved through the following technical solutions: a method for decoding Chinese character phrases and sentences based on stereotactic electroencephalogram signals, comprising: taking stereotactic electroencephalogram signals as input, using a neural network model as a vocalization detector to determine whether a neurolinguistic action is generated; then taking stereotactic electroencephalogram signals generated by neurolinguistic actions as input, using a deep neural network model to output the predicted probabilities of various types of initials, finals and tones, thereby determining the initials, finals and tones; finally, using a Bayesian statistical inference model or a large language model to obtain a Chinese character phrase or sentence.

[0005] Based on the predicted probabilities of the determined initials and finals, the perplexity of the pinyin classification is calculated, and then combined with the perplexity threshold, it is determined to use either the Bayesian statistical inference model or the large language model.

[0006] The neural network model is trained using stereotactic electroencephalogram signals marked with voice and silence; and the deep neural network model is trained using stereotactic electroencephalogram signals marked with initials, finals and tones.

[0007] Furthermore, it also includes: using a digital filter to perform real-time noise reduction processing on the stereotactic EEG signal to filter out the power frequency component and high-frequency noise signal; and performing statistical analysis to obtain the mean and standard deviation of each channel, and standardizing all signals to form a standardized sEEG data frame.

[0008] Furthermore, the voicing detector is used to characterize the probability that the current sEEG signal is in a voicing state;

[0009] Set the starting threshold and ending threshold, and then combine the probability of the sEEG signal being in the vocalization state to determine the input of the deep neural network model; specifically:

[0010] The probability of detecting that the sEEG signal is in the sound state is greater than the starting threshold. The current sEEG signal is used as the starting point until the probability of the sEEG signal in the sound state is less than the ending threshold or the signal sequence reaches the longest allowed value. The sEEG signal of this segment is used as the input of the deep neural network model.

[0011] Furthermore, it also includes: for the generated initials and finals, a similar pronunciation merging method similar to the fuzzy pinyin principle is used to include similar initials and finals in the initial and final tone candidate list to form multiple possible initial and final combinations as input to the Bayesian statistical inference model or the large language model.

[0012] Furthermore, based on the determined prediction probabilities of the initials and finals, determining to use one of a Bayesian statistical inference model and a large language model includes:

[0013] Use the information entropy formula to calculate the entropy values ​​of each type of initials and finals, and add them up to get the total entropy; if the total entropy is lower than the perplexity threshold p, use the Bayesian statistical inference model; otherwise use the large language model; or

[0014] Based on the probability of each pinyin, the perplexity is calculated using the complete perplexity formula algorithm. If the perplexity is lower than the perplexity threshold p, the Bayesian statistical inference model is used; otherwise, the large language model is used. Among them, the probability of each pinyin is calculated by the three classification results of its highest probability initial consonant, final vowel, and tone.

[0015] Furthermore, the perplexity threshold p is determined by calculating the formula max(H(S)+H(Y)+H(D)) / 2, where H(S), H(Y), and H(D) represent the classification logits information entropy of initials, finals, and tones, respectively.

[0016] Furthermore, the Bayesian statistical inference model uses the patient's daily language corpus to count the frequency of words, words, and short sentences to form a probability distribution statistic for words and sentences. Multiple pinyin result sequences composed of alternative lists of initials, finals, and tones are input into the Bayesian statistical inference model as sentences. The input sentences use the similarity of the oral muscle movements of the pinyin as the edit distance measure. Based on this measure, the closest feasible sentence is found and statistical inference is performed to obtain a Chinese character phrase or sentence.

[0017] Large language model: The alternative list of initials, finals, and tones is placed in a cache sequence. The pinyin of each character in the cache sequence is ordered to form a pinyin sequence. Here, each Chinese pinyin position has several alternative pinyins. The recording rule of the alternative pinyins is: take the n initials and n finals with the highest probability, and combine them into no more than n2 pinyin results. Then, the pinyin sequence with the highest probability and each remaining alternative pinyin are inserted into the instruction template of the large language model. Through the instruction template and system prompt words, the large language model is required to output the words, phrases, and sentences corresponding to the pinyins, and correct possible incorrect pinyins.

[0018] Furthermore, the neural network model includes an input layer, a multi-layer resnet block, a cnn classification layer and an activation layer which are connected in sequence;

[0019] The deep neural network model is a trunk + three-branch end-to-end model; the trunk part includes an input layer, a multi-layer resnet block and a cnn classification layer and is connected in sequence; each branch part includes an input layer, a two-layer resnet block, a cnn block, a cnn classification layer, a global avg pooling and an activation layer and is connected in sequence.

[0020] The present invention has the beneficial effect of more effectively decoding neurolinguistic activity by leveraging the rich channels of sEEG stereotactic EEG data. Furthermore, the use of both a language model and a large language model derived from Bayesian probabilistic modeling further improves the accuracy of the decoded correction results. Furthermore, the minimally invasive nature of sEEG stereotactic EEG in clinical practice surpasses common invasive EEG methods with window-type incisions, reducing the risks and costs in these applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a method for real-time decoding of Chinese characters, phrases and sentences based on stereotactic electroencephalogram signals. DETAILED DESCRIPTION

[0022] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0023] To enable streaming, and particularly real-time, decoding of Chinese language from EEG in practical applications, this paper proposes and designs several novel mechanisms to improve the effectiveness and performance of Chinese EEG decoding within brain-computer interfaces. This paper uses stereotactic EEG (sEEG) signals as input to increase the accuracy of the system and algorithm's perception of brain activity. Causal modeling techniques are employed to achieve rapid decoding responses, while phoneme decoding improves generalization and adaptability. A Bayesian probabilistic language model is employed in conjunction with a large oracle model to collaboratively optimize phrase and sentence parsing. The use of a large language model further enhances the accuracy of sentence representation.

[0024] The embodiment of the present invention provides a real-time decoding method for Chinese characters, phrases and sentences based on stereotactic electroencephalogram (sEEG) signals, such as Figure 1 As shown, the following steps are included:

[0025] Step 1: Obtain stereotactic EEG data, each data point is x∈R C The signal vector becomes an EEG data frame, where C is the number of channels, which is commonly 50 to 150 channels. A real-time digital filter is used to perform real-time noise reduction on the signal frame by frame, filtering out the 50 Hz and 100 Hz power frequency components, and removing high-frequency noise signals above 250 Hz.

[0026] Step 2: Perform statistical analysis on the sEEG signal processed in step 1, accumulate 10-20 minutes of EEG data, and form a continuous data matrix X of size (C×T), where C is the number of channels and T is the number of accumulated EEG data frames. Calculate the mean μ of each channel along the T dimension. c and standard deviation σ c , where c represents the channel number. Afterwards, all existing data and each newly acquired frame of data are normalized: each channel is subtracted from its own mean μ c , and divided by their respective standard deviations σ c , forming the processed feature data x'.

[0027] Step 3: The standardized feature x' generated by the stereo EEG is sent to the vocalization detector frame by frame for recognition and detection. The x' signal can accumulate 1 to multiple frames, and the vocalization detector (this embodiment uses a neural network model as a vocalization detector) supports processing, and the common setting is 1 to 20 frames. When the accumulated x' signal is smaller than the convolution kernel size of the network model, the insufficient length is filled with 0. The neural network model outputs a floating-point probability value each time, which represents the probability that the EEG is in a vocalization state at this moment. The structure of the neural network model is a multi-layer resnet block + cnn classification layer, see Table 1:

[0028] Table 1: Neural network model structure

[0029]

[0030] The neural network model is trained by first collecting sufficient sEEG data and recordings of vocal and silent periods. The sEEG data is segmented according to vocal and silent states. After segmentation, each segment of the sEEG data is filtered and normalized to obtain input signal features. 1 is used as the data label for vocal segments, and 0 is used as the label for silent segments. After data balancing, the neural network is trained on the labeled data using a sigmoid cross entropy and weight decay loss function, with a weight decay factor of 0.001. The Adam optimizer is used, and the learning rate is initially set to 0.01 and reduced tenfold every 2000 steps to a minimum of 0.0001.

[0031] Step 4: When the probability of a stereo EEG signal being vocalized is detected to be greater than the set starting threshold, the subsequent stereo EEG feature signals are saved and sent to the language decoding algorithm module. This probability value is determined based on the model's ROC / AUC characteristics and is typically set around 0.95, indicating a probability of 95% or greater that a neurological speech action has occurred. When the probability of vocalization falls below the set termination threshold, the transmission is truncated, typically set to 80%. If the duration of a vocalization exceeds the maximum allowed value, the transmission is also triggered, and the vocalization is considered complete. The typical timeout setting is 10 seconds.

[0032] Step 5: After receiving the entire segment of stereoscopic EEG feature data representing the vocalization process, the language decoding algorithm module will be fed into a deep neural network for processing and generate three predicted probability results (initials, finals, and tones). The decoding uses the complete Chinese pinyin as a multi-task classification element, including 24 initial consonants (23 initials and no initials), 24 finals, and five tones (four tones and no tones in Chinese). This network model is a backbone + multi-branch end-to-end model. The backbone structure is shown in Table 2:

[0033] Table 2: Main structure

[0034]

[0035] Frames in the backbone structure can be of any length, but 4096 is used as an example here.

[0036] The hidden features generated by the backbone structure will be sent to three branch networks simultaneously to predict initials, finals, and tones. The network structures of initials and finals are shown in Table 3:

[0037] Table 3: Network structure of initials and finals

[0038]

[0039] The above branch network predicts 23 initials + zero initials, a total of 24 types of initials. The final prediction network structure is the same.

[0040] The tone prediction network structure is shown in Table 4. The final output is 5 categories, corresponding to the four tones of Chinese and no tone;

[0041] Table 4: Tone prediction network structure

[0042]

[0043] The multi-branch language decoding neural network is trained as follows: sEEG data is collected in advance, and the pinyin of the pronunciation portion (initials, finals, and tones) of each EEG data segment is recorded. The EEG data is segmented, retaining the vocalized portion. The data is then filtered and normalized to generate feature data. Based on the pronunciation labels, the start and end times of each word's pronunciation are determined. The feature data is segmented according to these labels, and each segmented feature corresponds to the sEEG brain activity of a single word. Each word slice is annotated with its initials, finals, and tones as labels. The neural network is trained using this data using a loss function consisting of a sotfmax cross entropy between the initials, finals, and tones, and their respective labels, plus a weight decay loss. The loss coefficients for each of the three branches are 1 / 3, and the weight decay loss coefficient is 0.001. The optimizer is ADAM, with an initial learning rate of 0.01, which is reduced tenfold every 4000 steps. The minimum learning rate is capped at 0.00001.

[0044] Step six: For the generated initials and finals (the initials and finals are obtained based on the maximum probability predicted in step five), a similar pronunciation merging method similar to the fuzzy pinyin principle is used to include similar initials and finals in the "similar sounds" alternative list to form multiple possible initial-final combinations, so as to facilitate error correction and updating when forming words and sentences in the future. One feasible way is the fuzzy pinyin approximation method, which classifies flat and retroflex tongue pronunciations as approximate sounds, thereby forming an initial consonant approximation table. At the same time, n / l, r / l, m / n, g / k are also listed as approximate and added to the approximate matrix of initials and finals to form the final approximate matrix. For example, when a pinyin of an initial consonant and final is obtained, n is expanded to "n, m, x", and "iu" is expanded to "iu / ou" to form a cross-combination alternative pinyin list: "xiu / mou"

[0045] Step 7: This step is based on the degree of certainty of the initial and final classification results (the probability obtained in step 5) and is divided into two branches: a correction process based on the traditional Bayesian statistical inference model and a correction process based on the large language model. The specific implementation method of Step 7 is as follows:

[0046] (7.1) The classification results for initials and finals are vectors of length 24 and 24, respectively. The initial classification contains 24 floating-point values, representing the classification confidence of 23 initials and no initials, a total of 24 initials. The final classification contains 24 floating-point values, representing the same meaning. The entropy values ​​of the initial and final classifications are calculated using the information entropy formula and added together to obtain the total entropy. The formula used for information entropy (Shannon entropy) is as follows:

[0047]

[0048] Among them, P(x) is the probability of classification result X=x

[0049] (7.2) A perplexity threshold p is used to determine the level of total entropy. If the total entropy is lower than p, the current list of candidate initials, finals, and tones is sent to the Bayesian statistical inference correction module. The threshold p is estimated by cumulatively adding up the highest values ​​of information entropy. A typical threshold is max(H(S)+H(Y)+H(D)) / 2, where H(S), H(Y), and H(D) represent the categorical logits information entropy of initials, finals, and tones, respectively. A common value is 3.8. The Bayesian statistical inference model uses the patient's daily speech corpus to calculate word frequency, term frequency, and short sentence frequency, forming a word and sentence probability distribution. The word library includes the end-of-line (EOL) symbol. Each time, several possible single words are obtained and stored in the current sentence cache. Multiple possible combinations of each word position in the current sentence cache are combined to form multiple possible sentences. The sentence is then statistically inferred using a statistical model using the phonetic edit distance measure to find the closest feasible sentence. The phonetic edit matrix distance is defined based on the similarity of oral muscle movements. When the next character is inferred to be a text rather than an EOL, the current inferred sentence is output to the next step. If the next most likely character is inferred to be an EOL, the sentence is output in full, the buffer is cleared, and preparations are made to record the next decoded Chinese character.

[0050] In one embodiment, the perplexity complete formula algorithm is used to determine whether to use a Bayesian statistical inference algorithm or a large language model; the specific calculation formula is as follows:

[0051]

[0052] The method for estimating P(w0,w1,…,wN) in this algorithm is as follows:

[0053]

[0054] Where P(wi) is the classification prediction probability of each pinyin in the sentence string. The probability of each pinyin is calculated by the three classification results of its highest probability initial consonant, final vowel, and tone. The formula is as follows:

[0055]

[0056] Where P(S)P(Y)P(D) are the highest probabilities of initials, finals and tones respectively. When the complete perplexity formula is used as the basis for judgment, the perplexity threshold is determined by If the acceptable lower limit of the probability of each word classification is set to 50%, and the product of the probability and the perplexity threshold is 1, then the perplexity threshold p is calculated to be 2.

[0057] If the confusion If the perplexity is lower than the threshold p, the Bayesian statistical inference model is used; otherwise, the large language model is used.

[0058] (7.3) If the total entropy is higher than the p threshold specified above, the list of candidate initials, finals, and tones will be sent to the large language model correction module. The method of using the large language model is to put the list of candidate initials, finals, and tones into the cache sequence. The pinyin of each word in the cache sequence is arranged in order to form a pinyin sequence. Here, each Chinese pinyin position has several alternative pinyins. The recording rule of the alternative pinyins is: take the n initials and n finals with the highest probability (for example, n=2), which can be combined into no more than n2 pinyin results. Then, the pinyin sequence with the highest probability and each remaining alternative pinyin are inserted into the instruction template of the large language model. Through the instruction template and the system prompt words, the large language model is required to output the words, phrases, and sentences corresponding to the pinyin, and correct possible incorrect pinyins;

[0059] A feasible system prompt might be, "You are a professional Chinese language professor and dialect expert. Based on an ambiguous pinyin sequence, you will attempt to identify the most likely sentence behind it. In addition to the most likely pinyin for each word, several alternative pinyins may be provided. Please fully consider the possibility of misspellings, mispronunciations, and inaccurate dialect pronunciations. Analyze the most fluent possible pinyin sequence, correct the pinyin errors, and output the sentence the speaker most likely intended to say. If the sentence is complete, please output a period at the end. If the sentence is incomplete, please do not output a period."

[0060] A feasible instruction template is "The following is the pinyin corresponding to a paragraph. These pinyins may contain errors or dialect pronunciations. Please analyze the most likely correct sentence. The pinyin sequence with the highest probability is given below: wo wan lechu yao. In addition to this pinyin sequence, each character may also have the following pronunciations: wa / ho / ha, han / wang / hang, ne / nou / lou, chi / cu / ci, xiao / xia / ya. Please fully analyze and find the pinyin that truly corresponds to each character, and output the sentence composed of this pinyin. There is no need to output anything else. The sentence that this pinyin sequence most likely corresponds to is: ".

[0061] If the sentence is complete, a period or EOL tag is printed at the end and the buffered word sequence is cleared. If the sentence is incomplete, the correction result of the incomplete sentence is output without outputting the sentence end symbol.

[0062] Step 8: Output the words or sentences obtained from the above output to the display interface of the brain-computer interface in real time. When the output sentence ends with EOL, it can be decided whether to call the TTS function to read the complete sentence according to the user's choice.

[0063] The present invention also provides a Chinese character phrase and sentence decoding device based on stereotactic electroencephalogram signals, comprising:

[0064] A vocalization detection module is used to take the stereotactic EEG signal as input and use the neural network model as a vocalization detector to determine whether a neurolinguistic action has occurred;

[0065] The language decoding algorithm module is used to take the stereotactic EEG signal generated by neural language movements as input, use a deep neural network model to output the predicted probability of various types of initials, finals and tones, and thus determine the initials, finals and tones;

[0066] The dual-path correction module includes a Bayesian statistical inference correction module and a large language model correction module. By using the Bayesian statistical inference correction module or the large language model correction module, a Chinese character phrase or sentence is obtained; based on the predicted probabilities of the determined initial consonants and finals, one of the Bayesian statistical inference correction module and the large language model correction module is determined to be used.

[0067] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment. You can refer to the content of the above method embodiment and will not repeat it here.

[0068] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A method for decoding Chinese character phrases and sentences based on stereotactic electroencephalogram signals, characterized in that: include: Stereotactic EEG signals are fed into a vocalization detector frame by frame or by accumulating one or more frames for recognition and detection. A neural network model is used as the vocalization detector to determine whether a neurolinguistic action has occurred. The neural network model has a multi-layer ResNet block + CNN classification layer structure. The vocalization detector is used to characterize the probability that the current sEEG signal is in a vocalization state. Set the starting and ending thresholds, and then combine the probability of the sEEG signal being in the vocalization state to determine the input of the deep neural network model; Then, using the stereotactic EEG signals generated by neurolinguistic movements as input, a deep neural network model is used to output the predicted probabilities of various initials, finals, and tones, thereby determining the initials, finals, and tones. Finally, a Bayesian statistical inference model or a large language model is used to obtain Chinese character phrases or sentences. Based on the predicted probabilities of the determined initials and finals, determine to use one of a Bayesian statistical inference model and a large language model, including: Use the information entropy formula to calculate the entropy values ​​of each type of initials and finals, and add them up to get the total entropy; if the total entropy is lower than the perplexity threshold p, use the Bayesian statistical inference model; otherwise use the large language model; or Based on the probability of each pinyin, the perplexity is calculated using the complete perplexity formula algorithm. If the perplexity is lower than the perplexity threshold p, the Bayesian statistical inference model is used; otherwise, the large language model is used. Among them, the probability of each pinyin is calculated by the three classification results of its highest probability initial consonant, final vowel, and tone.

2. The method according to claim 1, characterized in that Also includes: For stereotactic EEG signals, a digital filter is used to perform real-time noise reduction processing to filter out power frequency components and high-frequency noise signals; statistical analysis is performed to obtain the mean and standard deviation of each channel, and all signals are standardized to form standardized sEEG data frames.

3. The method according to claim 1, characterized in that The phonation detector is used to characterize the probability that the current sEEG signal is in a phonation state; a starting threshold and an ending threshold are set, and then combined with the probability that the sEEG signal is in a phonation state, the input of the deep neural network model is determined; specifically: The probability of detecting that the sEEG signal is in the sound state is greater than the starting threshold. The current sEEG signal is used as the starting point until the probability of the sEEG signal in the sound state is less than the ending threshold or the signal sequence reaches the longest allowed value. The sEEG signal of this segment is used as the input of the deep neural network model.

4. The method according to claim 1, wherein Also includes: For the generated initials and finals, a similar pronunciation merging method similar to the fuzzy pinyin principle is used to include similar initials and finals in the alternative list of initial and final tones, forming multiple possible initial and final combinations as input to the Bayesian statistical inference model or large language model.

5. The method according to claim 1, wherein The perplexity threshold p is determined by calculating the formula max(H(S)+H(Y)+H(D)) / 2, where H(S), H(Y), and H(D) represent the classification logits information entropy of initials, finals, and tones, respectively.

6. The method according to claim 1 or 4, characterized in that Bayesian statistical inference model: Using the patient's daily language corpus to count word frequency, word frequency, and short sentence frequency, a statistical probability distribution of words and sentences is formed. Multiple pinyin result sequences composed of alternative lists of initials, finals, and tones are input into the Bayesian statistical inference model as sentences. The input sentences use the similarity of the oral muscle movements of the pinyin as the edit distance measure. Based on this measure, the closest feasible sentence is found and statistical inference is performed to obtain the Chinese character phrase or sentence. Large language model: put the candidate list of initials, finals and tones into the cache sequence. The pinyin of each word in the cache sequence is sequentially composed into a pinyin sequence. Here, each Chinese pinyin position has several alternative pinyins. The recording rule of the alternative pinyins is: take the n initials and n finals with the highest probability, and combine them into no more than n 2 The system then inserts the pinyin sequence with the highest probability and each remaining candidate pinyin into the instruction template of the large language model. Through the instruction template and system prompt words, the large language model is required to output the words, phrases and sentences corresponding to the pinyin and correct possible incorrect pinyin.

7. The method according to claim 1, characterized in that The neural network model includes an input layer, a multi-layer resnet block, a cnn classification layer and an activation layer which are connected in sequence; The deep neural network model is a trunk + three-branch end-to-end model; the trunk part includes an input layer, a multi-layer resnet block and a cnn classification layer and is connected in sequence; each branch part includes an input layer, a two-layer resnet block, a cnn block, a cnn classification layer, a global avg pooling and an activation layer and is connected in sequence.

8. The method according to claim 1, characterized in that The method is implemented by the following steps: Step 1: Obtain stereotactic EEG data and use a digital filter to perform real-time noise reduction to remove power frequency components and high-frequency noise signals; Step 2: Perform statistical analysis on the sequence signal processed in step 1 to obtain the mean and standard deviation of each channel, and standardize all signals to form a standardized sEEG data frame x'; Step 3: The standardized sEEG data frames are fed into the sound detector for recognition and detection. The x' signal can be accumulated for 1 to multiple frames. The sound detector outputs a floating-point probability value each time, representing the probability that the EEG is in the sound state at that moment; Step 4: When the probability of the stereo EEG signal being in the vocal state is detected to be greater than the set starting threshold, the subsequent stereo EEG signals are saved until the probability of vocalization decreases below the set ending threshold, or the vocalization duration exceeds the maximum allowed value; Step 5: After receiving the entire segment of stereoscopic EEG feature data representing the vocalization process saved in step 4, it will be sent to the deep neural network for processing and generate three prediction probability results, namely initial consonant, final vowel, and tone; Step 6: For the generated initials and finals, a similar pronunciation merging method similar to the fuzzy pinyin principle is used to add similar initials and finals to the candidate list of initials, finals and tones, forming multiple possible initial-final combinations to facilitate error correction and update when subsequently forming words and sentences; Step 7: This step is based on the initial and final prediction probability results in step 5 and is divided into two branches: a correction process based on the Bayesian statistical inference model and a correction process based on the large language model; The following sub-steps are included: (7.1) The classification results of initials and finals are vectors of length 24 and 24 respectively. The initial classification contains 24 floating-point values, representing the classification confidence of 23 initials and no initials, a total of 24 initials; the final classification contains 24 floating-point values, representing the classification confidence of 24 finals; Use the information entropy formula to calculate the entropy values ​​of the initial and final categories, and add them up to get the total entropy to replace the perplexity, or directly use the perplexity complete formula algorithm to calculate the perplexity; (7.2) If the perplexity is lower than the perplexity threshold p, the current list of candidate initials, finals, and tones is sent to the Bayesian statistical inference model. The Bayesian statistical inference model uses the patient's daily language corpus to count word frequency, word frequency, and short sentence frequency to form a probability distribution of words and sentences. The resulting multiple pinyin sequences composed of the candidate initials, finals, and tones are input into the Bayesian statistical inference model as sentences. The input sentences use the similarity of the oral muscle movements of the pinyin as the edit distance measure. The Bayesian statistical inference model finds the closest feasible sentence based on this measure and performs statistical inference. When it is inferred that the next character is a text rather than an EOL, the current inferred sentence is output to the next step. If the next most likely character is inferred to be EOL, the sentence is output completely, the buffer is cleared, and preparations are made to record the next decoded Chinese character. (7.3) If the perplexity is higher than p, the candidate list of initials, finals, and tones is sent to the large language model. The large language model is used as follows: the candidate list of initials, finals, and tones is placed in a cache sequence. The pinyin of each word in the cache sequence is sequentially composed into a pinyin sequence. Here, each Chinese pinyin position has several candidate pinyins. The recording rule of the candidate pinyins is: take the n initials and n finals with the highest probability, and combine them into no more than n 2 The system then inserts the pinyin sequence with the highest probability and each remaining candidate pinyin into the instruction template of the large language model. Through the instruction template and the system prompt words, the large language model is required to output the words and sentences corresponding to the pinyin and correct possible incorrect pinyin. If the sentence is complete, a period or EOL label is printed at the end, and the cached word sequence is cleared at the same time. If the sentence is incomplete, the correction result of the incomplete sentence is output without outputting the sentence end symbol. Step 8: Output the words or sentences obtained in step 7 to the display interface of the brain-computer interface in real time. When the output sentence ends with EOL, it can be decided whether to call the TTS function to read the complete sentence according to the user's choice.

9. A Chinese character phrase and sentence decoding device based on stereotactic electroencephalogram signals, characterized in that: include: The vocalization detection module is used to take the stereotactic EEG signal as input and send it to the vocalization detector frame by frame or by accumulating one or more frames for recognition and detection. A neural network model is used as the vocalization detector to determine whether a neurolinguistic action has occurred. The neural network model has a multi-layer ResNet block + CNN classification layer structure. The vocalization detector is used to represent the probability that the current sEEG signal is in a vocalization state. Set the starting and ending thresholds, and then combine the probability of the sEEG signal being in the vocalization state to determine the input of the deep neural network model; The language decoding algorithm module is used to take the stereotactic EEG signal generated by neural language movements as input, use a deep neural network model to output the predicted probability of various types of initials, finals and tones, and thus determine the initials, finals and tones; A dual-path correction module, including a Bayesian statistical inference correction module and a large language model correction module, obtains a Chinese character phrase or sentence by using the Bayesian statistical inference correction module or the large language model correction module; determines to use one of the Bayesian statistical inference correction module and the large language model correction module based on the predicted probabilities of the determined initial consonants and finals, including: Use the information entropy formula to calculate the entropy values ​​of each type of initials and finals, and add them up to get the total entropy; if the total entropy is lower than the perplexity threshold p, use the Bayesian statistical inference model; otherwise use the large language model; or Based on the probability of each pinyin, the perplexity is calculated using the complete perplexity formula algorithm. If the perplexity is lower than the perplexity threshold p, the Bayesian statistical inference model is used; otherwise, the large language model is used. Among them, the probability of each pinyin is calculated by the three classification results of its highest probability initial consonant, final vowel, and tone.

Images