Neural signal decoding method, brain-computer system, electronic equipment and storage medium

Abstract

The invention discloses a neural signal decoding method, a brain-computer system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a tested neural signal; determining the language state of the subject according to the neural signal and a preset language state detection model; determining a pre-trained language decoding model according to the language state; and decoding the neural signal by using the language decoding model to obtain a language decoding result. The technical problem that the neural signals corresponding to different language states cannot be decoded in real time when facing the neural signals corresponding to different language states is solved.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and more specifically, to a neural signal decoding method, a brain-computer interface system, an electronic device, and a storage medium. Background Technology

[0002] Intracranial neural signal recording technology plays a vital role in neuroscience research and clinical applications due to its high signal-to-noise ratio and unique ability to provide information about specific brain regions. Neural signals are widely used in epilepsy detection, neurological disease diagnosis, and basic neuroscience research. However, the complexity and high-dimensionality of neural signals pose challenges to the development of real-time brain-computer interface (BCI) systems, particularly for high-precision language decoding tasks. Current decoding systems are primarily optimized for single language states, such as decoding only spoken or imagined reading. These systems lack real-time adaptability to different language states (spoken reading, imagined reading), limiting their flexibility and versatility in practical applications.

[0003] In summary, existing technologies have the technical problem of being unable to decode neural signals corresponding to different language states in real time. To address the aforementioned problem, no effective solution has yet been proposed. Summary of the Invention

[0004] This invention provides a neural signal decoding method, a brain-computer interface, an electronic device, and a storage medium to at least solve the technical problem of being unable to decode neural signals corresponding to different language states in real time.

[0005] According to one aspect of the present invention, a neural signal decoding method is provided, comprising: acquiring neural signals of a subject; determining the subject's language state based on the neural signals and a preset language state detection model; determining a pre-trained language decoding model based on the language state; and decoding the neural signals using the language decoding model to obtain a language decoding result.

[0006] Optionally, the language state includes reading aloud. Based on the language state, determining the pre-trained language decoding model includes: in response to the language state being reading aloud, determining the first language state weights corresponding to the reading aloud; loading the first language state weights into the pre-built initial model to determine the language decoding model.

[0007] Optionally, the language state includes silent reading. Based on the language state, determining the pre-trained language decoding model includes: in response to the language state being silent reading, determining the second language state weights corresponding to the silent reading; loading the second language state weights into the pre-built initial model to determine the language decoding model.

[0008] Optionally, decoding the neural signals using a language decoding model to obtain the language decoding result includes: obtaining a decoding target, wherein the decoding target is used to guide the type of decoding result output by the language decoding model; and decoding the neural signals using the language decoding model according to the decoding target to obtain the language decoding result.

[0009] Optionally, depending on the decoding target, the neural signal is decoded using a language decoding model to obtain the language decoding result, including: in response to the decoding target being a word sequence, the neural signal is decoded using a language decoding model based on a first weight to obtain the language decoding result, wherein the first weight is a pre-loaded weight trained for a finite set of words and sentences.

[0010] Optionally, depending on the decoding target, the neural signal is decoded using a language decoding model to obtain the language decoding result, including: in response to the decoding target being a phoneme sequence, the neural signal is decoded using a language decoding model based on a second weight to obtain the language decoding result, wherein the second weight is a pre-loaded weight trained on the phoneme set.

[0011] Optionally, the language decoding model is used to decode the neural signal to obtain the language decoding result, which includes: decoding the neural signal using the language decoding model to obtain an initial decoding result, wherein the initial decoding result includes a word sequence or a phoneme sequence; and inputting the initial decoding result into a preset language model to obtain the language decoding result.

[0012] According to another aspect of the present invention, a neural signal decoding system is also provided, comprising: an acquisition module for acquiring neural signals of a subject; a first determination module for determining the language state of the subject based on the neural signals and a preset language state detection model; a second determination module for determining a pre-trained language decoding model based on the language state; and a decoding module for decoding the neural signals using the language decoding model to obtain a language decoding result.

[0013] According to another aspect of the present invention, a brain-computer interface system is also provided, comprising: a neural signal acquisition device for acquiring neural signals of a subject; a decoder for determining the subject's language state based on the neural signals and a preset language state detection model; determining a pre-trained language decoding model based on the language state; decoding the neural signals using the language decoding model to obtain a language decoding result; and an output component for outputting the language decoding result.

[0014] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of the present invention during runtime.

[0015] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0016] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0017] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0018] According to another aspect of the present invention, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of the present invention.

[0019] In this embodiment of the invention, the neural signals of the subject are acquired; the subject's language state is determined based on the neural signals and a preset language state detection model; a pre-trained language decoding model is determined based on the language state; and the neural signals are decoded using the language decoding model to obtain the language decoding result. This invention first determines the subject's language state, and then determines the corresponding language decoding model based on the language state. This enables real-time decoding of neural signals under different language states, making the decoding of neural signals more flexible and adaptable. This solves the technical problem of being unable to perform real-time decoding of neural signals corresponding to different language states. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0021] Figure 1 This is a flowchart of a neural signal decoding method provided according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating a neural signal decoding method according to an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of a neural signal decoding system provided according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to an embodiment of the present invention, a neural signal decoding method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] Figure 1 This is a method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0028] Step S101: Obtain the neural signals of the subject.

[0029] It should be noted that the embodiments of the present invention illustrate the neural signal decoding method based on neural signals recorded by sEEG electrodes. Practical applications of the present invention can also include decoding invasively recorded brain neural signals, including ECoG and spike signals.

[0030] In this step, the system first acquires real-time brain activity signals from the subject using an invasive recording device (such as sEEG). These signals are non-stationary time-series data containing brain activity patterns of the subject in different language tasks.

[0031] Optionally, to ensure signal quality, physiological noise suppression and signal enhancement processing may be performed during the acquisition process.

[0032] Step S102: Determine the subject's language state based on neural signals and a pre-set language state detection model.

[0033] The system uses a pre-trained language state detection model to identify whether the subject is currently engaged in language-related activities, such as reading aloud or silently (or imaginative reading). The language state detection model, based on features of sEEG signals, or ECoG and spike signals, is trained to distinguish neural activity patterns in different language states. Once a neural signal is input, the model can instantly calculate and determine whether the subject is reading aloud, silently (imaginatively), or in a non-verbal state. This step guides the loading of the correct language decoding model, as brain activity patterns differ fundamentally in different language states.

[0034] Step S103: Determine the pre-trained language decoding model based on the language state;

[0035] Based on the language state identified in step S102, the system selects a suitable language decoding model from the pre-trained model library, or loads the weights corresponding to the language state to obtain a suitable language decoding model for the current language state. Depending on the subject's language state (e.g., reading aloud or silently (imagine reading aloud)), the system loads the corresponding language decoding model, which has been trained for that specific language state. For example, if the language state is reading aloud, the system loads the language decoding model trained for that state; if the language state is silent (imagine reading aloud), the system loads the language decoding model trained for that state.

[0036] Step S104: Use the language decoding model to decode the neural signal to obtain the language decoding result.

[0037] After the corresponding language decoding model is determined in step S103, the neural signal is input into the language decoding model, and the language decoding model can output the corresponding language decoding result.

[0038] Understandably, regardless of the subject's language state, the language decoding model will produce a language decoding result, which can be a specific list of words or a coherent stream of text. This result reflects the subject's current speech content and can be used in various applications, such as assistive communication devices and brain-controlled text input, greatly enhancing the possibilities and flexibility of human-computer interaction.

[0039] In this embodiment of the invention, the neural signals of the subject are acquired; the subject's language state is determined based on the neural signals and a preset language state detection model; a pre-trained language decoding model is determined based on the language state; and the neural signals are decoded using the language decoding model to obtain the language decoding result. This invention first determines the subject's language state, and then determines the corresponding language decoding model based on the language state. This enables real-time decoding of neural signals under different language states, making the decoding of neural signals more flexible and adaptable. This solves the technical problem of being unable to perform real-time decoding of neural signals corresponding to different language states.

[0040] Optionally, the language state includes reading aloud. In step S103, determining the pre-trained language decoding model based on the language state may include the following steps: in response to the language state being reading aloud, determining the first language state weights corresponding to the reading aloud; loading the first language state weights into the pre-built initial model to determine the language decoding model.

[0041] It should be noted that before proceeding to step S103, real-time intracranial neural signal preprocessing is performed, and a preset language state detection model is used to determine whether the current reading state is reading aloud.

[0042] The system confirms that the current language state is reading aloud. From the pre-trained language decoding model weight library, it identifies a set of weights that match the reading-aloud state, known as the "first language state weights." These first language state weights are obtained from a large amount of training data on reading aloud and represent the optimal parameter configuration for the language decoding model when decoding reading aloud.

[0043] The system pre-builds an initial model with a general framework and parameters, but it has not been trained for a specific task or state. When the system detects that the language state is reading aloud, it loads the "first language state weights" into this initial model. This process is achieved by updating the weight matrix in the model, ensuring that the model's parameters reflect the optimal conditions required for decoding brain signals in the reading-aloud state.

[0044] Optionally, the language state includes silent reading (imaginary reading). In step S103, determining the pre-trained language decoding model based on the language state includes: in response to the language state being silent reading (imaginary reading), determining the second language state weights corresponding to the silent reading; loading the second language state weights into the pre-built initial model to determine the language decoding model.

[0045] The system confirms the current language state as silent reading (imaginary reading). From the pre-trained language decoding model weight library, it identifies a set of weights that match the silent reading state, known as the "second language state weights." These second language state weights are obtained from a large amount of silent reading training data and represent the optimal parameter configuration for the language decoding model when decoding silent reading (imaginary reading).

[0046] The system pre-builds an initial model with a general framework and parameters, but it has not been trained for a specific task or state. When the system detects a language state of silent reading (imaginary reading), it loads "second language state weights" into this initial model. This process is achieved by updating the weight matrix in the model, ensuring that the model's parameters reflect the optimal conditions required for decoding brain signals in the silent reading (imaginary reading) state.

[0047] Optionally, in step S104, decoding the neural signal using the language decoding model to obtain the language decoding result may include the following steps: obtaining the decoding target, wherein the decoding target is used to guide the type of decoding result output by the language decoding model; and decoding the neural signal using the language decoding model according to the decoding target to obtain the language decoding result.

[0048] The decoding target refers to the type of language information the system expects to decode from intracranial neural signals. It can be a specific word, a complete sentence, or a continuous sequence of phonemes. For example, if a user wants to identify words being read aloud or silently in real time, then the decoding target is a "word sequence"; if the user's goal is to generate continuous speech content, then the decoding target is a "phoneme sequence".

[0049] Optionally, in the system, the selection of the decoding target can be done through a user interface, allowing users to set it according to their needs. Furthermore, in some optional embodiments, the system can automatically infer a suitable decoding target based on the environment or user habits.

[0050] Once the decoding target is determined, the system will call the corresponding pre-trained model for real-time decoding. The specific implementation is as follows:

[0051] (a) Model initialization

[0052] As mentioned earlier, the system loads specific pre-trained weights into the initial model based on the detected language state (such as reading aloud) and the selected decoding target (such as word sequence), forming a language decoding model that is perfectly suited to the current task.

[0053] (b) Neural signal decoding

[0054] Using a language decoding model loaded with specific weights, the system begins decoding the neural signals received in real time. Specifically, the neural signal decoding model processes the input sEEG signal, extracting features relevant to the decoding target through spatial-temporal encoding, vector quantization, and masking modeling strategies.

[0055] For example, the decoding target is set to identify words read aloud by the user, and the current language state has been detected as reading aloud:

[0056] Target setting: The user or system determines that the decoding target is a "word sequence".

[0057] Model loading: In response to the decoding target, the system selects language decoding model weights from the pre-trained model library that are specifically designed for spoken reading and word recognition tasks.

[0058] Real-time signal processing: Receives and preprocesses real-time sEEG signals.

[0059] Feature extraction and decoding: The neural signal decoding model encodes and decodes the preprocessed signal based on the specific weights loaded, and outputs feature embeddings related to word recognition.

[0060] Word recognition: Feature embedding identifies words read aloud in real time through a pre-set word classification network.

[0061] Through the above steps, the system can not only accurately decode the brain signals of the user when reading aloud, but also flexibly adjust the form of the decoding results according to the user's actual needs. Whether it is words, sentences or phoneme sequences, they can be presented in real time and accurately.

[0062] Optionally, depending on the decoding target, the neural signal is decoded using a language decoding model to obtain the language decoding result, including: in response to the decoding target being a word sequence, the neural signal is decoded using a language decoding model based on a first weight to obtain the language decoding result, wherein the first weight is a pre-loaded weight trained for a finite set of words and sentences.

[0063] During the real-time decoding phase, the system first needs to identify the current decoding target. When the decoding target is set to a word sequence, this means the system's goal is to decode the received brain neural activity into the specific words the user is reading aloud or silently. This target is typically used for decoding finite sets of words, such as identifying words in a specific vocabulary list that the user is reading aloud or silently.

[0064] Understandably, the decoding objective is set based on the user's expressive needs.

[0065] The first weights refer to the model weights pre-trained on a large amount of intracranial neural signal data related to words read aloud (or silently). These weights are trained and optimized to effectively convert neural signals from specific brain regions into language information, making them suitable for decoding tasks with limited word sets. In the system, the first weights are stored in the model database and are selected and loaded into the initial model when the decoding target is determined to be a word sequence, becoming part of the neural signal decoding model.

[0066] After loading the first weight, the real-time received intracranial neural signals are decoded. The process includes:

[0067] Signal preprocessing: The real-time received sEEG signal is preprocessed, including filtering and resampling, to improve signal quality.

[0068] Feature extraction: The neural signal decoding model extracts features related to word recognition.

[0069] Word classification: The model's output embeddings are processed by a pre-trained word classification network (usually a multilayer perceptron, MLP), which maps the embeddings to words in a finite set of words and phrases, thereby identifying the words that the user is thinking about or reading aloud.

[0070] To improve the accuracy and coherence of the decoding results, the system can employ an N-Gram language model to correct the identified word sequence. Based on the statistical probabilities of words in the sequence, the N-Gram model can predict the next most likely word, thus helping the system correct potential recognition errors and ensuring the grammatical and semantic correctness of the output.

[0071] Finally, the system will use the word sequence identified by the word classification network, combined with the correction from the N-Gram language model, to generate the final language decoding result. This result is presented in word order, reflecting the language content during the user's real-time reading or thinking process.

[0072] Optionally, depending on the decoding target, the neural signal is decoded using a language decoding model to obtain the language decoding result, including: in response to the decoding target being a phoneme sequence, the neural signal is decoded using a language decoding model based on a second weight to obtain the language decoding result, wherein the second weight is a pre-loaded weight trained on the phoneme set.

[0073] During the real-time decoding phase, the system first needs to identify the current decoding target. When the decoding target is set to a phoneme sequence, this means that the system's goal is to decode the received brain neural activity into the sentences that the user is reading aloud or silently.

[0074] The second weights refer to model weights pre-trained on a large amount of intracranial neural signal data related to spoken (or silent) sentences, based on phoneme sets. These weights are trained and optimized to effectively convert neural signals from specific brain regions into language information, making them suitable for handling phoneme-based decoding tasks. In the system, the second weights are stored in the model database and are selected and loaded into the initial model when the decoding target is determined to be a phoneme sequence, becoming part of the neural signal decoding model.

[0075] After loading the second weight, the real-time received intracranial neural signals are decoded, and the process includes:

[0076] Signal preprocessing: The real-time received sEEG signal is preprocessed, including filtering and resampling, to improve signal quality.

[0077] Feature extraction: The neural signal decoding model extracts features related to phoneme recognition.

[0078] Phoneme Decoding: Building upon feature extraction, the neural signal decoding model employs a specific TokenPrediction Head, combined with CTC (Connectivist Temporal Classification) technology, to predict the most likely phoneme or blank label at each time point. The model not only identifies individual phonemes but also understands the sequence relationships between them, a crucial component of phoneme sequence recognition.

[0079] To improve the accuracy and consistency of the decoding results, the system can employ an N-Gram language model to correct the recognized phoneme sequence. Based on the statistical probability of words in the sequence, the N-Gram model can predict the next most likely phoneme in the sequence, thereby helping the system correct potential recognition errors and ensuring the accuracy of the output.

[0080] Finally, the system will generate the final language decoding result by using a unique phoneme sequence and correcting it with an N-Gram language model. This result is presented in the form of sentences (including Chinese words, short sentences, phrases, and long sentences), reflecting the language content of the user during real-time reading or thinking.

[0081] Optionally, the language decoding model is used to decode the neural signal to obtain the language decoding result, which includes: decoding the neural signal using the language decoding model to obtain an initial decoding result, wherein the initial decoding result includes a word sequence or a phoneme sequence; and inputting the initial decoding result into a preset language model to obtain the language decoding result.

[0082] The real-time system receives intracranial neural signals (such as sEEG signals) and performs necessary preprocessing to improve signal quality.

[0083] Depending on the decoding target (word sequence or phoneme sequence recognition), the system selects and loads the corresponding neural signal decoding model weights. If the decoding target is a word sequence, the system loads the first weight; if it is a phoneme sequence, it loads the second weight.

[0084] The neural signal decoding model performs deep learning processing on the processed signals, extracts features from different brain regions through a spatial encoder, and then combines time series processing, using a Transformer encoder or similar structure to capture dynamic changes over time.

[0085] Sequence prediction: Based on the feature representation processed by the neural signal decoding model, the model predicts word sequences or phoneme sequences through classification or regression heads. If the target is a word sequence, the model will directly output the most likely word sequence; if it is a phoneme sequence, the output is a phoneme-level probability distribution, which needs to be converted into the actual phoneme sequence later.

[0086] The model's predicted output needs to be converted into an easily understandable and processable format. For word sequence recognition, this directly outputs a list of words; for phoneme sequence recognition, the predicted probability distribution is converted into the corresponding phoneme sequence.

[0087] Language model selection: The system pre-sets language models, such as the N-Gram model, to enhance the coherence and grammatical correctness of the decoding results. These models are trained on a large amount of natural language data and are able to understand language structure and contextual relationships.

[0088] Input sequence: The initial decoding result (word sequence or phoneme sequence) obtained from the neural signal decoding model is used as input and fed into the preset language model.

[0089] Syntax and semantic checks: The pre-defined language model checks the syntactic and semantic coherence of the decoding results and corrects any errors or unnatural expressions that may occur.

[0090] Probability scoring: The model scores each potential decoding sequence according to language rules and selects the sequence with the highest probability and correct syntax as the final result.

[0091] The optimized decoding results are output in real time as word sequences or phoneme sequences. Depending on the needs, these results may be further converted into speech or text for display by users or researchers.

[0092] Optionally, in some embodiments of the present invention, reference is made to... Figure 2 The neural signal decoding method is implemented as follows:

[0093] The system first receives neural signals (sEEG signals) collected from electrodes implanted inside the brain. These signals are records of the electrophysiological activity generated by the brain during language activities and form the basis for subsequent language state detection and decoding.

[0094] The received raw signal needs to be preprocessed, including filtering to remove noise, resampling to ensure time synchronization, and possible rereference or baseline correction, in order to improve signal quality and facilitate subsequent analysis.

[0095] The preprocessed signal is sent to the real-time language state detection module. Through deep learning models, such as convolutional neural networks or recurrent neural networks, the system can identify the user's current language state, such as "reading aloud" or "reading silently".

[0096] Based on the detected language state, the system will switch to the corresponding decoding target and model. For example, if the detected state is a spoken reading, the system will prepare to decode words or phoneme sequences.

[0097] Based on the detected language state and the user-selected decoding target (word sequence or phoneme sequence), the system calls the pre-trained language decoding model and the corresponding weights (first weight or second weight).

[0098] If the user selects a limited number of words and sentences as the decoding target (such as 60 commonly used Chinese words or 20 commonly used Chinese sentences), the system will use a pre-trained language decoding model to perform the word classification task.

[0099] The language decoding model decodes real-time neural signals based on pre-loaded weights, predicting the most likely word at each time point using a multilayer perceptron (MLP). The output word sequence may require further grammatical and semantic correction using an N-Gram language model to ensure the accuracy and coherence of the decoding results.

[0100] If the user selects open set sentence generation as the decoding target, the system will use a pre-trained language decoding model to classify and generate continuous phoneme units.

[0101] The Du-IN model performs phoneme-level decoding of the real-time signal based on the loaded second weights. Using CTC (Connectionist Temporal Classification) technology in conjunction with a Transformer encoder, the model generates a continuous sequence of phonemes. This phoneme sequence is then corrected and optimized using an N-Gram language model to generate coherent speech or text output.

[0102] Whether it's word sequence classification or phoneme sequence generation, the decoding results are optimized for syntax and semantics through the N-Gram language model to ensure that the output language results conform to the natural laws of human language.

[0103] Output component: The optimized language decoding results are output to the user interface in real time, which may be played in the form of speech synthesis or displayed on the screen as text for users or observers to use or provide feedback immediately.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0105] This invention also provides a neural signal decoding system for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0106] Figure 3 This is a structural block diagram of a neural signal decoding system according to one embodiment of the present invention, such as... Figure 3 As shown, the system includes: an acquisition module 301 for acquiring the neural signals of the subject; a first determination module 302 for determining the language state of the subject based on the neural signals and a preset language state detection model; a second determination module 303 for determining a pre-trained language decoding model based on the language state; and a decoding module 304 for decoding the neural signals using the language decoding model to obtain the language decoding result.

[0107] Optionally, the language state includes reading aloud, and the second determining module 303 is further configured to: in response to the language state being reading aloud, determine the first language state weight corresponding to the reading aloud; load the first language state weight into the pre-built initial model, and determine the language decoding model.

[0108] Optionally, the language state includes silent reading (imaginary reading), and the second determining module 303 is further configured to: in response to the language state being silent reading (imaginary reading), determine the second language state weight corresponding to the silent reading; load the second language state weight into the pre-built initial model, and determine the language decoding model.

[0109] Optionally, the decoding module 304 is further configured to: obtain a decoding target, wherein the decoding target is used to guide the type of decoding result output by the language decoding model; and decode the neural signal using the language decoding model according to the decoding target to obtain the language decoding result.

[0110] Optionally, the decoding module 304 is further configured to: in response to the decoding target being a word sequence, decode the neural signal using a language decoding model based on a first weight to obtain a language decoding result, wherein the first weight is a pre-loaded weight trained on a finite set of words and sentences.

[0111] Optionally, the decoding module 304 is further configured to: in response to the decoding target being a phoneme sequence, decode the neural signal using a language decoding model based on a second weight to obtain a language decoding result, wherein the second weight is a pre-loaded weight trained on the phoneme set.

[0112] Optionally, the decoding module 304 is further configured to: decode the neural signal using a language decoding model to obtain an initial decoding result, wherein the initial decoding result includes a word sequence or a phoneme sequence; and input the initial decoding result into a preset language model to obtain a language decoding result.

[0113] This invention also provides a brain-computer interface system, comprising: a neural signal acquisition unit for acquiring neural signals of a subject; a decoder for determining the subject's language state based on the neural signals and a preset language state detection model; determining a pre-trained language decoding model based on the language state; decoding the neural signals using the language decoding model to obtain a language decoding result; and an output component for outputting the language decoding result.

[0114] The neural signal acquisition device is responsible for acquiring implantable or non-implantable neural signals, including but not limited to sEEG (scanned electrical signals). It can capture the weak electrical signals generated by the brain during language activities, providing raw data for subsequent processing.

[0115] Acquisition devices typically include multiple high-precision electrodes that cover key language regions of the brain, ensuring signal integrity and accuracy. Real-time streaming data acquisition technology ensures signal continuity and real-time performance.

[0116] The decoder is the brain of the system, responsible for determining the subject's current language state (such as reading aloud, reading silently, or reading aloud in imagination) based on the acquired neural signals and the preset language state detection model.

[0117] Language state detection models use deep learning techniques, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), to classify and identify language activity types based on the features of real-time neural signals. This model can identify specific neural patterns corresponding to different language states, providing a basis for selecting appropriate decoding strategies.

[0118] Based on the language state determined by the decoder, the system automatically selects and activates a pre-trained language decoding model to decode neural signals into language information. The model is trained to decode neural signals for specific language states.

[0119] Pre-trained language decoding models are trained on finite sets of words or phonemes. These models capture the complex relationship between neural signals and language output and optimize decoding performance by adjusting model weights. Each model is optimized for a different decoding target (word sequences or phoneme sequences) to ensure accuracy and coherence in decoding.

[0120] The decoder uses a selected pre-trained language decoding model to perform deep learning decoding on neural signals to obtain initial decoding results, including word sequences or phoneme sequences.

[0121] The language decoding model employs spatial encoders and time-series processing techniques, combined with VQ-VAE and MAE pre-training strategies, to transform neural signals into embedded representations closely related to language states. These representations are then transformed into language decoding results through classification or generative networks.

[0122] The output component is responsible for presenting the decoded language information in a user-friendly way, such as through text display, speech synthesis, or direct brain-computer interface feedback.

[0123] Output components may include a text-to-speech (TTS) system, a display screen, or a specific brain-computer interface feedback mechanism. For example, a TTS system can quickly convert decoded text into speech to provide real-time feedback to the user or observer; a display screen directly shows the decoding results for visual confirmation.

[0124] The following is an example of how a brain-computer interface works:

[0125] Taking the identification of a user in an imagined reading state, with the decoding target being a word sequence, as an example:

[0126] Signal Acquisition: The neural signal acquisition device continuously captures the sEEG signals generated by the brain during silent reading.

[0127] State detection: The decoder uses a preset language state detection model to analyze signal characteristics and determine whether the user is in a silent reading state.

[0128] Model selection: The system automatically selects a pre-trained neural signal decoding model optimized for word sequence recognition based on the silent reading state.

[0129] Word sequence decoding: The neural signal decoding model decodes the neural signal based on the first weight and outputs preliminary word sequence recognition results.

[0130] Results Optimization and Output: The initial decoding results are further optimized using an N-Gram language model to ensure grammatical correctness and semantic coherence. The optimized language decoding results are then fed back to the user in real time via a display screen or a TTS system.

[0131] The brain-computer system in this embodiment of the invention is a closed-loop system from neural signal acquisition to language decoding result output. By adjusting the model and strategy in real time, it can efficiently and accurately identify the user's intention in different language states and present it in the form of natural language, which greatly enhances the application potential of brain-computer interface in the field of language decoding and user experience.

[0132] Embodiments of this application also provide an electronic device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention during runtime.

[0133] Optionally, in this embodiment, the processor in the above-described electronic device can be configured to run an executable program to perform the following steps:

[0134] Step S101: Obtain the neural signals of the subject;

[0135] Step S102: Determine the subject's language state based on neural signals and a pre-set language state detection model;

[0136] Step S103: Determine the pre-trained language decoding model based on the language state;

[0137] Step S104: Use the language decoding model to decode the neural signal to obtain the language decoding result.

[0138] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0139] Optionally, in this embodiment, the executable program can be configured to store an executable program for performing the following steps:

[0140] Step S101: Obtain the neural signals of the subject;

[0141] Step S102: Determine the subject's language state based on neural signals and a pre-set language state detection model;

[0142] Step S103: Determine the pre-trained language decoding model based on the language state;

[0143] Step S104: Use the language decoding model to decode the neural signal to obtain the language decoding result.

[0144] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0145] Optionally, in this embodiment, the computer program, when executed by the processor, performs the following steps:

[0146] Step S101: Obtain the neural signals of the subject;

[0147] Step S102: Determine the subject's language state based on neural signals and a pre-set language state detection model;

[0148] Step S103: Determine the pre-trained language decoding model based on the language state;

[0149] Step S104: Use the language decoding model to decode the neural signal to obtain the language decoding result.

[0150] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of the present invention.

[0151] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of the present invention.

[0152] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0157] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A neural signal decoding method, characterized in that, include: Acquire the neural signals of the subjects; The subject's language state is determined based on the neural signals and the preset language state detection model; Based on the language state, determine the pre-trained language decoding model; The neural signal is decoded using the language decoding model to obtain the language decoding result.

2. The neural signal decoding method according to claim 1, characterized in that, The language state includes spoken reading, and determining the pre-trained language decoding model based on the language state includes: In response to the language state being the spoken reading, a first language state weight corresponding to the spoken reading is determined; The first language state weights are loaded into the pre-built initial model to determine the language decoding model.

3. The neural signal decoding method according to claim 1, characterized in that, The language state includes silent reading, and determining the pre-trained language decoding model based on the language state includes: In response to the language state being silent reading, determine the weight of the second language state corresponding to the silent reading; The second language state weights are loaded into the pre-built initial model to determine the language decoding model.

4. The neural signal decoding method according to claim 1, characterized in that, The neural signal is decoded using the language decoding model to obtain the language decoding results, including: Obtain a decoding target, wherein the decoding target is used to guide the type of decoding result output by the language decoding model; Based on the decoding objective, the neural signal is decoded using the language decoding model to obtain the language decoding result.

5. The neural signal decoding method according to claim 4, characterized in that, The step of decoding the neural signal using the language decoding model according to the decoding target to obtain the language decoding result includes: In response to the decoding target being a word sequence, the neural signal is decoded using the language decoding model based on a first weight to obtain the language decoding result, wherein the first weight is a pre-loaded weight trained on a finite set of words and sentences.

6. The neural signal decoding method according to claim 4, characterized in that, The step of decoding the neural signal using the language decoding model according to the decoding target to obtain the language decoding result includes: In response to the decoding target being a phoneme sequence, the neural signal is decoded using the language decoding model based on a second weight to obtain the language decoding result, wherein the second weight is a pre-loaded weight obtained by training on the phoneme set.

7. The neural signal decoding method according to claim 1, characterized in that, The process of decoding the neural signal using the language decoding model to obtain the language decoding result includes: The neural signal is decoded using the language decoding model to obtain an initial decoding result, wherein the initial decoding result includes a word sequence or a phoneme sequence; The initial decoding result is input into a preset language model to obtain the language decoding result.

8. A neural signal decoding system, characterized in that, include: The acquisition module is used to acquire the neural signals of the subjects; The first determining module is used to determine the subject's language state based on the neural signals and a preset language state detection model; The second determining module is used to determine the pre-trained language decoding model based on the language state; The decoding module is used to decode the neural signal using the language decoding model to obtain the language decoding result.

9. A brain-computer interface system, characterized in that, include: A neural signal acquisition device is used to acquire neural signals from the subject. A decoder is used to determine the subject's language state based on the neural signals and a preset language state detection model; Based on the language state, a pre-trained language decoding model is determined; the language decoding model is used to decode the neural signals to obtain the language decoding result; An output component is used to output the language decoding result.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 7.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images