Neural signal decoding method, brain-computer system, electronic equipment and storage medium

Abstract

The invention discloses a neural signal decoding method, a brain-computer system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a training neural signal set; based on the training neural signal set, a pre-constructed initial neural signal decoding model is trained, a target signal decoding model is obtained, the target signal decoding model comprises an encoder, a vector quantizer and a regression device, the encoder is used for converting neural signals in the neural signal set into an embedded sequence, and the vector quantizer is used for converting the neural signals in the neural signal set into the regression device; the vector quantizer is used for mapping the embedded sequence to a neural code book to obtain vector embedding, and the regression device is used for reconstructing the vector embedding to obtain a reconstructed signal; obtaining a neural signal to be decoded; and decoding the neural signal to be decoded by using the target signal decoding model to obtain a neural signal decoding result. The technical problem that decoding of a neural signal decoding model in the prior art is not accurate enough is solved.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, and more specifically, to a neural signal decoding method, a brain-computer interface system, an electronic device, and a storage medium. Background Technology

[0002] Intracranial neural signal recording technology plays a vital role in neuroscience research and clinical applications due to its high signal-to-noise ratio and unique ability to provide information about specific brain regions. However, the complexity and high dimensionality of neural signals pose challenges to their decoding. Existing decoding schemes primarily focus on channel-level classification tasks such as epilepsy detection, while performing poorly on more challenging group-level classification tasks such as language decoding. This is because decoding language requires complex and precise processing of neural signals in language-related brain regions. Currently, there is no standard channel configuration for neural signal recording, making it even more difficult to model the spatial relationships between neural signals. In summary, existing technologies suffer from technical problems related to insufficient accuracy in neural signal decoding models.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a neural signal decoding method, a brain-computer interface, an electronic device, and a storage medium to at least address the technical problem of inaccurate decoding in existing neural signal decoding models.

[0005] According to one aspect of the present invention, a neural signal decoding method is provided, comprising: acquiring a training neural signal set; training a pre-constructed initial neural signal decoding model based on the training neural signal set to obtain a target signal decoding model, wherein the target signal decoding model includes an encoder, a vector quantizer, and a regressor, the encoder being used to convert neural signals in the neural signal set into embedded sequences, the vector quantizer being used to map the embedded sequences to neural encoders to obtain vector embeddings, and the regressor being used to reconstruct the vector embeddings to obtain a reconstructed signal; acquiring a neural signal to be decoded; and decoding the neural signal to be decoded using the target signal decoding model to obtain a neural signal decoding result.

[0006] Optionally, the training neural signal set includes neural signal sets corresponding to multiple language states. Based on the training neural signal set, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model, which includes: training the pre-constructed initial neural signal decoding model according to the neural signal sets corresponding to multiple language states to obtain the target signal decoding model.

[0007] Optionally, the training neural signal set includes a first signal set and a second signal set. The first signal set is used to characterize the neural signals corresponding to the subject's aloud reading state, and the second signal set is used to characterize the neural signals corresponding to the subject's imagined reading state. Based on the neural signal sets corresponding to multiple language states, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model, including: training the initial neural signal decoding model based on the first signal set to obtain the first language state weights; training the initial neural signal decoding model based on the second signal set to obtain the second language state weights; and determining the target signal decoding model based on the first language state weights and the second language state weights. The first language state weights and the second language state weights can be selectively loaded into the initial neural signal decoding model to obtain the target signal decoding model.

[0008] Optionally, the target signal decoding model includes multiple signal decoding targets. Based on the training neural signal set, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model.

[0009] Optionally, the initial neural signal decoding model includes a first sub-model and a second sub-model. The decoding target of the first sub-model is a word / sentence sequence, and the decoding target of the second sub-model is a phoneme sequence. Based on the training neural signal set and multiple signal decoding targets, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model, including: training the first sub-model based on the training neural signal set and a preset word / sentence set to obtain a first target model; training the second sub-model based on the training neural signal set and a preset phoneme set to obtain a second target model; and determining the target signal decoding model based on the first target model and the second target model.

[0010] Optionally, training a pre-constructed initial neural signal decoding model based on a training neural signal set to obtain a target signal decoding model includes: determining a first loss function based on the training neural signal set and the initial neural signal decoding model; determining a second loss function based on the training neural signal set, the first loss function, the mask autoencoder, and the initial neural signal decoding model; and training the initial encoder, the initial vector quantizer, and the initial regressor in the initial neural signal decoding model based on the training neural signal set and the second loss function to obtain the target signal decoding model.

[0011] Optionally, determining the first loss function based on the training neural signal set and the initial neural signal decoding model includes: using the initial encoder to transform the neural signals in the training neural signal set to obtain an embedding sequence; using a vector quantizer to map the embedding sequence to the neural codebook to obtain a vector embedding; using a regressor to reconstruct the vector embedding to obtain a reconstructed signal; and determining the first loss function based on the reconstructed signal and the training neural signal set.

[0012] Optionally, determining the second loss function based on the training neural signal set, the first loss function, the masked autoencoder, and the initial neural signal decoding model includes: using the masked autoencoder to mask the segment embeddings determined by the initial encoder based on the training neural signal set to obtain masked embeddings and unmasked embeddings; using the initial vector quantizer to determine the masked vector embedding of the masked embedding based on the unmasked embedding; determining the masked loss based on the masked vector embedding and the initial regressor; and determining the second loss function based on the masked loss and the first loss function.

[0013] Optionally, the target signal decoding model is used to decode the neural signal to be decoded to obtain the neural signal decoding result, which includes: determining the subject's language state based on the neural signal to be decoded; determining the language state weights corresponding to the language state based on the subject's language state, wherein the language state includes the spoken reading state and the imagined reading state, and the language state weights include the model weights trained in the spoken reading state and the model weights trained in the imagined reading state; after loading the language state weights into the initial neural signal decoding model to obtain the target signal decoding model, the neural signal to be decoded is decoded to obtain the neural signal decoding result.

[0014] Optionally, the target signal decoding model is used to decode the neural signal to be decoded to obtain the neural signal decoding result, which includes: obtaining decoding requirements, wherein the decoding requirements include word-based decoding and phoneme-based decoding; determining the decoding requirement weights corresponding to the decoding requirements, wherein the decoding requirement weights include model weights trained based on a preset set of words and sentences and model weights trained based on a preset set of phonemes; loading the decoding requirement weights into the initial neural signal decoding model to obtain the target signal decoding model, and then decoding the neural signal to be decoded to obtain the neural signal decoding result.

[0015] According to another aspect of the present invention, a neural signal decoding system is also provided, comprising: a first acquisition module for acquiring a training neural signal set; a training module for training a pre-constructed initial neural signal decoding model based on the training neural signal set to obtain a target signal decoding model, wherein the target signal decoding model includes an encoder, a vector quantizer, and a regressor, the encoder being used to convert neural signals in the neural signal set into embedded sequences, the vector quantizer being used to map the embedded sequences to neural encoders to obtain vector embeddings, and the regressor being used to reconstruct the vector embeddings to obtain a reconstructed signal; a second acquisition module for acquiring a neural signal to be decoded; and a decoding module for decoding the neural signal to be decoded using the target signal decoding model to obtain a neural signal decoding result.

[0016] According to another aspect of the present invention, a brain-computer interface system is also provided, comprising: a neural signal acquisition unit for acquiring neural signals to be decoded from a subject; a decoder for decoding the neural signals to be decoded using a target signal decoding model to obtain a neural signal decoding result, wherein the target signal decoding model is obtained by training a pre-constructed initial neural signal decoding model based on a training neural signal set, the target signal decoding model includes an encoder, a vector quantizer, and a regressor, the encoder is used to convert neural signals in the neural signal set into embedded sequences, the vector quantizer is used to map the embedded sequences to neural encodings to obtain vector embeddings, and the regressor is used to reconstruct the vector embeddings to obtain a reconstructed signal; and an output component for outputting the neural signal decoding result.

[0017] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of the present invention during runtime.

[0018] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0020] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0021] According to another aspect of the present invention, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of the present invention.

[0022] In this embodiment of the invention, a training neural signal set is obtained; based on the training neural signal set, a pre-constructed initial neural signal decoding model is trained to obtain a target signal decoding model. The target signal decoding model includes an encoder, a vector quantizer, and a regressor. The encoder converts the neural signals in the neural signal set into embedded sequences, the vector quantizer maps the embedded sequences to a neural codebase to obtain vector embeddings, and the regressor reconstructs the vector embeddings to obtain a reconstructed signal. A neural signal to be decoded is obtained; the target signal decoding model is used to decode the neural signal to be decoded to obtain the neural signal decoding result. The target signal decoding model used in this invention converts the original neural signal into an embedded sequence through an encoder, then maps it to a preset neural codebase via a vector quantizer to achieve a discretized representation of the signal, and finally uses a regressor to complete the fine reconstruction of the signal. When the target signal decoding model is applied to the neural signal to be decoded, the model can accurately identify and restore its inherent dynamic changes, thereby outputting a high-precision neural signal decoding result, improving the accuracy and efficiency of brain signal analysis, and thus solving the technical problem of insufficient decoding accuracy in existing neural signal decoding models. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0024] Figure 1 This is a flowchart illustrating a neural signal decoding method according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a neural signal decoding system provided according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to an embodiment of the present invention, a neural signal decoding method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 1 This is a method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0030] Step S101: Obtain the training neural signal set.

[0031] It should be noted that the embodiments of the present invention illustrate the neural signal decoding method based on neural signals recorded by sEEG electrodes. Practical applications of the present invention can also include decoding invasively recorded brain neural signals, including ECoG and spike signals.

[0032] Specifically, in this step, an intracranial neural signal dataset is collected to train the target signal decoding model. The training neural signal set should contain a large number of neural signal samples obtained from the brains of patients with implanted sEEG electrodes. This type of dataset needs to cover various language states (such as reading aloud, silent reading (imaginary reading), etc.) and different decoding targets (such as limited word and sentence classification or open set sentence generation).

[0033] Optionally, in some embodiments of the invention, the collected signals undergo appropriate preprocessing, such as filtering, resampling, and rereference, to ensure signal quality and reduce noise interference. The preprocessed signals are then used as training data for subsequent model training.

[0034] Step S102: Based on the training neural signal set, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model. The target signal decoding model includes an encoder, a vector quantizer, and a regressor. The encoder is used to convert the neural signals in the neural signal set into an embedding sequence. The vector quantizer is used to map the embedding sequence to the neural code to obtain a vector embedding. The regressor is used to reconstruct the vector embedding to obtain the reconstructed signal.

[0035] In this step, a pre-built initial neural signal decoding model is trained using a training neural signal set, thereby obtaining the target signal decoding model. The trained target signal decoding model includes at least three components: an encoder, a vector quantizer, and a regressor.

[0036] Encoder: Used to process raw neural signals and transform them into embedded sequences. The encoder fuses signal information from different channels through operations such as one-dimensional deep convolution and batch normalization to extract feature representations based on specific brain regions.

[0037] Vector quantizer: Maps the embedding sequence generated by the encoder to the neural code, resulting in discrete vector embeddings. This process involves transforming the continuous neural signal representation into a more compact, discrete form to improve the model's learning efficiency and representational power.

[0038] Regressor: Used to reconstruct the vector embeddings to recover the original neural signals. This helps the model learn how to reconstruct signals from neural code, enhancing its understanding and recovery capabilities of neural signals.

[0039] Optionally, the training process of the target signal decoding model includes two stages: VQ-VAE pre-training and MAE pre-training, as detailed below:

[0040] VQ-VAE (Vector Quantization Variational Autoencoder) pre-training: In this stage, the model learns how to convert neural signal segments into compact discrete representations using masking and vector quantization techniques, and reconstructs the original signal using a regressor. This helps the model learn the key features and structure of neural signals, as well as their connection to language processing.

[0041] MAE (Mask Autoencoder) pre-training: In this stage, a mask autoencoder is used for training. By masking part of the signal segment and predicting the masked part, the model can learn richer contextual information and dependencies in the time series. By introducing a symmetric masking strategy, the model's generalization ability and contextual understanding ability are further enhanced.

[0042] Optionally, in some embodiments of the present invention, training a pre-constructed initial neural signal decoding model based on a training neural signal set to obtain a target signal decoding model includes: firstly, pre-training the initial neural signal decoding model based on the training neural signal set to obtain a pre-trained model M1, and then using task signals to train M1 to obtain a fine-tuned model M2, which is the target neural signal decoding model.

[0043] Step S103: Obtain the neural signal to be decoded.

[0044] Specifically, the system receives intracranial neural signals from the subject during use, which can be real-time transmitted sEEG signals.

[0045] Optionally, after acquiring the neural signal to be decoded, the system will perform real-time preprocessing to ensure signal quality and reduce noise, and then send it to the real-time decoding system for processing.

[0046] Step S104: Use the target signal decoding model to decode the neural signal to be decoded to obtain the neural signal decoding result.

[0047] Specifically, the target signal decoding model is used to interpret the incoming neural signal to be decoded in real time, and the model will output the decoded result.

[0048] In this embodiment of the invention, a training neural signal set is obtained; based on the training neural signal set, a pre-constructed initial neural signal decoding model is trained to obtain a target signal decoding model. The target signal decoding model includes an encoder, a vector quantizer, and a regressor. The encoder converts the neural signals in the neural signal set into embedded sequences, the vector quantizer maps the embedded sequences to a neural codebase to obtain vector embeddings, and the regressor reconstructs the vector embeddings to obtain a reconstructed signal. A neural signal to be decoded is obtained; the target signal decoding model is used to decode the neural signal to be decoded to obtain the neural signal decoding result. The target signal decoding model used in this invention converts the original neural signal into an embedded sequence through an encoder, then maps it to a preset neural codebase via a vector quantizer to achieve a discretized representation of the signal, and finally uses a regressor to complete the fine reconstruction of the signal. When the target signal decoding model is applied to the neural signal to be decoded, the model can accurately identify and restore its inherent dynamic changes, thereby outputting a high-precision neural signal decoding result, improving the accuracy and efficiency of brain signal analysis, and thus solving the technical problem of insufficient decoding accuracy in existing neural signal decoding models.

[0049] Optionally, the training neural signal set includes neural signal sets corresponding to multiple language states. In step S102, training the pre-constructed initial neural signal decoding model based on the training neural signal set to obtain the target signal decoding model may include the following steps: training the pre-constructed initial neural signal decoding model according to the neural signal sets corresponding to multiple language states to obtain the target signal decoding model.

[0050] Specifically, firstly, a dataset of intracranial neural signals containing multiple language states is collected and organized. This dataset should cover neural signals from different individuals under different language tasks, such as reading words aloud, reading words silently, reading sentences aloud, and reading sentences silently. Then, based on the neural signal datasets corresponding to multiple language states, a pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model.

[0051] Optionally, the training neural signal set includes a first signal set and a second signal set. The first signal set is used to characterize the neural signals corresponding to the subject's aloud reading state, and the second signal set is used to characterize the neural signals corresponding to the subject's silent reading (imaginary reading) state. Based on the neural signal sets corresponding to multiple language states, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model, including: training the initial neural signal decoding model based on the first signal set to obtain the first language state weights; training the initial neural signal decoding model based on the second signal set to obtain the second language state weights; and determining the target signal decoding model based on the first language state weights and the second language state weights. The first language state weights and the second language state weights can be selectively loaded into the initial neural signal decoding model to obtain the target signal decoding model.

[0052] The first signal set: This dataset contains sEEG signals, ECoG signals, spike signals, etc., generated in the brain during the participants' reading aloud. These signals are electrophysiological activities recorded directly from specific areas of the brain, reflecting the neural dynamics of the brain during the specific language task of reading aloud.

[0053] The second signal set: This dataset contains sEEG signals, ECoG signals, spike signals, etc., generated in the brain during silent reading (imaginary reading). Silent reading (imaginary reading) refers to the subject reading silently in their mind without actually uttering a sound. This signal set reflects the neural activity of the brain processing language information in the absence of external sound stimulation.

[0054] Training based on the first signal set: A first signal set containing neural signals during spoken reading is used for pre-training with an initial signal decoding model. The purpose of this step is to teach the model to recognize and decode neural signals during spoken reading. During training, the model's parameters are adjusted based on the signals and corresponding outputs (such as spoken text), ultimately resulting in a set of "first language state weights." These weights represent the optimal state of the model when processing neural signals during spoken reading.

[0055] Training based on a second signal set: Similarly, the initial signal decoding model is pre-trained using a second signal set containing neural signals during silent reading (imaginary reading). This process enables the model to decode neural signals during silent reading (imaginary reading). By adjusting the model parameters, another set of "second language state weights" is trained, which represents the optimal state of the model in processing neural signals during silent reading (imaginary reading).

[0056] Integration of first language state weights and second language state weights: After initial training, depending on the application scenario, the "first language state weights" or "second language state weights" can be flexibly selected to be loaded into the initial signal decoding model to obtain the target signal decoding model.

[0057] Determining the target signal decoding model: The final target signal decoding model will selectively employ first-language state weights or second-language state weights, or simultaneously support switching between the two states in the system, to achieve real-time decoding of neural signals under different language states (aloud reading and silent reading). When the real-time system detects that the subject is in an aloud reading state, it will use the first-language state weights; when it detects a silent reading (imaginary reading) state, it will use the second-language state weights; or it will switch between the two through some mechanism (such as user selection or automatic system recognition) to achieve the best decoding effect.

[0058] Optionally, the target signal decoding model includes multiple signal decoding targets. In step S102, training the pre-constructed initial neural signal decoding model based on the training neural signal set to obtain the target signal decoding model may include the following steps: training the pre-constructed initial neural signal decoding model based on the training neural signal set and multiple signal decoding targets to obtain the target signal decoding model.

[0059] Specifically, by using a training neural signal set, the pre-constructed initial neural signal decoding model is trained under different decoding targets to obtain a target signal decoding model. The trained target signal decoding model can then decode for different decoding targets.

[0060] Optionally, the initial neural signal decoding model includes a first sub-model and a second sub-model. The decoding target corresponding to the first sub-model is a word / sentence sequence, and the decoding target corresponding to the second sub-model is a phoneme sequence. In step S102, training the pre-constructed initial neural signal decoding model based on the training neural signal set and multiple signal decoding targets to obtain the target signal decoding model may include the following steps: training the first sub-model based on the training neural signal set and a preset word / sentence set to obtain a first target model; training the second sub-model based on the training neural signal set and a preset phoneme set to obtain a second target model; and determining the target signal decoding model based on the first target model and the second target model.

[0061] The first sub-model is designed for word and sentence sequence decoding, with the goal of recognizing and decoding specific words and complete sentences derived from brain signals. For example, in a Chinese context, it can be trained to recognize 60 common words or 20 common sentences.

[0062] The second sub-model focuses on the continuous classification and generation of phoneme units, aiming to decode the basic phoneme units that constitute speech from continuous neural signals, and then combine them into words or sentences. A phoneme is the smallest unit of pronunciation in a language; in Chinese, phoneme units can include various initials, finals, and tones.

[0063] The first sub-model is trained using a training neural signal set—a large-scale set of preprocessed neural signal data—along with a pre-defined set of words and sentences (containing examples of target words and sentences). This training process enables the model to learn to associate specific neural signal patterns with specific words and sentences, thereby enabling it to recognize and decode word and sentence signals emitted by the brain in real-time applications.

[0064] While still utilizing a training set of neural signals, this time it incorporates a pre-defined set of phonemes, including a comprehensive set of phoneme units. The second sub-model learns to convert neural signals into phoneme sequences, and then combines these phoneme sequences into words or sentences. This approach is more flexible and suitable for sentence generation scenarios with open sets.

[0065] Depending on the user's specific needs or application scenarios (such as whether it is necessary to recognize words and sentences in a fixed vocabulary list or to generate open sentences), the first target model (the first sub-model after training) and the second target model (the second sub-model after training) can be selected or combined.

[0066] Ultimately, by examining the performance and complementarity of the first and second target models, a target signal decoding model is determined to meet the high-efficiency and accurate requirements of the real-time decoding system under different language states and decoding targets.

[0067] Understandably, the dual-track parallel model training and determination strategy utilizes different levels of information from words and phonemes, ensuring the system's accuracy when processing fixed words and phrases while providing flexibility and creativity in generating new sentences, thereby optimizing the real-time decoding capability and application potential of the language brain-computer interface.

[0068] Optionally, in step S102, training the pre-constructed initial neural signal decoding model based on the training neural signal set to obtain the target signal decoding model may include the following steps: determining a first loss function based on the training neural signal set and the initial neural signal decoding model; determining a second loss function based on the training neural signal set, the first loss function, the mask autoencoder, and the initial neural signal decoding model; and training the initial encoder, the initial vector quantizer, and the initial regressor in the initial neural signal decoding model based on the training neural signal set and the second loss function to obtain the target signal decoding model.

[0069] The initial neural signal decoding model first receives signals from the training neural signal set as input. These signals are encoded by the model, producing a series of neural encoded representations. Simultaneously, the model converts these encoded representations into discrete neural codes using its internal vector quantizer. This step aims to capture key features of the neural signals and map them onto a finite set of representations. The determination of the first loss function depends on the difference between the model's output and the actual neural signals; a reconstruction loss (such as mean squared error, cross-entropy loss, etc.) can be used to measure the model's ability to reconstruct the original signal. The loss function guides the model on how to improve its encoding and quantization processes to more accurately represent intracranial neural signals.

[0070] After determining the first loss function, a masked autoencoder (MAE) is introduced to enhance the model's understanding and prediction capabilities. The MAE partially masks the input neural signal, randomly obscuring segments, and then allows the model to attempt to predict the content of these masked segments. The second loss function is calculated based on the first loss function and the model's prediction accuracy for the masked signal segments. It is achieved by decoding the model's output and comparing it with the actual unmasked signal. During MAE training, the model not only learns the features of the preserved signal segments but also learns to infer the masked portions based on these segments, thereby improving its sensitivity to signal patterns and context.

[0071] The initial neural signal decoding model (including the initial encoder, initial vector quantizer, and initial regressor) begins training under the joint guidance of the first and second loss functions. Backpropagation of the loss functions updates the model's weights and parameters to reduce loss and improve performance in signal decoding and signal segment prediction. During training, the model gradually learns how to extract meaningful features from sEEG signals and transform them into a form usable for prediction and decoding. After sufficient training, the encoder, vector quantizer, and regressor parts of the model are optimized to process and understand intracranial neural signals more efficiently.

[0072] Ultimately, the target signal decoding model is an optimized neural network model with stronger decoding and prediction capabilities. It can achieve real-time intracranial neural signal decoding and is suitable for the analysis of various language tasks and states.

[0073] Optionally, determining the first loss function based on the training neural signal set and the initial neural signal decoding model includes: using the initial encoder to transform the neural signals in the training neural signal set to obtain an embedding sequence; using a vector quantizer to map the embedding sequence to the neural codebook to obtain a vector embedding; using a regressor to reconstruct the vector embedding to obtain a reconstructed signal; and determining the first loss function based on the reconstructed signal and the training neural signal set.

[0074] First, the neural signals in the training neural signal set are processed using the initial encoder in the initial neural signal decoding model. The initial encoder is responsible for converting the raw sEEG signals into a more abstract, higher-level representation, namely, embedding sequences. These embedding sequences contain the key features of the signal for subsequent processing and analysis.

[0075] Next, a vector quantizer is used to map the resulting embedding sequence to a discrete coding space, known as the neural codebook. The neural codebook is a fixed set of discrete vectors, each representing a specific neural state or pattern. Vector quantization is performed by finding the vector in the neural codebook that is closest to each embedding; this process is typically based on Euclidean distance or other similarity metrics. The resulting vector embeddings are discrete, more compact representations of the neural signals.

[0076] The resulting vector embeddings are then fed into the regressor in the initial neural signal decoding model. The regressor's role is to attempt to reconstruct the original sEEG signal from these discrete vector embeddings. The regressor may contain one or more transposed convolutional layers and a Transformer decoder, which work together to inversely parse the sEEG signal.

[0077] To evaluate and optimize the model's performance, a loss function needs to be defined. The first loss function is calculated based on the difference between the reconstructed signal and the original training neural signal set. Specifically, this loss function measures the mean squared error between the reconstructed signal and the original sEEG signal.

[0078] For example, the encoder can process any input intracranial neural signal of arbitrary length. Its core step is to segment the neural signal into patches. For each sample X, it is segmented into patches using a non-overlapping window of length W. These patches are then sequentially passed through a spatial encoder, combined with temporal embedding and a Transformer encoder.

[0079] First, the spatial encoder fuses signals from different channels and extracts information. It consists of a linear projection layer and several convolutional blocks. The linear projection transforms the original neural signals into a hidden neural space and uses its weights for subsequent analysis. The convolutional blocks consist of 1D depthwise convolution layers and batch normalization layers. The spatial encoder transforms each neural signal segment into a patch embedding. Second, to enable the model to perceive the temporal context of the patch embeddings, parameter-free positional embeddings are used in conjunction with the patch embeddings. Finally, the embedded sequence is directly fed into the Transformer encoder to obtain the final embedded sequence.

[0080] In the encoder phase, a neural codebook is defined. C = {c j |j = 1 , ..., Ncodex} ∈ Ncodex×dcodex ,in N codex It is the number of discrete neural embeddings. d codex This is the number of dimensions for each embedding. For the embeddings output by the encoder... E = { e i ∈ dFor each neural code segment |i = 1, ..., N}, the embeddings are projected into the space of the neural code using a linear projection. The code then finds the nearest neighbor element to each projected embedding, which is the vector-quantized embedding Z. Next, the regressor consists of a Transformer decoder and a set of transposed convolutional layers. For the vector-quantized embedding Z, the regressor transforms the discrete embeddings back into the original neural signal and uses the mean squared error as the loss function for regression (the first loss function).

[0081] Optionally, determining the second loss function based on the training neural signal set, the first loss function, the masked autoencoder, and the initial neural signal decoding model includes: using the masked autoencoder to mask the segment embeddings determined by the initial encoder based on the training neural signal set to obtain masked embeddings and unmasked embeddings; using the initial vector quantizer to determine the masked vector embedding of the masked embedding based on the unmasked embedding; determining the masked loss based on the masked vector embedding and the initial regressor; and determining the second loss function based on the masked loss and the first loss function.

[0082] For example, to enhance the neural signal decoding model's ability to learn contextual representations, masked neural signal modeling is used. For the segment embeddings obtained from the spatial encoder, approximately 50% of the embeddings are selected block by block for masking, and the mask position matrix is ​​denoted as M. The masked embeddings are replaced with a shared, learnable embedding. Subsequent processing is the same as that of the encoder, and its output feature embeddings are passed through a linear classifier to predict the encoding in the corresponding encoding copy for each masked segment. Further, a symmetric masking strategy is used: the inverse of the mask matrix M is calculated, and masking is performed sequentially for the new mask matrix to obtain the symmetric masking loss. This loss is added to the previous first loss function as the loss function for pre-training the neural signal decoding model.

[0083] Optionally, in step S104, decoding the neural signal to be decoded using the target signal decoding model to obtain the neural signal decoding result may include the following steps: determining the subject's language state based on the neural signal to be decoded; determining the language state weights corresponding to the language state based on the subject's language state, wherein the language state includes aloud reading state and silent reading (imaginary reading) state, and the language state weights include the model weights trained in the aloud reading state and the model weights trained in the silent reading (imaginary reading) state; loading the language state weights into the initial neural signal decoding model to obtain the target signal decoding model, and then decoding the neural signal to be decoded to obtain the neural signal decoding result.

[0084] Specifically, determining the subject's language state involves several steps. First, the system needs to monitor and determine the subject's current language state in real time, i.e., whether the subject is reading aloud or silently (imagine reading). This process is performed by a real-time language state detection module, which uses a convolutional fully connected network to classify preprocessed neural signals and distinguish different language states. The real-time system can respond instantly, ensuring that the decoding process remains synchronized with the subject's current activity.

[0085] Determining Language State Weights: Based on a given language state, the system selects the model weights corresponding to that state. Since the target neural signal decoding model has already trained different model weights for both aloud reading and silent reading (imaginary reading) during the pre-training phase, if the subject is in the aloud reading state, the system will load the model weights trained in the aloud reading state; if the subject is in the imaginary reading state, it will load the model weights trained in the silent reading (imaginary reading) state. These weights reflect the brain activity patterns under different language states and are crucial for the model's accurate decoding.

[0086] Model weight loading: Based on the language state weights determined in the above steps, the real-time system loads the corresponding model weights into the initial neural signal decoding model to obtain the target neural signal decoding model. This means that the model will use parameters specific to the current language state to improve decoding performance and accuracy.

[0087] Neural signal decoding: Once the model weights are correctly loaded, the target neural signal decoding model begins decoding the neural signal to be decoded. The model converts the input signal into encoding units based on specific brain regions, and then performs deep learning through vector quantization and masked autoencoder strategies, ultimately outputting the neural signal decoding result. This result can be identified words, sentences, or continuous phoneme sequences, depending on the chosen decoding target and task.

[0088] Obtaining neural signal decoding results: After the target neural signal decoding model completes decoding, the system will output the neural signal decoding results. If the decoding target is the classification of finite words and sentences, the system will directly output the classification results; if the target is the generation of sentences from an open set, the system will output a long phoneme unit sequence, which will be generated and corrected by a language model to finally generate readable sentences.

[0089] Optionally, in step S104, decoding the neural signal to be decoded using the target signal decoding model to obtain the neural signal decoding result may include the following steps: obtaining decoding requirements, wherein the decoding requirements include word-based decoding and phoneme-based decoding; determining the decoding requirement weights corresponding to the decoding requirements, wherein the decoding requirement weights include model weights obtained by training based on a preset set of words and phrases and model weights obtained by training based on a preset set of phonemes; loading the decoding requirement weights into the initial neural signal decoding model to obtain the target signal decoding model, and then decoding the neural signal to be decoded to obtain the neural signal decoding result.

[0090] First, the system needs to identify the user's decoding needs, which can include word-based decoding (e.g., identifying specific words the user is silently reading or reciting) or phoneme-based decoding (decoding consecutive phoneme units that the user pronounces or imagines, thereby generating sentences). This need can be identified through a user interface or preset decoding modes.

[0091] Based on the user's selected decoding requirements, the system determines the model weights that best match the decoding needs from a pre-trained model library. If the user requires word-based decoding, the system will load model weights trained on a preset set of words and phrases; if the user requires phoneme-based decoding, the system will load model weights trained on a preset set of phonemes. These weights have been pre-trained and are capable of capturing the brain's neurodynamics associated with a specific decoding target.

[0092] After determining the model weights corresponding to the decoding requirements, the system loads these weights into the initial neural signal decoding model to obtain the target neural signal decoding model, ensuring that the model can accurately decode for specific decoding requirements.

[0093] Once the model is ready, the real-time transmitted neural signals are fed into the target neural signal decoding model for decoding. If the model is pre-trained for word decoding, it will directly output word sequences; if the model is pre-trained for phoneme decoding, it will output phoneme unit sequences, which will then be fed into a phoneme-to-text conversion module, such as Connectionist Temporal Classification (CTC) or Recurrent Neural Network (RNN), to generate sentences.

[0094] Language model correction: The decoding results of both words and phonemes are corrected by an N-Gram language model to improve the accuracy and fluency of decoding.

[0095] Through the above steps, the system can dynamically adjust to meet different decoding needs, whether based on limited word classification or open set generation based on phonemes, thereby achieving efficient real-time brain-computer interface decoding. This flexibility and adaptability are lacking in existing intracranial neural signal decoding systems. The combination of the target neural signal decoding model and the real-time decoding system can better meet the needs of different scenarios and applications.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0097] This invention also provides a brain-computer interface system, comprising: a neural signal acquisition unit for acquiring neural signals to be decoded from a subject; a decoder for decoding the neural signals to be decoded using a target signal decoding model to obtain neural signal decoding results, wherein the target signal decoding model is obtained by training a pre-constructed initial neural signal decoding model based on a training neural signal set, the target signal decoding model including an encoder, a vector quantizer, and a regressor, the encoder being used to convert neural signals in the neural signal set into embedded sequences, the vector quantizer being used to map the embedded sequences to neural encoders to obtain vector embeddings, and the regressor being used to reconstruct the vector embeddings to obtain reconstructed signals; and an output component for outputting neural signal decoding results.

[0098] This invention also provides a neural model processing system for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] Figure 2 This is a structural block diagram of a neural signal processing system according to one embodiment of the present invention, such as... Figure 2As shown, the system includes: a first acquisition module 201, used to acquire a training neural signal set; a training module 202, used to train a pre-constructed initial neural signal decoding model based on the training neural signal set to obtain a target signal decoding model, wherein the target signal decoding model includes an encoder, a vector quantizer, and a regressor, the encoder is used to convert the neural signals in the neural signal set into an embedding sequence, the vector quantizer is used to map the embedding sequence to the neural encoder to obtain a vector embedding, and the regressor is used to reconstruct the vector embedding to obtain a reconstructed signal; a second acquisition module 203, used to acquire the neural signal to be decoded; and a decoding module 204, used to decode the neural signal to be decoded using the target signal decoding model to obtain the neural signal decoding result.

[0100] Embodiments of this application also provide an electronic device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention during runtime.

[0101] Optionally, in this embodiment, the processor in the above-described electronic device can be configured to run an executable program to perform the following steps:

[0102] Step S101: Obtain the training neural signal set.

[0103] Step S102: Based on the training neural signal set, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model. The target signal decoding model includes an encoder, a vector quantizer, and a regressor. The encoder is used to convert the neural signals in the neural signal set into an embedding sequence. The vector quantizer is used to map the embedding sequence to the neural code to obtain a vector embedding. The regressor is used to reconstruct the vector embedding to obtain the reconstructed signal.

[0104] Step S103: Obtain the neural signal to be decoded.

[0105] Step S104: Use the target signal decoding model to decode the neural signal to be decoded to obtain the neural signal decoding result.

[0106] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0107] Optionally, in this embodiment, the executable program can be configured to store an executable program for performing the following steps:

[0108] Step S101: Obtain the training neural signal set.

[0109] Step S102: Based on the training neural signal set, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model. The target signal decoding model includes an encoder, a vector quantizer, and a regressor. The encoder is used to convert the neural signals in the neural signal set into an embedding sequence. The vector quantizer is used to map the embedding sequence to the neural code to obtain a vector embedding. The regressor is used to reconstruct the vector embedding to obtain the reconstructed signal.

[0110] Step S103: Obtain the neural signal to be decoded.

[0111] Step S104: Use the target signal decoding model to decode the neural signal to be decoded to obtain the neural signal decoding result.

[0112] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0113] Optionally, in this embodiment, the computer program, when executed by the processor, performs the following steps:

[0114] Step S101: Obtain the training neural signal set.

[0115] Step S102: Based on the training neural signal set, the pre-constructed initial neural signal decoding model is trained to obtain the target signal decoding model. The target signal decoding model includes an encoder, a vector quantizer, and a regressor. The encoder is used to convert the neural signals in the neural signal set into an embedding sequence. The vector quantizer is used to map the embedding sequence to the neural code to obtain a vector embedding. The regressor is used to reconstruct the vector embedding to obtain the reconstructed signal.

[0116] Step S103: Obtain the neural signal to be decoded.

[0117] Step S104: Use the target signal decoding model to decode the neural signal to be decoded to obtain the neural signal decoding result.

[0118] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of the present invention.

[0119] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of the present invention.

[0120] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0125] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of neural signal decoding, the method comprising: The method comprises the following steps: obtaining a training neural signal set; training a pre-constructed initial neural signal decoding model based on the training neural signal set to obtain a target signal decoding model, wherein the target signal decoding model comprises an encoder, a vector quantizer and a regressor, the encoder is used to convert a neural signal in the neural signal set into an embedding sequence, the vector quantizer is used to map the embedding sequence to a neural codebook to obtain a vector embedding, and the regressor is used to reconstruct the vector embedding to obtain a reconstructed signal; obtaining a to-be-decoded neural signal; decoding the to-be-decoded neural signal by using the target signal decoding model to obtain a neural signal decoding result.

2. The neural signal decoding method according to claim 1, characterized by, The training neural signal set comprises neural signal sets corresponding to multiple language states, and the training of the pre-constructed initial neural signal decoding model based on the training neural signal set to obtain the target signal decoding model comprises: training the pre-constructed initial neural signal decoding model based on the neural signal sets corresponding to the multiple language states to obtain the target signal decoding model.

3. The neural signal decoding method according to claim 2, characterized by, The training neural signal set comprises a first signal set and a second signal set, the first signal set is used to represent the neural signal corresponding to the subject in the out-loud reading state, and the second signal set is used to represent the neural signal corresponding to the subject in the silent reading state, and the training of the pre-constructed initial neural signal decoding model based on the neural signal sets corresponding to the multiple language states to obtain the target signal decoding model comprises: training the initial neural signal decoding model based on the first signal set to obtain a first language state weight; training the initial neural signal decoding model based on the second signal set to obtain a second language state weight; determining the target signal decoding model according to the first language state weight and the second language state weight, wherein the first language state weight and the second language state weight can be loaded into the initial neural signal decoding model to obtain the target signal decoding model.

4. The neural signal decoding method according to claim 1, characterized by, The target signal decoding model comprises multiple signal decoding targets, and the training of the pre-constructed initial neural signal decoding model based on the training neural signal set to obtain the target signal decoding model comprises: training the pre-constructed initial neural signal decoding model based on the training neural signal set and the multiple signal decoding targets to obtain the target signal decoding model.

5. The neural signal decoding method according to claim 4, characterized by, The initial neural signal decoding model comprises a first sub-model and a second sub-model, the decoding target corresponding to the first sub-model is a word sequence, and the decoding target corresponding to the second sub-model is a phoneme sequence, and the training of the pre-constructed initial neural signal decoding model based on the training neural signal set and the multiple signal decoding targets to obtain the target signal decoding model comprises: training the first sub-model based on the training neural signal set and a preset word set to obtain a first target model; training the second sub-model based on the training neural signal set and a preset phoneme set to obtain a second target model; According to the first target model and the second target model, a target signal decoding model is determined.

6. The neural signal decoding method according to claim 1, characterized by, The training of the pre-constructed initial neural signal decoding model based on the training neural signal set to obtain the target signal decoding model includes: According to the training neural signal set and the initial neural signal decoding model, a first loss function is determined. According to the training neural signal set, the first loss function, a mask autoencoder, and the initial neural signal decoding model, a second loss function is determined. According to the training neural signal set and the second loss function, the initial encoder, the initial vector quantizer, and the initial regressor in the initial neural signal decoding model are trained to obtain the target signal decoding model.

7. The neural signal decoding method according to claim 6, characterized by, The determination of the first loss function according to the training neural signal set and the initial neural signal decoding model includes: The neural signals in the training neural signal set are transformed by the initial encoder to obtain embedding sequences; The embedding sequences are mapped to the neural codebook by the vector quantizer to obtain vector embeddings; The vector embeddings are reconstructed by the regressor to obtain reconstructed signals; The first loss function is determined according to the reconstructed signals and the training neural signal set.

8. The neural signal decoding method according to claim 6, characterized by, The determination of the second loss function according to the training neural signal set, the first loss function, a mask autoencoder, and the initial neural signal decoding model includes: The mask autoencoder is used to mask the segment embeddings determined by the initial encoder based on the training neural signal set to obtain mask embeddings and non-mask embeddings; The initial vector quantizer is used to determine the mask vector embeddings of the mask embeddings based on the non-mask embeddings; According to the mask vector embeddings and the initial regressor, a mask loss is determined; According to the mask loss and the first loss function, the second loss function is determined.

9. The neural signal decoding method of claim 1, wherein, The decoding of the to-be-decoded neural signal by the target signal decoding model to obtain a neural signal decoding result includes: According to the to-be-decoded neural signal, a language state of a subject is determined; According to the language state of the subject, a language state weight corresponding to the language state is determined, wherein the language state includes an out-loud reading state and a silent reading state, and the language state weight includes a model weight trained in the out-loud reading state and a model weight trained in the silent reading state; After loading the language state weight into the initial neural signal decoding model to obtain the target signal decoding model, the to-be-decoded neural signal is decoded to obtain a neural signal decoding result.

10. The neural signal decoding method according to claim 1, characterized by, The decoding of the to-be-decoded neural signal by the target signal decoding model to obtain a neural signal decoding result includes: A decoding requirement is obtained, wherein the decoding requirement includes word-based decoding and phoneme-based decoding; According to the decoding requirement, a decoding requirement weight corresponding to the decoding requirement is determined, wherein the decoding requirement weight includes a model weight trained based on a preset word set and a model weight trained based on a preset phoneme set; After the decoding demand weight is loaded into the initial neural signal decoding model to obtain the target signal decoding model, the neural signal to be decoded is decoded to obtain a neural signal decoding result.

11. A neural signal decoding system, comprising: The method comprises the steps of: The first obtaining module is configured to obtain a training neural signal set. The training module is configured to train a pre-constructed initial neural signal decoding model based on the training neural signal set to obtain a target signal decoding model, wherein the target signal decoding model comprises an encoder, a vector quantizer and a regressor, the encoder is configured to convert a neural signal in the training neural signal set into an embedding sequence, the vector quantizer is configured to map the embedding sequence to a neural codebook to obtain a vector embedding, and the regressor is configured to reconstruct the vector embedding to obtain a reconstructed signal. The second obtaining module is configured to obtain a neural signal to be decoded. The decoding module is configured to decode the neural signal to be decoded by using the target signal decoding model to obtain a neural signal decoding result.

12. A brain-computer system, characterized in that The method comprises the steps of: The neural signal collector is configured to obtain a neural signal to be decoded of a subject. The decoder is configured to decode the neural signal to be decoded by using a target signal decoding model to obtain a neural signal decoding result, wherein the target signal decoding model is obtained by training a pre-constructed initial neural signal decoding model based on a training neural signal set, the target signal decoding model comprises an encoder, a vector quantizer and a regressor, the encoder is configured to convert a neural signal in the training neural signal set into an embedding sequence, the vector quantizer is configured to map the embedding sequence to a neural codebook to obtain a vector embedding, and the regressor is configured to reconstruct the vector embedding to obtain a reconstructed signal. The output component is configured to output the neural signal decoding result.

13. An electronic device, comprising: The method comprises the steps of: The memory stores an executable program. The processor is configured to run the program, wherein the program performs the method of any one of claims 1 to 10 when running.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored executable program, wherein the executable program controls the device where the storage medium is located to perform the method of any one of claims 1 to 10 when running.

15. A computer program product, characterised in that, The computer program comprises a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

Images