Bayesian brain-inspired brain-computer interface decoding method, terminal device and computer readable storage medium

Abstract

The application is suitable for the field of brain-computer interface and neural signal decoding technology, and provides a Bayesian brain-inspired brain-computer interface decoding method, a terminal device and a computer readable storage medium, comprising: acquiring a brain-computer interface signal of a target subject; encoding according to the brain-computer interface signal to obtain a low-layer neural observation representation; performing constraint processing on the brain-computer interface signal according to preset brain structure prior information to obtain a middle-layer structure prior representation; performing Bayesian joint inference according to the low-layer neural observation representation and the middle-layer structure prior representation to obtain a high-layer meta-neural semantic representation corresponding to the brain-computer interface signal; and performing semantic decoding according to the high-layer meta-neural semantic representation to obtain a decoding result. Through the above method, the decoding accuracy of the brain-computer interface signal can be effectively improved, thereby improving the decoding effect.

Description

Technical Field

[0001] This application belongs to the field of brain-computer interface and neural signal decoding technology, and particularly relates to a Bayesian brain-inspired brain-computer interface decoding method, terminal device and computer-readable storage medium. Background Technology

[0002] Brain-computer interfaces (BCIs) establish direct pathways between brain neural activity and external devices, holding significant value in areas such as neural function reconstruction, assisted communication, and human-machine collaboration. Decoding is the core component of a BCI system, its task being to convert neural observation signals from electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), and electrocorticography (ECoG) into information usable by the external system. In other words, the goal of BCI decoding is to convert the neural activity of a subject in tasks such as motor imagery, emotion recognition, and open semantic expression into results usable by the external system, such as control commands, natural language text, and visual content. Existing decoding methods typically target external stimuli, making it difficult to decode the subject's own internal semantic judgments and subjective evaluations, resulting in poor decoding performance. Summary of the Invention

[0003] This application provides a Bayesian brain-computer interface decoding method, terminal device, and computer-readable storage medium that can effectively improve the decoding accuracy of brain-computer interface signals, thereby improving the decoding effect.

[0004] In a first aspect, embodiments of this application provide a Bayesian brain-computer interface decoding method, including: Acquire brain-computer interface signals from the target subject; The brain-computer interface signals are encoded to obtain low-level neural observation representations. The brain-computer interface signal is constrained based on the preset prior information of brain structure to obtain the prior representation of the middle layer structure. The high-level neuro-semantic representation corresponding to the brain-computer interface signal is inferred by jointly inferring the low-level neural observation representation and the mid-level structural prior representation; wherein, the high-level neuro-semantic representation is used to represent the cognitive state inside the brain; Semantic decoding is performed based on the high-level neuromorphic semantic representation to obtain the decoding result.

[0005] In this embodiment, the low-level neural observation representation is equivalent to the observation features obtained by encoding the brain-computer interface signal; the mid-level structural prior representation is the feature obtained by constraining the brain-computer interface signal based on preset brain structural prior information, which is equivalent to constraining the brain-computer interface signal by brain structural prior; then, the high-level neural semantic representation corresponding to the brain-computer interface signal is inferred by jointly inferring the low-level neural observation representation and the mid-level structural prior representation, which is equivalent to jointly inferring the neural semantics by fusing the observation features obtained by encoding the brain-computer interface signal with the brain structural prior constraint features. By relying on physiological priors to correct signal noise and spatial resolution defects, more accurate and physiologically interpretable brain-computer semantic decoding is achieved, which is conducive to improving the decoding effect of brain-computer interface signals.

[0006] In one possible implementation of the first aspect, the encoding based on the brain-computer interface signal to obtain a low-level neural observation representation includes: Obtain a pre-trained encoder; wherein the pre-trained encoder is trained based on a loss function, the loss function being used to evaluate the encoding result of the training samples after masking, and the masking being used to mask part of the signal in the training samples; The brain-computer interface signal is encoded using the pre-trained encoder to obtain encoded information.

[0007] In the above implementation, since the masking process can mask part of the signal in the training sample, it is equivalent to increasing the training difficulty and increasing the encoder's ability to understand the signal. Therefore, the encoder trained according to the loss function used to evaluate the encoding result of the training sample after masking can understand the brain-computer interface signal more accurately, thereby encoding more accurate encoded information.

[0008] In one possible implementation of the first aspect, the prior information on brain structure includes prior information on the geometry of the cerebral cortex; The step of constraining the brain-computer interface signal based on preset prior information about brain structure to obtain a priori representation of the mid-level structure includes: Obtain prior information about the geometry of the cerebral cortex; The brain-computer interface signal is constrained based on the prior information constructed from the geometry of the cerebral cortex to obtain the prior representation of the middle layer structure.

[0009] In the above implementation method, the brain-computer interface signal is constrained by the prior information of brain structure constructed by the geometry of the cerebral cortex, and a prior representation of the middle structure is generated. This can use the prior information of human brain anatomical geometry to constrain the EEG signal, reduce irrelevant noise interference, and improve the physiological authenticity and feature discrimination of the brain-computer signal representation, thereby helping to improve the decoding accuracy.

[0010] In one possible implementation of the first aspect, the prior information based on the construction of cerebral cortex geometry includes a general cortical geometry spectrum; the general cortical geometry spectrum includes multiple cortical spectra of the cerebral cortex and spectral values ​​corresponding to each cortical spectra; The step of constraining the brain-computer interface signal based on the prior information constructed from the cerebral cortex geometry to obtain the prior representation of the mid-level structure includes: Obtain the individual cortical geometry spectrum of the target subject; The target cortical geometry spectrum is obtained by adjusting the general cortical geometry spectrum based on the individual cortical geometry spectrum. The geometric spectral coefficients corresponding to the brain-computer interface signal are calculated based on the geometric spectrum of the target cortex.

[0011] In the above implementation method, personalized adjustments are made for the target subject. The general cortical geometry spectrum is corrected by combining the individual cortical geometry spectrum to obtain the target cortical geometry spectrum that is suitable for the target subject. Then, the geometry spectrum coefficients are solved as prior information of brain structure. Taking into account the differences between the general anatomical base and individual brain morphology, it can effectively improve the personalized adaptation of brain-computer interface signal constraints, thereby helping to improve the accuracy of semantic inference.

[0012] In one possible implementation of the first aspect, calculating the geometric spectral coefficients corresponding to the brain-computer interface signal based on the target cortical geometric spectrum includes: For the first type of signal in the brain-computer interface signal, the first type of signal is projected onto the geometric spectrum of the target cortex to obtain the geometric spectrum coefficients corresponding to the first type of signal; For the second type of signal in the brain-computer interface signal, the geometric spectrum of the target cortex is projected onto the sensor space corresponding to the second type of signal to obtain the sensor space geometric pattern matrix; The sensor spatial geometric pattern matrix is ​​decomposed or orthogonalized to filter out effective sensor spatial geometric patterns. The geometric spectral coefficients corresponding to the second type of signal are calculated based on the effective sensor spatial geometric pattern and the spectral values ​​corresponding to each cortical spectrum in the target cortical geometric spectrum; wherein the spatial resolution of the second type of signal is lower than that of the first type of signal.

[0013] In the above implementation, a differentiated geometric spectral coefficient solution strategy is adopted for two types of brain-computer interface signals with different spatial resolutions. This strategy takes into account the efficient solution of high-precision signals by direct projection and the adaptation of low-resolution signals to cortical geometric spectrum calculation through spatial projection and matrix decomposition. The spectral coefficient conversion of the two types of signals is completed by relying on the target cortical geometric spectrum, which adapts to the cortical map analysis requirements of EEG signals with different acquisition precisions.

[0014] In one possible implementation of the first aspect, the step of obtaining the high-level neuro-semantic representation corresponding to the brain-computer interface signal by performing Bayesian joint inference based on the low-level neural observation representation and the mid-level structural prior representation includes: The first feature information is obtained by statistically analyzing the feature distribution of the low-level neural observation representation. The second feature information is obtained by statistically analyzing the feature distribution of the prior representation of the middle layer structure. Based on the first feature information and the second feature information, Bayesian joint inference is performed to obtain the high-level meta-neural semantic representation.

[0015] In the above implementation method, the semantic features of the brain-computer interface signal are inferred by Bayesian joint inference after separately statistically analyzing the distribution information of the two types of features. This fully explores the effective information of the distribution dimensions of the two types of features, and the fusion can improve the completeness of semantic feature extraction and prediction accuracy.

[0016] In one possible implementation of the first aspect, the step of obtaining the high-level neuro-semantic representation corresponding to the brain-computer interface signal by performing Bayesian joint inference based on the low-level neural observation representation and the mid-level structural prior representation includes: Obtain the trained weight parameters; wherein the weight parameters are generated based on the non-negativity constraint function; Based on the weight parameters, Bayesian joint inference is performed on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neural semantic representation.

[0017] In the above implementation, the weight parameters obtained by training with non-negative constraints are used to fuse two types of features to derive the semantic features of brain-computer interface signals. The non-negative constraint of the weights can avoid interference from invalid negative weights, making the feature fusion results more stable and in line with the physical characteristics of EEG signals, thereby improving the accuracy and reliability of semantic feature prediction.

[0018] In one possible implementation of the first aspect, the step of performing semantic decoding based on the high-level meta-neural semantic representation to obtain the decoding result includes: Obtain the calibration parameters after training; Based on the calibration parameters, the high-level neuron semantic representation is converted into information recognizable by the trained decoder to obtain the input information; The input information is input into the decoder to obtain the decoding result.

[0019] In the above implementation method, the semantic features are uniformly converted into decoder-adapted input by using the trained calibration parameters, and then the decoding is completed. This can effectively eliminate the deviation of the mismatch between the semantic features and the decoder input format, and improve the adaptability of semantic decoding and the accuracy of output results.

[0020] Secondly, embodiments of this application provide a Bayesian brain-computer interface decoding device, comprising: Acquisition unit, used to acquire brain-computer interface signals from the target subject; The encoding unit is used to encode the brain-computer interface signals to obtain low-level neural observation representations; The constraint unit is used to constrain the brain-computer interface signal according to the preset brain structure prior information to obtain the mid-level structure prior representation. The inference unit is used to perform Bayesian joint inference based on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level meta-neural semantic representation corresponding to the brain-computer interface signal; wherein, the high-level meta-neural semantic representation is used to represent the cognitive state inside the brain. The decoding unit is used to perform semantic decoding based on the high-level meta-neural semantic representation to obtain the decoding result.

[0021] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the Bayesian brain-inspired brain-computer interface decoding method as described in any of the first aspects above.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the Bayesian brain-inspired brain-computer interface decoding method as described in any of the first aspects above.

[0023] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the Bayesian brain-computer interface decoding method described in any of the first aspects above.

[0024] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a Bayesian brain-computer interface decoding method provided in one embodiment of this application. Figure 2 This is a schematic diagram of the structure of a Bayesian brain-computer interface decoding device provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0033] Brain-computer interfaces (BCIs) establish direct pathways between brain neural activity and external devices, playing a crucial role in neural function reconstruction, assisted communication, and human-machine collaboration. Decoding is the core component of a BCI system. Its task is to convert neural observation signals, such as those acquired through electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), electrocogesis (ECoG), functional near-infrared spectroscopy (fNIRS), functional ultrasound imaging (fUS), stereoelectroencephalography (sEEG), and microelectrode arrays (MEA), into information usable by external systems. In other words, the goal of BCI decoding is to convert the neural activity of a subject in tasks such as motor imagery, emotion recognition, and open semantic expression into results usable by external systems, such as control commands, natural language text, and visual content. Existing decoding methods typically target external stimuli, making it difficult to decode the subject's own internal semantic judgments and subjective evaluations, resulting in poor decoding performance.

[0034] Based on this, this application provides a Bayesian brain-inspired brain-computer interface decoding method. In this application, the low-level neural observation representation is equivalent to the observation features obtained by encoding the brain-computer interface signal; the mid-level structural prior representation is the feature obtained by constraining the brain-computer interface signal based on preset brain structural prior information, which is equivalent to subjecting the brain-computer interface signal to brain structural prior constraints; then, Bayesian joint inference is performed based on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neuro-semantic representation corresponding to the brain-computer interface signal, which is equivalent to fusing the observation features obtained by encoding the brain-computer interface signal with the brain structural prior constraint features to jointly infer neural semantics. By relying on physiological priors to correct signal noise and spatial resolution defects, more accurate and physiologically interpretable brain-computer semantic decoding is achieved, thereby improving the decoding effect of brain-computer interface signals.

[0035] See Figure 1 This is a flowchart illustrating the Bayesian brain-computer interface decoding method provided in this application embodiment. It is intended as an example and not a limitation. The method may include the following steps: S101, acquire brain-computer interface signals from the target subject.

[0036] Brain-computer interfaces (BCIs) establish a direct pathway between brain neural activity and external devices.

[0037] This application supports both non-invasive and invasive neural observation modalities. The non-invasive modalities include electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), and functional ultrasound imaging (fUS). The invasive modalities include stereoelectroencephalography (sEEG), electrocorticography (ECoG), and microelectrode arrays (MEAs).

[0038] In other words, the brain-computer interface signals in the embodiments of this application may include non-invasive modal signals, such as EEG signals, MEG signals, fMRI signals, fNIRS signals, and fUS signals; or they may include invasive modal signals, such as ECoG signals, sEEG signals, and MEA signals.

[0039] S102 encodes the brain-computer interface signal to obtain a low-level neural observation representation.

[0040] In one embodiment, S102 includes: Obtain the pre-trained encoder; encode the brain-computer interface signals based on the pre-trained encoder to obtain the encoded information.

[0041] The pre-trained encoder is trained based on a loss function, which is used to evaluate the encoding results of the training samples after masking. The masking process is used to mask part of the signal in the training samples.

[0042] For example, the loss function is ;in, and These are the original training samples and the encoded training samples, respectively. For masking instructions, such as a mask matrix.

[0043] In the above implementation, since the masking process can mask part of the signal in the training sample, it is equivalent to increasing the training difficulty and increasing the encoder's ability to understand the signal. Therefore, the encoder trained according to the loss function used to evaluate the encoding result of the training sample after masking can understand the brain-computer interface signal more accurately, thereby encoding more accurate encoded information.

[0044] In some implementations, large-scale neural observation data can be used as training samples to pre-train the encoder. These training samples can cover multiple subjects, various recording conditions, and multiple task scenarios.

[0045] Optionally, the encoder may include encoding modules for different types of neural observation data. These modules may employ different encoding methods to achieve accurate encoding of different types of neural observation data. For example, the encoder may include a first encoding module and a second encoding module. The first encoding module encodes fMRI signals, and the second encoding module encodes EEG and / or MEG signals. The first encoding module may map the input signal to a cortical coordinate space and combine this with cortical vertex coordinates, brain region location encoding, spatial adjacency relationships, and brain region masking reconstruction to learn a neural observation representation that preserves the spatial organizational relationships of the cortex, thus obtaining a low-level neural observation representation that reflects the distribution of blood oxygen acquisition. The second encoding module may represent the input signal as a neural representation sequence containing channel, time segment, and frequency band information, and combine this with channel location encoding, time location encoding, and a missing mask to adapt to data with different numbers of channels, sampling rates, and recording durations. It is understood that the first encoding module learns the spatial sequence features of the input signal, while the second encoding module learns the neural temporal dynamics features of the input signal.

[0046] S103, constrain the brain-computer interface signal according to the preset brain structure prior information to obtain the mid-level structure prior representation.

[0047] In one embodiment, prior information on brain structure includes prior information based on the geometry of the cerebral cortex.

[0048] Accordingly, S103 includes: Obtain prior information based on the geometry of the cerebral cortex; The brain-computer interface signal is constrained based on the prior information constructed from the geometry of the cerebral cortex to obtain the prior representation of the middle layer structure.

[0049] In the above implementation method, the brain-computer interface signal is constrained by the prior information of brain structure constructed by the geometry of the cerebral cortex, and a prior representation of the middle structure is generated. This can use the prior information of human brain anatomical geometry to constrain the EEG signal, reduce irrelevant noise interference, and improve the physiological authenticity and feature discrimination of the brain-computer signal representation, thereby helping to improve the decoding accuracy.

[0050] In one implementation, the prior information based on the construction of cerebral cortex geometry includes a general cortical geometry spectrum; the general cortical geometry spectrum includes multiple cortical spectra of the cerebral cortex and the spectral value corresponding to each cortical spectra.

[0051] Accordingly, the step of constraining the brain-computer interface signal based on the prior information constructed from the cerebral cortex geometry to obtain the prior representation of the mid-level structure includes: Obtain the individual cortical geometry spectrum of the target subject; The target cortical geometry spectrum is obtained by adjusting the general cortical geometry spectrum based on the individual cortical geometry spectrum. Calculate the geometric spectral coefficients corresponding to the brain-computer interface signals based on the geometric spectrum of the target cortex.

[0052] The cortical geometric spectrum, also known as the cerebral cortex geometric eigenvalue spectrum, is essentially a set of cortical spectra (or cortical geometric eigenmodes, basis functions) and corresponding spectral values ​​(eigenvalues) ordered by frequency from smallest to largest using the Laplace-Beltrammian operator (LBO) to perform spectral decomposition on the surface grid of the cerebral cortex. This entire set of basis functions and spectral values ​​is called the cortical geometric spectrum (PMC). In short, if we consider the folded surface of the cerebral cortex as a "vibrating membrane," the geometric spectrum is the inherent set of all the fundamental vibrational waveforms (cortical spectra) and corresponding vibrational frequencies (spectral values) of this membrane. It is entirely determined by the cortex's own three-dimensional folds, curvature, and spatial topological geometry, and is unrelated to electroencephalogram (EEG) signals.

[0053] For example, according to the formula The geometric spectrum of the target cortex is obtained. Among them, This represents the spatial correlation coefficient between the individual cortical geometric spectrum and the universal cortical geometric spectrum. For symbolic functions, Let be the spectral value of the i-th cortical spectrum in the individual cortical geometry spectrum of the target subject. is the spectral value of the i-th cortical spectrum in the general cortical geometric spectrum.

[0054] Optionally, during the training phase, only a general cortical geometry spectrum can be used for training, and then adjustments can be made based on the individual cortical geometry spectrum during the application phase.

[0055] Optionally, during the training phase, adjustments can be made to the individual cortical geometry of each subject corresponding to each training sample, and then training can be performed based on the adjusted target cortical geometry spectrum.

[0056] It is understandable that if adjustments are made only based on the individual cortical geometry spectrum during the application phase, then during the training process, the geometric spectral coefficients of the brain-computer interface signals corresponding to the training samples are calculated based on the general cortical geometry spectrum.

[0057] In the above implementation method, personalized adjustments are made for the target subject. The general cortical geometry spectrum is corrected by combining the individual cortical geometry spectrum to obtain the target cortical geometry spectrum that is suitable for the target subject. Then, the geometry spectrum coefficients are solved as prior information of brain structure. Taking into account the differences between the general anatomical base and individual brain morphology, it can effectively improve the personalized adaptation of brain-computer interface signal constraints, thereby helping to improve the accuracy of semantic inference.

[0058] As described in the above embodiments, brain-computer interface signals can include various types of neural observation signals, and different types of neural observation signals have different resolutions. Based on this consideration, in this application embodiment, the geometric spectral coefficients are calculated in different ways for different types of neural observation signals, specifically in the following two cases.

[0059] Case 1: For the first type of signal in the brain-computer interface signal, project the first type of signal onto the geometric spectrum of the target cortex to obtain the geometric spectrum coefficients corresponding to the first type of signal.

[0060] For example, the first type of signal is fMRI. The specific steps for obtaining the geometric spectral coefficients of the first type of signal may include: projecting the fMRI signal onto the base of the target cortical geometric spectrum; and solving for the geometric spectral coefficients using methods such as least squares and regularized least squares.

[0061] Case 2: For the second type of signal in the brain-computer interface signal, project the geometric spectrum of the target cortex onto the sensor space corresponding to the second type of signal to obtain the sensor space geometric pattern matrix; Perform matrix decomposition or orthogonalization on the sensor spatial geometric pattern matrix to filter out effective sensor spatial geometric patterns. The geometric spectral coefficients corresponding to the second type of signal are calculated based on the spectral values ​​corresponding to each cortical spectrum in the effective sensor spatial geometric pattern and the target cortical geometric spectrum; wherein, the spatial resolution of the second type of signal is lower than that of the first type of signal.

[0062] Optionally, matrix decomposition can employ singular value decomposition. Correspondingly, the specific process in Case Two may include: selecting effective sensor spatial geometric patterns based on preset effective component selection criteria to construct an orthogonal sensor spatial geometric basis; projecting the target cortical geometric spectrum corresponding to the brain-computer interface signal onto this sensor spatial geometric basis to obtain geometric spectral coefficients. For example, during the selection process, effective sensor spatial geometric patterns can be selected based on the ratio of singular values ​​to the maximum singular value. In an example where the preset effective component selection criterion is 0.001, left singular vectors with singular values ​​greater than the maximum singular value of 0.0001 can be retained to construct an orthogonal sensor spatial geometric basis.

[0063] Alternatively, matrix decomposition can also employ eigenvalue decomposition, rank-revealing decomposition, or other matrix processing methods that can obtain effective orthogonal bases.

[0064] For example, the second type of signal is EEG or MEG. The steps for calculating the geometric spectral coefficients of the second type of signal may include: projecting the target cortical geometric spectrum onto the sensor space corresponding to the EEG or MEG to obtain the sensor space geometric pattern matrix; performing singular value decomposition or orthogonalization on the sensor space geometric pattern matrix to obtain the decomposed elements; and then applying the formula... Calculate the geometric spectral coefficients; where, Let i be the geometric spectral coefficient. The element in the j-th row and i-th column is the decomposition element. Let be the spectral value of the j-th cortical spectrum in the target cortical geometric spectrum.

[0065] In the above implementation, a differentiated geometric spectral coefficient solution strategy is adopted for two types of brain-computer interface signals with different spatial resolutions. This strategy takes into account the efficient solution of high-precision signals by direct projection and the adaptation of low-resolution signals to cortical geometric spectrum calculation through spatial projection and matrix decomposition. The spectral coefficient conversion of the two types of signals is completed by relying on the target cortical geometric spectrum, which adapts to the cortical map analysis requirements of EEG signals with different acquisition precisions.

[0066] It is understood that, in some embodiments, in addition to prior information based on cortical geometry, brain structural prior information may also include intrinsic brain prior information constructed based on anatomical connectivity, functional connectivity, and / or neurodynamic principles. In still other embodiments, brain structural prior information may be a combination of one or more of the following: prior information based on cortical geometry, intrinsic brain prior information constructed based on anatomical connectivity, functional connectivity, and neurodynamic principles. For example, intrinsic brain prior information based on anatomical connectivity includes an anatomical connectivity matrix; intrinsic brain prior information based on functional connectivity includes a functional connectivity matrix; and intrinsic brain prior information constructed based on neurodynamic principles includes neurodynamic equations. In still other embodiments, brain structural prior information may be combined with other prior information to constrain brain-computer interface information. For example, brain structural prior information may be combined with semantic concept prior information to jointly constrain brain-computer interface signals.

[0067] S104. Based on the low-level neural observation representation and the mid-level structural prior representation, Bayesian joint inference is performed to obtain the high-level neural semantic representation corresponding to the brain-computer interface signal.

[0068] The high-level neuro-semantic representation is used to characterize the cognitive state within the brain.

[0069] In one embodiment, S104 includes: The first feature information is obtained by statistically analyzing the feature distribution of low-level neural observation representations. The second feature information is obtained by statistically analyzing the feature distribution of the prior representation of the middle layer structure. Based on the first and second feature information, Bayesian joint inference is performed to obtain the high-level neuron semantic representation.

[0070] In the above implementation method, the semantic features of the brain-computer interface signal are inferred by Bayesian joint inference after separately statistically analyzing the distribution information of the two types of features. This fully explores the effective information of the distribution dimensions of the two types of features, and the fusion can improve the completeness of semantic feature extraction and prediction accuracy.

[0071] In one implementation, a Bayesian joint inference can be performed on the first and second feature information based on a reconstruction model using Bayes' theorem.

[0072] Optionally, the first feature information includes the mean and variance, and the second feature information includes the mean and variance.

[0073] For example, according to the formula The high-level meta-neural semantic representation is obtained. Here, z represents the high-level meta-neural semantic representation. , These are the mean and variance, respectively, from the first feature information. , These are the mean and variance, respectively, in the second feature information.

[0074] In the above implementation method, feature fusion based on the Bayesian reconstruction model can simultaneously incorporate the observational likelihood of brain-computer signals and the prior constraints of brain structure. By fusing multi-dimensional brain features under a unified probabilistic framework, it can effectively suppress reconstruction artifacts and improve the physiological rationality and prediction accuracy of cortical representation and semantic feature extraction.

[0075] In another embodiment, S104 includes: Obtain the trained weight parameters; wherein the weight parameters are generated based on the non-negativity constraint function; Based on the weight parameters, Bayesian joint inference is performed on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neural semantic representation.

[0076] In the above implementation, the weight parameters obtained by training with non-negative constraints are used to fuse two types of features to derive the semantic features of brain-computer interface signals. The non-negative constraint of the weights can avoid interference from invalid negative weights, making the feature fusion results more stable and in line with the physical characteristics of EEG signals, thereby improving the accuracy and reliability of semantic feature prediction.

[0077] For example, according to the formula High-level neuron semantic representations were obtained. For low-level neural observation characterization, This is a priori representation of the mid-layer structure; and These are the weight parameters for low-level neural observation representation and mid-level structural prior representation, respectively; For non-negativity constraint functions, the softplus function can be used; and These are the learnable scalar parameters.

[0078] Understandably, the above implementation method includes learnable weight parameters. During training, for the first training iteration, the weight parameters can be set to initial values, for example, by setting... and When set to 0, and using softplus, =ln2; After obtaining the decoding result, if the current training does not meet the preset training conditions (such as the number of iterations not reaching the preset number, or the decoding accuracy not reaching the preset accuracy), then adjust the weight parameters and continue the next round of training; and so on, until the training is completed. Accordingly, the trained weight parameters can be obtained after the training is completed.

[0079] S105, semantic decoding is performed based on the high-level neuron semantic representation to obtain the decoding result.

[0080] In one embodiment, the calibration parameters after training are obtained; Based on the calibration parameters, the high-level neuron semantic representation is converted into information that the trained decoder can recognize, thus obtaining the input information; the input information is then input into the decoder to obtain the decoding result.

[0081] For example, according to the formula Perform the conversion. Among them, This represents a high-level neuronal semantic representation. The converted input information, and These are learnable calibration parameters.

[0082] Understandably, during the training process, for the first training session, the calibration parameters can be set to initial values. After obtaining the decoding results, if the current training does not meet the preset training conditions (such as the number of iterations not reaching the preset number, or the decoding accuracy not reaching the preset accuracy), the calibration parameters are adjusted, and the next round of training continues. This process is repeated until training is completed. Accordingly, the calibration parameters after training can be obtained after training is completed.

[0083] In the above implementation method, the semantic features are uniformly converted into decoder-adapted input by using the trained calibration parameters, and then the decoding is completed. This can effectively eliminate the deviation of the mismatch between the semantic features and the decoder input format, and improve the adaptability of semantic decoding and the accuracy of output results.

[0084] Optionally, semantic decoding can be a discriminative decoding task. For example, the decoding goal could be to determine the target subject's emotions, motor intentions, fatigue state, etc., based on high-level meta-neural semantic representations.

[0085] Optionally, semantic decoding can be a generative decoding task. For example, the decoding goal is to generate text, speech, images, etc., based on high-level meta-neural semantic representations.

[0086] Figure 1 In the illustrated embodiment, the low-level neural observation representation is equivalent to the observation features obtained from the encoding of the brain-computer interface signal; the mid-level structural prior representation is the feature obtained after constraining the brain-computer interface signal based on preset brain structural prior information, which is equivalent to subjecting the brain-computer interface signal to brain structural prior constraints; then, the high-level neural semantic representation corresponding to the brain-computer interface signal is inferred by jointly inferring the low-level neural observation representation and the mid-level structural prior representation, which is equivalent to jointly inferring the neural semantics by fusing the observation features obtained from the encoding of the brain-computer interface signal with the brain structural prior constraint features. By relying on physiological priors to correct signal noise and spatial resolution defects, more accurate and physiologically interpretable brain-computer semantic decoding is achieved, which is beneficial to improving the decoding effect of brain-computer interface signals.

[0087] It should be noted that the embodiments of this application include a training process and an application process. Figure 1 The described process is an application process. The following is an example of a training process, specifically including: S201, Obtain training samples; wherein each training sample includes a brain-computer interface signal from a subject.

[0088] S202, based on the training samples, encodes them to obtain the low-level neural observation representation of the training samples.

[0089] S203, constrain the training samples according to the preset brain structure prior information to obtain the mid-level structure prior representation of the training samples.

[0090] S204. Based on the low-level neural observation representation and the mid-level structural prior representation of the training samples, Bayesian joint inference is performed to obtain the high-level neural semantic representation corresponding to the training samples.

[0091] Understandably, during the first training process, S204 jointly infers the fifth and sixth feature information based on the initial values ​​of the weight parameters to obtain the high-level neuron semantic representation.

[0092] S205, Semantic decoding is performed based on the high-level meta-neural semantic representations corresponding to the training samples to obtain the first training result.

[0093] Understandably, during the first training process, S205 converts the high-level meta-neural semantic representations of the training samples into information recognizable by the decoder based on the initial values ​​of the calibration parameters.

[0094] The implementation methods of S201-S205 are the same as those of S101-S105, and the details can be found in the descriptions in the embodiments of S101-S105.

[0095] S206, determine whether the first training result meets the preset training conditions.

[0096] Optionally, the training condition is that the number of iterations has not reached the preset number, or the decoding accuracy has not reached the preset accuracy.

[0097] S207 If the training conditions are not met, adjust the calibration parameters and weight parameters, and continue with the next training iteration until the training ends.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0099] Corresponding to the Bayesian brain-inspired brain-computer interface decoding method described in the above embodiments, Figure 2 This is a structural block diagram of the Bayesian brain-computer interface decoding device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0100] Reference Figure 2 The device 2 includes: Acquisition unit 21 is used to acquire brain-computer interface signals from the target subject.

[0101] The encoding unit 22 is used to encode the brain-computer interface signal to obtain a low-level neural observation representation.

[0102] The constraint unit 23 is used to constrain the brain-computer interface signal according to the preset brain structure prior information to obtain the mid-level structure prior representation.

[0103] The inference unit 24 is used to perform Bayesian joint inference based on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level meta-neural semantic representation corresponding to the brain-computer interface signal; wherein, the high-level meta-neural semantic representation is used to represent the cognitive state inside the brain.

[0104] Decoding unit 25 is used to perform semantic decoding based on the high-level meta-neural semantic representation to obtain the decoding result.

[0105] Optionally, the encoding unit 22 is also used for: Obtain a pre-trained encoder; wherein the pre-trained encoder is trained based on a loss function, the loss function being used to evaluate the encoding result of the training samples after masking, and the masking being used to mask part of the signal in the training samples; The brain-computer interface signal is encoded using the pre-trained encoder to obtain encoded information.

[0106] Optionally, constraint unit 23 is also used for: Obtain the prior information based on the geometry of the cerebral cortex; The brain-computer interface signal is constrained based on the prior information constructed from the geometry of the cerebral cortex to obtain the prior representation of the middle layer structure.

[0107] Optionally, constraint unit 23 is also used for: Obtain the individual cortical geometry spectrum of the target subject; The target cortical geometry spectrum is obtained by adjusting the general cortical geometry spectrum based on the individual cortical geometry spectrum. The geometric spectral coefficients corresponding to the brain-computer interface signal are calculated based on the geometric spectrum of the target cortex.

[0108] Optionally, constraint unit 23 is also used for: For the first type of signal in the brain-computer interface signal, the first type of signal is projected onto the geometric spectrum of the target cortex to obtain the geometric spectrum coefficients corresponding to the first type of signal; For the second type of signal in the brain-computer interface signal, the geometric spectrum of the target cortex is projected onto the sensor space corresponding to the second type of signal to obtain the sensor space geometric pattern matrix; The sensor spatial geometric pattern matrix is ​​decomposed or orthogonalized to filter out effective sensor spatial geometric patterns. The geometric spectral coefficients corresponding to the second type of signal are calculated based on the effective sensor spatial geometric pattern and the spectral values ​​corresponding to each cortical spectrum in the target cortical geometric spectrum; wherein the spatial resolution of the second type of signal is lower than that of the first type of signal.

[0109] Optionally, the speculation unit 24 is also used for: The first feature information is obtained by statistically analyzing the feature distribution of the low-level neural observation representation. The second feature information is obtained by statistically analyzing the feature distribution of the prior representation of the middle layer structure. Based on the first feature information and the second feature information, Bayesian joint inference is performed to obtain the high-level meta-neural semantic representation.

[0110] Optionally, the speculation unit 24 is also used for: Obtain the trained weight parameters; wherein the weight parameters are generated based on the non-negativity constraint function; Based on the weight parameters, Bayesian joint inference is performed on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neural semantic representation.

[0111] Optionally, the decoding unit 25 is also used for: Obtain the calibration parameters after training; Based on the calibration parameters, the high-level neuron semantic representation is converted into information recognizable by the trained decoder to obtain the input information; The input information is input into the decoder to obtain the decoding result.

[0112] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0113] in addition, Figure 2 The Bayesian brain-computer interface decoding device shown can be a software unit, a hardware unit, or a combination of software and hardware built into an existing terminal device, or it can be integrated into the terminal device as an independent component, or it can exist as a standalone terminal device.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0115] Figure 3 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 3 As shown, the terminal device 3 in this embodiment includes: at least one processor 30 ( Figure 3(Only one is shown in the diagram) a processor, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 executes the computer program 32 to implement the steps in any of the above embodiments of the Bayesian brain-inspired brain-computer interface decoding methods.

[0116] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 3 This is merely an example of terminal device 3 and does not constitute a limitation on terminal device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0117] The processor 30 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0118] In some embodiments, the memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. In other embodiments, the memory 31 may be an external storage device of the terminal device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 3. Furthermore, the memory 31 may include both internal and external storage units of the terminal device 3. The memory 31 is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0119] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0120] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments.

[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0124] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A Bayesian brain-computer interface decoding method, characterized in that, include: Acquire brain-computer interface signals from the target subject; The brain-computer interface signals are encoded to obtain low-level neural observation representations. The brain-computer interface signal is constrained based on the preset prior information of brain structure to obtain the prior representation of the middle layer structure. Based on the low-level neural observation representation and the mid-level structural prior representation, Bayesian joint inference is performed to obtain the high-level meta-neural semantic representation corresponding to the brain-computer interface signal; wherein, the high-level meta-neural semantic representation is used to represent the cognitive state inside the brain; Semantic decoding is performed based on the high-level neuromorphic semantic representation to obtain the decoding result.

2. The method as described in claim 1, characterized in that, The process of encoding the brain-computer interface signals to obtain low-level neural observation representations includes: Obtain a pre-trained encoder; wherein the pre-trained encoder is trained based on a loss function, the loss function being used to evaluate the encoding result of the training samples after masking, and the masking being used to mask part of the signal in the training samples; The brain-computer interface signal is encoded using the pre-trained encoder to obtain encoded information.

3. The method as described in claim 1, characterized in that, The prior information on brain structure includes prior information based on the geometry of the cerebral cortex; The step of constraining the brain-computer interface signal based on preset prior information about brain structure to obtain a priori representation of the mid-level structure includes: Obtain the prior information based on the geometry of the cerebral cortex; The brain-computer interface signal is constrained based on the prior information constructed from the geometry of the cerebral cortex to obtain the prior representation of the middle layer structure.

4. The method as described in claim 3, characterized in that, The prior information based on the geometry of the cerebral cortex includes a general cortical geometry spectrum; the general cortical geometry spectrum includes multiple cortical spectra of the cerebral cortex and the spectral value corresponding to each cortical spectra; The step of constraining the brain-computer interface signal based on the prior information constructed from the cerebral cortex geometry to obtain the prior representation of the mid-level structure includes: Obtain the individual cortical geometry spectrum of the target subject; The target cortical geometry spectrum is obtained by adjusting the general cortical geometry spectrum based on the individual cortical geometry spectrum. The geometric spectral coefficients corresponding to the brain-computer interface signal are calculated based on the geometric spectrum of the target cortex.

5. The method as described in claim 4, characterized in that, The step of calculating the geometric spectral coefficients corresponding to the brain-computer interface signal based on the target cortical geometric spectrum includes: For the first type of signal in the brain-computer interface signal, the first type of signal is projected onto the geometric spectrum of the target cortex to obtain the geometric spectrum coefficients corresponding to the first type of signal; For the second type of signal in the brain-computer interface signal, the geometric spectrum of the target cortex is projected onto the sensor space corresponding to the second type of signal to obtain the sensor space geometric pattern matrix; The sensor spatial geometric pattern matrix is ​​decomposed or orthogonalized to filter out effective sensor spatial geometric patterns. The geometric spectral coefficients corresponding to the second type of signal are calculated based on the effective sensor spatial geometric pattern and the spectral values ​​corresponding to each cortical spectrum in the target cortical geometric spectrum; wherein the spatial resolution of the second type of signal is lower than that of the first type of signal.

6. The method according to any one of claims 1 to 5, characterized in that, The step of performing Bayesian joint inference based on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neuro-semantic representation corresponding to the brain-computer interface signal includes: The first feature information is obtained by statistically analyzing the feature distribution of the low-level neural observation representation. The second feature information is obtained by statistically analyzing the feature distribution of the prior representation of the middle layer structure. Based on the first feature information and the second feature information, Bayesian joint inference is performed to obtain the high-level meta-neural semantic representation.

7. The method according to any one of claims 1 to 5, characterized in that, The step of performing Bayesian joint inference based on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neuro-semantic representation corresponding to the brain-computer interface signal includes: Obtain the trained weight parameters; wherein the weight parameters are generated based on the non-negativity constraint function; Based on the weight parameters, Bayesian joint inference is performed on the low-level neural observation representation and the mid-level structural prior representation to obtain the high-level neural semantic representation.

8. The method as described in claim 1, characterized in that, The step of performing semantic decoding based on the high-level neuron semantic representation to obtain the decoding result includes: Obtain the calibration parameters after training; Based on the calibration parameters, the high-level neuron semantic representation is converted into information recognizable by the trained decoder to obtain the input information; The input information is input into the decoder to obtain the decoding result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images