Brain-computer interface signal processing method and brain-computer interface system

Abstract

Embodiments of the present disclosure provide a brain-computer interface signal processing method and a brain-computer interface system. The brain-computer interface signal processing method comprises: receiving a brain activity detection signal from a target object; mapping a parameter matrix G to a computing-in-memory array, wherein a first matrix H, a second matrix W and a third matrix T for performing time domain filtering, spatial filtering and template matching on the brain activity detection signal are obtained on the basis of a training model comprising a plurality of stimulation task templates, the parameter matrix G for further decoding the brain activity detection signal is computed, and G=TWH; using the computing-in-memory array to which the parameter matrix G is mapped to further decode the brain activity detection signal, to obtain a decoding result corresponding to the brain activity detection signal; on the basis of the decoding result, determining, from among the plurality of stimulation task templates, a control instruction of a target stimulation task corresponding to the brain activity detection signal; and sending the control instruction to execute the task. According to the brain-computer interface signal processing method, the computation amount and computation delay for an algorithm can be reduced, thereby reducing circuit area overhead and energy consumption overhead.

Description

Brain-computer interface signal processing method and brain-computer interface system

[0001] This application claims priority to Chinese Patent Application No. 202410109207.6 filed on January 25, 2024, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to a brain-computer interface signal processing method and a brain-computer interface system. Background Art

[0003] Brain-Computer Interface (BCI) technology enables information exchange and control by directly connecting the human brain and external devices. BCI allows the human brain to communicate directly with computer systems or other artificial devices without going through traditional input and output channels such as keyboards, mice, or touch screens.

[0004] Brain-computer interface technology uses various physiological signals to capture brain activity and converts these signals into commands or control signals that can be understood by external devices. Through this technology, people can control external devices such as moving a wheelchair, operating a robotic arm, and operating a computer simply by thinking or using brain electrical activity. Summary of the Invention

[0005] At least one embodiment of the present disclosure provides a brain-computer interface signal processing method, which includes: receiving a brain activity detection signal from a target object; mapping a parameter matrix G into a storage and computing integrated array, wherein a first matrix H for time-domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal, and a third matrix T for template matching of the brain activity detection signal are obtained according to a training model including multiple stimulation task templates, and the parameter matrix G for one-step decoding processing of the brain activity detection signal is calculated according to the first matrix H, the second matrix W, and the third matrix T, where G=TWH; using the storage and computing integrated array mapping the parameter matrix G to perform the one-step decoding processing on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal; determining, according to the decoding result, a control instruction for the target stimulation task corresponding to the brain activity detection signal from the multiple stimulation task templates; and sending the control instruction to execute the task.

[0006] For example, in the brain-computer interface signal processing method provided in at least one embodiment of the present disclosure, the storage and computing integrated array that maps the parameter matrix G is used to perform the one-step decoding processing on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal, including: inputting the brain activity detection signal into multiple signal input terminals of the storage and computing integrated array that maps the parameter matrix G; controlling the storage and computing integrated array that maps the parameter matrix G to perform multiplication and accumulation calculations on the brain activity detection signal, so as to obtain multiple analog output signals after the multiplication and accumulation calculations are performed from multiple signal output terminals of the storage and computing integrated array; receiving multiple digital output signals after analog-to-digital conversion of the multiple analog output signals, and obtaining a decoding result corresponding to the brain activity detection signal.

[0007] For example, the brain-computer interface signal processing method provided by at least one embodiment of the present disclosure also includes: obtaining the number of signal channels Nc and the number of sampling points Ns of the brain activity detection signal; determining the number of stimulation categories Nf according to the number of the multiple stimulation task templates, and determining the number of storage and computing units used for calculation in the storage and computing array as Nc×Ns×Nf according to the number of stimulation categories Nf, the number of signal channels Nc and the number of sampling points Ns.

[0008] For example, the brain-computer interface signal processing method provided by at least one embodiment of the present disclosure also includes: determining the size of the test data matrix as P×Q based on the number of signal channels Nc and the number of sampling points Ns of the brain activity detection signal, where P=Nc×Ns, Q=1.

[0009] For example, in the brain-computer interface signal processing method provided in at least one embodiment of the present disclosure, a first matrix H for time domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal, and a third matrix T for template matching of the brain activity detection signal are obtained according to a training model including multiple stimulation task templates, including: obtaining a time domain filter response function for performing the time domain filtering on the brain activity detection signal, a spatial domain filter matrix for performing the spatial filtering on the brain activity detection signal, and the multiple stimulation task templates; obtaining the first matrix H, the second matrix W, and the third matrix T with mutually matching dimensions according to the test data matrix and the time domain filter response function, the spatial domain filter matrix, and the multiple stimulation task templates.

[0010] For example, the brain-computer interface signal processing method provided by at least one embodiment of the present disclosure also includes: in response to the time domain filtering and the spatial filtering of the brain activity detection signal, approximating the multiple parameters in the third matrix T as multiple templates after normalizing the multiple stimulation task templates.

[0011] For example, in the brain-computer interface signal processing method provided in at least one embodiment of the present disclosure, the brain activity detection signal includes a steady-state visual evoked potential signal or a functional near-infrared spectroscopy signal.

[0012] For example, in the brain-computer interface signal processing method provided in at least one embodiment of the present disclosure, the training model includes a model trained according to an integrated task-related component analysis algorithm.

[0013] At least one embodiment of the present disclosure further provides a brain-computer interface system, which includes a signal receiving device, a processing device and an output device, wherein the signal receiving device is configured to receive a brain activity detection signal from a target object; the processing device includes a storage and computing integrated array and is configured to: map a parameter matrix G into the storage and computing integrated array, wherein a first matrix H for time-domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal and a third matrix T for template matching of the brain activity detection signal are obtained according to a training model including multiple stimulation task templates, and the parameter matrix G for one-step decoding processing of the brain activity detection signal is calculated according to the first matrix H, the second matrix W and the third matrix T, G=TWH; the brain activity detection signal is subjected to the one-step decoding processing using the storage and computing integrated array that maps the parameter matrix G to obtain a decoding result corresponding to the brain activity detection signal; based on the decoding result, a control instruction of the target stimulation task corresponding to the brain activity detection signal is determined from the multiple stimulation task templates; the output device is configured to send the control instruction to execute the task.

[0014] For example, the brain-computer interface system provided by at least one embodiment of the present disclosure further includes an effector, wherein the effector is coupled to the output device and is configured to be controlled by the control instruction to perform a corresponding operation.

[0015] For example, in the brain-computer interface system provided in at least one embodiment of the present disclosure, the storage and computing array includes a memristor array. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0017] FIG1A is a schematic structural diagram of an exemplary integrated storage and computing array provided by at least one embodiment of the present disclosure;

[0018] FIG1B is a schematic diagram of a storage-computing integrated unit with a 1T1R structure provided by at least one embodiment of the present disclosure;

[0019] FIG1C is a schematic diagram of a 2T2R storage-computing integrated unit provided by at least one embodiment of the present disclosure;

[0020] FIG1D is a schematic diagram of a storage and computation integrated array capable of realizing positive and negative values, provided by at least one embodiment of the present disclosure;

[0021] FIG2 is a flowchart of a brain-computer interface signal processing method provided by at least one embodiment of the present disclosure;

[0022] FIG3 is a schematic diagram of an exemplary time-domain filter provided by at least one embodiment of the present disclosure;

[0023] FIG4 is a schematic diagram of an exemplary spatial filter matrix provided by at least one embodiment of the present disclosure;

[0024] FIG5 is a schematic diagram of an exemplary stimulation task template provided by at least one embodiment of the present disclosure;

[0025] FIG6 is a schematic diagram illustrating mapping values ​​of a parameter matrix for one-step decoding processing onto a storage-computation integrated array according to at least one embodiment of the present disclosure;

[0026] FIG7 is a schematic diagram showing a comparison of the computational complexity of an original decoding process and a one-step decoding process provided by at least one embodiment of the present disclosure;

[0027] FIG8 is a schematic block diagram of a brain-computer interface system provided by at least one embodiment of the present disclosure; and

[0028] FIG9 is a schematic structural diagram of an electronic device provided by at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] Unless otherwise defined, technical or scientific terms used in this disclosure should have the ordinary meanings understood by a person of ordinary skill in the art to which this disclosure belongs. The terms "first," "second," and similar terms used in this disclosure do not denote any order, quantity, or importance, but are simply used to distinguish different components. Terms such as "include" or "comprising" mean that the element or object preceding the term includes the elements or objects listed after the term, and their equivalents, without excluding other elements or objects. Terms such as "connected" or "connected" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit steps shown. The scope of this disclosure is not limited in this respect.

[0031] The present disclosure is described below using several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numeral in each drawing.

[0032] Brain-computer interface technology can use EEG signals or other physiological signals to capture the brain activity of humans or animals. For example, EEG signals can be collected from the cerebral cortex through signal acquisition equipment, or the level of neural activity in the brain can be measured through other methods such as spectroscopy. The collected brain activity detection signals (e.g., EEG signals) can be converted into signals that can be recognized by the computer through amplification, filtering, A / D conversion, and other processing. Then, the computer or digital signal processor performs preprocessing and feature extraction on the signal to obtain the characteristics of the signal, and then uses these characteristics to classify or pattern recognize the signal, thereby completing the decoding of the signal. Finally, the computer converts the decoded signal into specific instructions to control the external device, thereby achieving control of the external device.

[0033] Generally speaking, EEG signal processing often uses algorithms such as canonical correlation analysis, task-related component analysis, and neural networks, and runs on common integrated circuits such as central processing units (CPUs), graphics processing units (GPUs), and field programmable gate arrays (FPGAs). As Moore's Law approaches its limits, the improvement in computing power and energy efficiency of computing devices based on the von Neumann architecture is approaching its end. EEG signals require more efficient information processing platforms and processing algorithms compatible with these platforms.

[0034] Memristors are a new type of micro-nanoelectronic device whose resistance state can be adjusted by external voltage stimulation. Memristor-based neuromorphic computing breaks through the von Neumann architecture of traditional computing devices. Computation and storage are performed in the same location, reducing data transfer time and requiring high energy efficiency, low power consumption, and a small area for computation. A voltage pulse signal is input to one end of a memristor array. According to Kirchhoff's current law and Ohm's law, the output current at the other end of the array is the product of the input voltage vector and the conductance matrix. Thanks to these characteristics, memristor-based brain-computer interface signal processing has the potential to achieve low-power, high-performance brain-computer interface systems.

[0035] As previously mentioned, decoding EEG signals involves preprocessing, feature extraction, and classification. Each step is computationally intensive. Using a memristor array to perform these preprocessing, feature extraction, and classification calculations can improve computational efficiency. For example, a memristor array can be used to preprocess an EEG signal, resulting in a preprocessed signal that has been filtered out of noise and interference. The memristor array can then be used to perform feature extraction on this preprocessed signal, obtaining the signal's features. Finally, the memristor array can be used to perform classification or template matching on the extracted features, thereby classifying different tasks. Alternatively, the extracted features can be compared with a predefined task template to obtain the signal's decoding result.

[0036] However, the inventors of this disclosure have noted that due to the non-ideal characteristics of memristor devices themselves, memristor arrays accumulate significant errors after performing continuous pre-processing, feature extraction, and classification calculations, significantly reducing the computational accuracy of EEG signal processing and affecting the performance of the brain-computer interface system. Furthermore, multiple computations result in significant computational delays, and the use of multiple memristor arrays results in significant hardware area and energy consumption overhead, thus impacting the deployment of brain-computer interface systems in wearable or implantable scenarios.

[0037] At least one embodiment of the present disclosure provides a brain-computer interface signal processing method. The brain-computer interface signal processing method includes: receiving a brain activity detection signal from a target object; mapping a parameter matrix G into a storage and calculation integrated array, wherein a first matrix H for time-domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal, and a third matrix T for template matching of the brain activity detection signal are obtained according to a training model including multiple stimulation task templates; a parameter matrix G for one-step decoding processing of the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, where G=TWH; using the storage and calculation integrated array to which the parameter matrix G is mapped, one-step decoding processing is performed on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal; based on the decoding result, determining a control instruction for a target stimulation task corresponding to the brain activity detection signal from multiple stimulation task templates; and sending the control instruction to execute the task.

[0038] The brain-computer interface signal processing method provided in the above-mentioned embodiment of the present disclosure can combine the calculations in the three processing steps into one step, and realize one-step decoding of brain activity detection signals through a storage and computing integrated array, thereby reducing error accumulation, improving the calculation accuracy of EEG signal processing, reducing the algorithm's calculation amount and calculation delay, and also reducing the hardware area and energy consumption overhead.

[0039] At least one embodiment of the present disclosure further provides a brain-computer interface system. The brain-computer interface system includes a signal receiving device, a processing device, and an output device, wherein the signal receiving device is configured to receive a brain activity detection signal from a target subject; the processing device includes a storage and computing integrated array and is configured to: map a parameter matrix G into the storage and computing integrated array, wherein a first matrix H for time-domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal, and a third matrix T for template matching of the brain activity detection signal are obtained based on a training model including multiple stimulation task templates; a parameter matrix G for one-step decoding processing of the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, where G=TWH; the storage and computing integrated array to which the parameter matrix G is mapped is used to perform one-step decoding processing on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal; based on the decoding result, a control instruction for a target stimulation task corresponding to the brain activity detection signal is determined from the multiple stimulation task templates; and the output device is configured to send the control instruction to execute the task.

[0040] The brain-computer interface system provided by at least one embodiment of the present disclosure has the advantages of low energy consumption and low overhead, and can operate more flexibly in wearable and implantable scenarios.

[0041] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0042] Figure 1A is a schematic diagram of the structure of an exemplary integrated storage and computing array provided in at least one embodiment of the present disclosure. As shown in Figure 1A, the integrated storage and computing array includes multiple integrated storage and computing units and multiple word lines, bit lines, and source lines. The multiple integrated storage and computing units form an array with M rows and N columns, where M and N are both positive integers. Each integrated storage and computing unit typically includes at least one switching element and at least one integrated storage and computing device.

[0043] In Figure 1A, WL <1> 、WL <2> ...WL <m>Represent the word lines of the first row, the second row, ... the Mth row, respectively. The control end (e.g., the gate of the transistor) of the switch element in the storage-computing integrated unit in each row is connected to the word line corresponding to the row; BL <1> BL <2> ...BL <n>Represent the bit lines of the first column, the second column, ... the Nth column respectively. The storage-computing integrated device in each column of the storage-computing integrated unit is connected to the bit line corresponding to the column; SL <1> , SL <2> ...SL <m>They represent the source lines of the first row, the second row, ... the Mth row respectively, and one end of the switching element in the storage and computing integrated unit of each row (such as the source of the transistor) is connected to the source line corresponding to the row.

[0044] The embodiments of the present disclosure have no restrictions on the type, structure, etc. of the storage and computing integrated unit; the switching element can be a transistor or other switching device with the same characteristics. For example, the transistor used here can be a thin film transistor or a field effect transistor (such as a MOS field effect transistor). The source and drain of the transistor can be symmetrical in structure, so its source and drain can be structurally indistinguishable. The embodiments of the present disclosure do not limit the type of transistor used.

[0045] In the embodiments of the present disclosure, the storage and computing integrated devices in the storage and computing integrated unit include but are not limited to memristors (for example, the memristors may be resistive random access memory (RRAM)), static random access memory (SRAM), dynamic random access memory (DRAM), phase change memory (PCM), flash memory (Flash), magnetic random access memory (MRAM) or other appropriate non-volatile storage devices.

[0046] The storage and computing unit can be used as both a storage unit and a computing unit. For example, when the storage and computing unit is used as a storage unit, different resistance values ​​(or conductance values, etc.) of the storage and computing device can be used to store different data information; when the storage and computing unit is used as a computing unit, the conductance value of the storage and computing device can be used as a number (such as a multiplier) in the multiplication calculation. The storage and computing array applies an input voltage to the storage and computing device, so that the input voltage and the current generated by the storage and computing device are accumulated on the column line to complete the multiplication and accumulation calculation. For example, the multiplication operation is completed based on Ohm's law, and the addition operation is completed based on Kirchhoff's current law.

[0047] For example, in an embodiment of the present disclosure, operations on the storage-computation-in-one array include read operations, program operations, and computing operations.

[0048] The read operation can obtain the value stored in each storage and computing device in the storage and computing array, for example, 0 or 1 or other values. The storage and computing device generally includes an upper electrode, a lower electrode, and a dielectric layer located between the upper electrode and the lower electrode. When the amplitude of the input voltage applied between the upper electrode and the lower electrode of the storage and computing device is less than the threshold voltage of the storage and computing device, the resistance value of the storage and computing device will not be changed. In this case, the current storage value of the storage and computing device can be read by applying a read voltage to the storage and computing device. The operation of applying a read voltage to the storage and computing device can be called a read operation.

[0049] The programming operation can change the resistance value of each storage and computing device in the storage and computing integrated array, and then change the value stored in each storage and computing integrated device. When the amplitude of the input voltage applied between the upper electrode and the lower electrode of the storage and computing integrated device is greater than the threshold voltage of the storage and computing integrated device, the resistance value of the storage and computing integrated device can be changed. For example, the resistance value of the storage and computing integrated device can be reduced or increased according to the set (Set) voltage or reset (Reset) voltage applied between the upper electrode and the lower electrode of the storage and computing integrated device, so that the storage and computing integrated device is in a different resistance state. For example, when the storage and computing integrated device is in a low resistance state, its corresponding storage value is 0; when the storage and computing integrated device is in a high resistance state, its corresponding storage value is 1. For example, the set voltage can be a positive voltage pulse, and the reset voltage can be a negative voltage pulse. In an embodiment of the present disclosure, the operation of applying a set voltage or a reset voltage to the storage and computing integrated device can be referred to as a programming operation (or write operation). After the storage and computing integrated device is programmed, the storage and computing integrated device can also be verified to ensure that the resistance value of the storage and computing integrated device reaches the expected resistance value.

[0050] The storage-computation-in-one unit in the storage-computation-in-one array of FIG1A may have, for example, a 1T1R structure or a 2T2R structure, wherein the storage-computation-in-one unit of the 1T1R structure includes one transistor and one storage-computation-in-one device, and the storage-computation-in-one unit of the 2T2R structure includes two transistors and two storage-computation-in-one devices. It should be noted that the embodiments of the present disclosure do not limit the structure of the storage-computation-in-one unit, and storage-computation-in-one units of other structural forms that can implement multiplication and accumulation operations may also be adopted (for example, including but not limited to a 2T1R structure (two transistors and one storage-computation-in-one device)).

[0051] FIG1B is a schematic diagram of a storage-computing-in-one unit of a 1T1R structure provided by at least one embodiment of the present disclosure. As shown in FIG1B , the storage-computing-in-one unit includes a transistor M0 and a storage-computing-in-one device R0. For example, when the transistor M0 is an N-type transistor, its gate is connected to the word line terminal WL, for example, when the word line terminal WL is input with a high level, the transistor M0 is turned on; the first electrode of the transistor M0 can be a source electrode and configured to be connected to the source line terminal SL, for example, the transistor M0 can receive a reset voltage through the source line terminal SL; the second electrode of the transistor M0 can be a drain electrode and configured to be connected to the second electrode (for example, the negative electrode) of the storage-computing-in-one device R0, and the first electrode (for example, the positive electrode) of the storage-computing-in-one device R0 is connected to the bit line terminal BL, for example, the storage-computing-in-one device R0 can receive a set voltage through the bit line terminal BL. For example, when the transistor M0 is a P-type transistor, its gate is connected to the word line terminal WL, for example, when the word line terminal WL is input with a low level, the transistor M0 is turned on; the first electrode of the transistor M0 can be a drain and configured to be connected to the source line SL, for example, the transistor M0 can receive a reset voltage through the source line SL; the second electrode of the transistor M0 can be a source and configured to be connected to the second electrode (for example, the negative electrode) of the storage-computing integrated device R0, and the first electrode (for example, the positive electrode) of the storage-computing integrated device R0 is connected to the bit line BL, for example, the storage-computing integrated device R0 can receive a set voltage through the bit line BL. It should be noted that the structure of the storage-computing integrated unit can also be implemented as other structures, such as a structure in which the second electrode of the storage-computing integrated device R0 is connected to the source line terminal SL, etc., and the embodiments of the present disclosure are not limited to this.

[0052] The function of the word line terminal WL is to apply a corresponding voltage to the gate of the transistor M0, thereby controlling the transistor M0 to be turned on or off. When performing a write operation on the storage-computing integrated device R0, such as a set operation or a reset operation, it is necessary to turn on the transistor M0 first, that is, it is necessary to apply a turn-on voltage to the gate of the transistor M0 through the word line terminal WL. After the transistor M0 is turned on, the resistance state of the storage-computing integrated device R0 can be changed by applying a voltage to the storage-computing integrated device R0 at the source line terminal SL and the bit line terminal BL. For example, a set voltage can be applied through the bit line terminal BL to put the storage-computing integrated device R0 in a low-resistance state; for another example, a reset voltage can be applied through the source line terminal SL to put the storage-computing integrated device R0 in a high-resistance state. For example, the resistance value of the high-resistance state can be more than ten times the resistance value of the low-resistance state, such as more than one hundred times or more than one thousand times.

[0053] Figure 1C is a schematic diagram of a 2T2R integrated storage and computing unit according to at least one embodiment of the present disclosure. As shown in Figure 1C, the 2T2R integrated storage and computing unit includes two transistors M1 and M2 and two integrated storage and computing devices R1 and R2. The following description of this embodiment uses the example of transistors M1 and M2 both being N-type transistors.

[0054] The gate of the transistor M1 is connected to the word line terminal WL1. For example, when a high level is input to the word line terminal WL1 of M1, the transistor M1 is turned on. The gate of the transistor M2 is connected to the word line terminal WL2. For example, when a high level is input to the word line terminal WL2 of M2, the transistor M2 is turned on. The first electrode of the transistor M1 can be a source and is configured to be connected to the source line terminal SL1. For example, the transistor M1 can receive a reset voltage through the source line terminal SL1. The first electrode of the transistor M2 can be a source and is configured to be connected to the source line terminal SL1. For example, the transistor M2 can receive a reset voltage through the source line terminal SL1. The first electrode of the transistor M1 is connected to the first electrode of the transistor M2, and they are connected together to the source line terminal SL1. The second electrode of the transistor M1 can be a drain and is configured to be connected to the second electrode (e.g., the negative electrode) of the storage-computing integrated device R1, and the first electrode (e.g., the positive electrode) of the storage-computing integrated device R1 is connected to the bit line terminal BL1. For example, the storage-computing integrated device R1 can receive a set voltage through the bit line terminal BL1. The second electrode of the transistor M2 can be a drain and is configured to be connected to the second electrode (e.g., the negative electrode) of the storage-computing integrated device R2, and the first electrode (e.g., the positive electrode) of the storage-computing integrated device R2 is connected to the bit line terminal BL2. For example, the storage-computing integrated device R2 can receive a set voltage through the bit line terminal BL2. It should be noted that the transistors M1 and M2 in the storage-computing integrated unit of the 2T2R structure can also be P-type transistors, which will not be repeated here.

[0055] Computational operations refer to matrix multiplication and addition calculations performed using a memory-computation device. For example, after writing the values ​​of the parameter elements in the parameter matrix to the memory-computation array (for example, after mapping the weight matrix of a neural network to the memory-computation array), multiple input voltages less than the threshold voltage can be input to multiple bit lines of the memory-computation array. The currents generated by the interaction of the multiple input voltages with the multiple memory-computation devices in a row or column are accumulated to obtain the output current.

[0056] For example, analog signals are input to the multiple column signal input terminals of the integrated storage and computing array after the conductance setting (programming) is completed. For example, the analog signals can be voltage signals, so that the integrated storage and computing array can be controlled to perform matrix multiplication operations. According to Kirchhoff's law, the output current of the integrated storage and computing array can be obtained according to the following formula (1):

[0057] In formula (1), V n Indicates the voltage input to the nth column signal input terminal among multiple column signal input terminals, I m Represents the current signal output by the mth row signal output terminal among multiple row signal output terminals. mn = represents the overall conductance of the memory-computation unit located in the mth row and nth column. According to Kirchhoff's law, the memory-computation array can complete multiplication and accumulation calculations in parallel.

[0058] For example, to implement the calculation of multiplying the input vector V by the matrix G to obtain the corresponding vector I, the value of each element G11 to Gmn in the matrix G can be mapped to the conductance value of each memory-computation-integrated device in the memory-computation-integrated array. For example, the conductance values ​​of the memory-computation-integrated devices in the m×n memory-computation-integrated units of the memory-computation-integrated array in FIG1A are respectively programmed to correspond to each element G11 to Gmn in the matrix G; the value of each element V1, V2, ..., Vn of the input vector V is mapped to multiple input voltages, and applied to each bit line BL of the memory-computation-integrated array accordingly. <1> BL <2> ...BL <n>On each word line WL <1> 、WL <2> ...WL <m>After applying the turn-on voltage to each transistor corresponding to the row, each source line SL <1> , SL <2> ...SL <m>The output current value of is the value of the corresponding element I1, I2...Im in the vector I. For example, the source line SL <1> The output current value is determined by n bit lines BL <1> BL <2> ...BL <n>The input voltage values ​​V1, V2, ... Vn applied to the source line SL are multiplied by <1> The conductance values ​​G11, G12...G1n of the n storage and computing devices on the source line SL are obtained by accumulating the current values ​​of each. <1> The output current value is the value of element I1 in vector I, so the result of matrix-vector multiplication can be obtained by measuring the output current values ​​of all rows.

[0059] It should be noted that, in the embodiment of the present disclosure, the directions of the "rows" and "columns" of the storage and computing integrated array are not limited to the situation in FIG1A , but can be set as needed.

[0060] For example, a storage-computing integrated array can realize positive elements, negative elements, or zero by combining two storage-computing integrated devices into a computing unit. Figure 1D is a schematic diagram of a storage-computing integrated array capable of realizing positive and negative values, according to at least one embodiment of the disclosure.

[0061] As shown in FIG1D , the storage-computing integrated array includes a positive weight column and a negative weight column. For example, the storage-computing integrated device 801 is located on the positive weight column, and the storage-computing integrated device 802 is located on the negative weight column. The conductance value of the storage-computing integrated device 801 is expressed as G 11 The conductance value of the storage and computing integrated device 802 is expressed as G 12 When the storage-computing device 801 receives a positive input voltage signal v(t), the storage-computing device 802 also receives a positive input voltage signal v(t). The storage-computing device 801 and the storage-computing device 802 are connected to different SLs, and the output current passing through the storage-computing device 801 and the output current passing through the storage-computing device 802 are subtracted at the end of the SL. Therefore, the result of the multiplication and accumulation calculation of the storage-computing device 801 and the storage-computing device 802 is v(t)G 11 -v(t)G 12 , that is, v(t)(G 11 -G 12 ). Therefore, the computing unit including the storage and computing integrated device 801 and the storage and computing integrated device 802 can correspond to a weight element, and the weight element is G 11 -G 12 , by configuring G 11 -G 12 The numerical relationship can realize positive, zero, and negative elements. For example, the storage and calculation integrated unit of the 2T2R structure shown in Figure 1C can also be used to realize positive elements, negative elements, or zero.

[0062] FIG2 is a flow chart of a brain-computer interface signal processing method provided by at least one embodiment of the present disclosure. As shown in FIG2 , the brain-computer interface signal processing method includes the following steps S100 to S500.

[0063] Step S100: receiving a brain activity detection signal from a target subject.

[0064] Step S200: Map the parameter matrix G into a storage and computing integrated array, wherein a first matrix H for time-domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal, and a third matrix T for template matching of the brain activity detection signal are obtained according to a training model including multiple stimulation task templates. The parameter matrix G for one-step decoding processing of the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, where G=TWH.

[0065] Step S300: using the storage-computation-in-one array of the mapping parameter matrix G to perform a one-step decoding process on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal.

[0066] Step S400: Determine, based on the decoding result, a control instruction of a target stimulation task corresponding to the brain activity detection signal from a plurality of stimulation task templates.

[0067] Step S500: Send a control instruction to execute the task.

[0068] In at least one embodiment of the present disclosure, the brain activity detection signal received in step S100 may be a brain activity detection signal obtained by real-time acquisition of the brain of a human or animal as a target object, or may be a non-real-time brain activity detection signal obtained from a signal storage device, or may be a remote brain activity detection signal obtained from a signal transmission device. The embodiments of the present disclosure do not limit the method for obtaining the brain activity detection signal.

[0069] In at least one embodiment of the present disclosure, a brain activity detection signal refers to a signal obtained by detecting brain activity of a human or animal, including but not limited to a steady-state visual evoked potential (SSVEP) or a functional near-infrared spectroscopy (fNIRS). For example, steady-state visual evoked potential (SSVEP) is an electroencephalogram (EEG) signal that is often induced by a visual stimulus block with a certain frequency. In areas such as the occipital region of the brain, EEG signals with the same frequency as the input stimulus and a multiple of the frequency can be observed. SSVEP has become a commonly used brain-computer interface control signal because of its obvious characteristics and high information transmission rate. For example, functional near-infrared spectroscopy (fNIRS) is a signal that detects changes in the concentration of, for example, absorbing chromophores in tissues or measures changes in blood oxygen content and blood flow in the brain through infrared spectroscopy, and can be used to monitor brain activity. The embodiments of the present disclosure do not limit the type of brain activity detection signal.

[0070] In at least one embodiment of the present disclosure, the brain activity detection signal is an analog signal, such as a current signal or a voltage signal. The brain activity detection signal can be a continuous analog signal or a discrete analog signal. For example, a neural probe can be placed in contact with the brain to collect continuous or discrete EEG signals.

[0071] For example, the brain-computer interface signal processing method provided in at least one embodiment of the present disclosure further includes: obtaining the number of signal channels Nc and the number of sampling points Ns of the brain activity detection signal.

[0072] The number of signal channels Nc and the number of sampling points Ns are important parameters used to describe the acquisition and processing of brain activity detection signals (e.g., EEG signals). For example, the number of signal channels Nc can refer to the number of electrodes used in an EEG, that is, the number of channels used to collect EEG signals. Each electrode can be considered an independent channel, and multiple electrodes can be placed on the scalp to record electrical activity in different brain regions. The size of the number of signal channels Nc depends on the number of electrodes used in the EEG acquisition system and can range from a few to hundreds. For example, the number of sampling points (or time points) Ns refers to the number of times or data points that the brain activity detection signal is sampled over a period of time, indicating the number of samples of the brain activity detection signal obtained within a specific time period. The number of sampling points determines the temporal resolution of the signal. For example, the size of the number of sampling points Ns can be the product of the sampling rate (i.e., the number of samples per second) and the sampling time.

[0073] In the field of brain-computer interface technology, a variety of algorithms can be used to process brain activity detection signals, including but not limited to task-related component analysis (TRCA) and ensemble TRCA (eTRCA). TRCA and eTRCA algorithms generally include three major processing steps: preprocessing calculations for time-domain signal filtering, feature extraction calculations for spatial filtering, and classification calculations similar to template matching.

[0074] In step S200, a first matrix H for performing time-domain filtering on brain activity detection signals, a second matrix W for performing spatial filtering on brain activity detection signals, and a third matrix T for performing template matching on brain activity detection signals can be obtained based on a training model including multiple stimulation task templates. The training model here can be a model trained using an integrated task-related component analysis algorithm (eTRCA) or a model trained using a task-related component analysis algorithm (TRCA).

[0075] For example, under normal circumstances, it is necessary to first use a large amount of user data to perform model training on the preset stimulation tasks, and then use the eTRCA algorithm to decode the EEG signals. For example, when performing model training, multiple stimulation tasks can be preset first, and the multiple stimulation tasks include but are not limited to visual stimulation tasks, auditory stimulation tasks, and action stimulation tasks. For example, the multiple stimulation tasks can be that the target object watches flashing stimulation blocks with different frequencies or watches different types of images, or that the target object hears sounds of different frequencies or recognizes or understands specific voices, or that the target object imagines or performs different specific actions (for example, imagine clicking fingers or pushing a wheelchair). The embodiments of the present disclosure do not limit the types of multiple stimulation tasks.

[0076] By training a large amount of user data in advance, a training model including multiple stimulation task templates can be obtained. After obtaining the training model, the brain activity detection signal that actually needs to be processed can be input into the training model, and then the decoding result of the brain activity detection signal can be obtained through a series of calculations.

[0077] For example, a current decoding scheme based on the eTRCA algorithm is to first obtain the Template and integrated spatial filtering U e The training model, Among them, u k N represents the spatial filter, which is trained by maximizing the covariance of inter-class signals from the k-th stimulus data. f Represents the number of stimulus categories, that is, the number of multiple stimulus tasks preset in the training model. Then, the Template and integrated spatial filtering U e Apply to the actual test signal to decode the test signal. For example, for a pre-processed signal x′, you can compare x′U e and χ k U e The correlation between them can be used to identify specific task-related components. The correlation coefficient r(k) between the two can be calculated by formula (2): r(k)=ρ(x′U e , χ k U e ) (2)

[0078] Here, ρ(a, b) represents the Pearson correlation coefficient between a and b.

[0079] It can be seen from this that in actual calculation, it is usually necessary to first perform a preprocessing calculation of the test signal by time domain signal filtering to obtain the signal x′, and then perform a feature extraction calculation of spatial filtering on the preprocessed signal x′ to obtain x′U e , and finally x′U e Perform template matching calculations to obtain the correlation coefficient r(k). After that, the template that matches the test signal can be determined based on the size of the correlation coefficient.

[0080] In at least one embodiment of the present disclosure, step S200 can obtain the coefficient matrices for the three steps of preprocessing calculation for time domain signal filtering of the test signal, feature extraction calculation for spatial filtering, and classification calculation similar to template matching based on formula (2), that is, the first matrix H, the second matrix W, and the third matrix T. Then, the parameter matrix G for performing one-step decoding processing on the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, where G=TWH. Next, the parameter matrix G is mapped to the storage and computing integrated array, and the storage and computing integrated array is used to perform one-step decoding processing on the brain activity detection signal, so as to obtain the decoding result corresponding to the brain activity detection signal directly from the output end of the storage and computing integrated array.

[0081] That is to say, the brain-computer interface signal processing method provided by at least one embodiment of the present disclosure can combine the calculations in the three processing steps of time domain filtering, spatial filtering and template matching into one-step calculation. For example, the coefficient matrices in the three processing steps of time domain filtering, spatial filtering and template matching are fused into a parameter matrix for one-step calculation, and the fused parameter matrix is ​​mapped to the storage and computing integrated array for calculation, so that the storage and computing integrated array can decode the brain activity detection signal with only one calculation, avoiding the errors generated by multiple calculations through the storage and computing integrated array, improving the calculation accuracy, and greatly reducing the algorithm's computational complexity and calculation delay, reducing the hardware area and energy consumption overhead.

[0082] For example, in at least one example of the embodiments of the present disclosure, the brain-computer interface signal processing method also includes: determining the size of the test data matrix as P×Q based on the number of signal channels Nc and the number of sampling points Ns of the brain activity detection signal, where P=Nc×Ns, Q=1.

[0083] For example, it is necessary to first determine the size of the original input signal, that is, to determine the number of signal channels Nc and the number of sampling points Ns for the brain activity detection signal. Then, based on the number of signal channels Nc and the number of sampling points Ns, the size of the test data matrix is ​​determined, and the number of signal input terminals required for the integrated storage and computing array is determined. Alternatively, the number of signal channels Nc and the number of sampling points Ns for the brain activity detection signal can be determined based on the scale of the integrated storage and computing array, although this is not a limitation in the embodiments of the present disclosure.

[0084] For example, in at least one example of the embodiments of the present disclosure, the brain-computer interface signal processing method also includes: determining the number of stimulation categories Nf based on the number of multiple stimulation task templates, and determining the number of storage and computing units used for calculation in the storage and computing array as Nc×Ns×Nf based on the number of stimulation categories Nf, the number of signal channels Nc and the number of sampling points Ns.

[0085] For example, if the size of the storage and computing integrated array is A rows and B columns, then A should be greater than Nc×Ns, B should be greater than Nf, or B should be greater than Nc×Ns, and A should be greater than Nf.

[0086] For example, in one example, the number of signal channels Nc is 8, the number of sampling points Ns is 100, and the number of stimulus categories Nf is 12. Then the test data matrix should be 800 rows and 1 column (or, a column vector of size 800), and the storage-computation integrated array should include at least 800 rows and 12 columns of computing units (such as memristor units), or at least 12 rows and 800 columns of computing units, with at least 800 signal input terminals, and the number of storage-computation integrated units used for calculation is at least 8×100×12. For example, in one example, if it is necessary to realize positive values, negative values, or zero, a storage-computation integrated unit with a 2T2R structure can be used, and the storage-computation integrated array can include 800 rows and 24 columns of computing units, or 24 rows and 800 columns of computing units.

[0087] For example, in at least one example of the embodiments of the present disclosure, a first matrix H for time domain filtering of brain activity detection signals, a second matrix W for spatial filtering of brain activity detection signals, and a third matrix T for template matching of brain activity detection signals are obtained based on a training model including multiple stimulation task templates. A specific example may include: obtaining a time domain filter response function for time domain filtering of brain activity detection signals, a spatial domain filter matrix for spatial filtering of brain activity detection signals, and multiple stimulation task templates; obtaining a first matrix H, a second matrix W, and a third matrix T with mutually matching dimensions based on a test data matrix and the time domain filter response function, the spatial domain filter matrix, and multiple stimulation task templates.

[0088] The following describes in detail how to obtain the first matrix H, the second matrix W, and the third matrix T.

[0089] In the traditional first step of time domain filtering, the multi-channel EEG signal x k (k=1,2,…,N c ) can be mathematically described as the original calculation formula (3):

[0090] Among them, N c is the number of signal channels, y k (k=1,2,…,N c ) is the filtered signal. J is the order of the filter, and h represents the filter response function.

[0091] Formula (3) can be rewritten as formula (4) to obtain the first matrix H: y=Hx (4) x k =[x k (1) x k (2)…x k (N s )](k=1,2,…,N c ) (6) y k =[y k (1) y k (2)…y k (N s )](k=1,2,…,N c ) (8)

[0092] Where x is the test data matrix, Ns is the number of sampling points per channel, and Nc is the number of signal channels.

[0093] For example, the first matrix H can be obtained according to the time domain filter response function for performing time domain filtering on the brain activity detection signal. The first matrix H can be expressed as formula (9): H = diag(F, F, ..., F) (9) h=[h(J) h(J-1)…h(0)] (13)

[0094] Among them, O m×n represents a zero matrix with m rows and n columns. In formula (9), the number of matrices F is N c .

[0095] FIG3 is a schematic diagram of an exemplary time-domain filter provided by at least one embodiment of the present disclosure. As shown in FIG3 , an exemplary time-domain filter has an order J = 48 (49 filter taps are shown in the figure), so the time-domain filter response function h = [h(48) h(47)…h(0)].

[0096] In the traditional second-step spatial filtering process, the original calculation can be expressed as formula (4): Z = W0Y (14)

[0097] Among them, W0 is the spatial filter matrix, w k (k=1,2,…,N f ) is the filter for the k-th signal, z k (k=1,2,…,N f ) represents the signal y k (k=1,2,…,N c ) after spatial filtering.

[0098] Formula (14) can be rewritten as formula (19) to obtain the second matrix W: z = Wy (19) z k =[z k (1) z k (2)…z k (N s )](k=1,2,…,N f ) (twenty one)

[0099] Wherein, z represents the signal after spatial domain filtering of signal y.

[0100] It should be noted that the matrix dimensions of z in formula (20) and Z in formula (15) are different.

[0101] FIG4 is a schematic diagram of an exemplary spatial filter matrix provided by at least one embodiment of the present disclosure. As shown in FIG4 , the number of stimulus categories Nf is 12, that is, the number of stimulus task templates and the number of instructions corresponding to the stimulus task templates is 12. For example, the number of signal channels Nc shown in FIG4 is 8.

[0102] For example, the second matrix W can be obtained according to the spatial domain filter matrix used to perform spatial filtering on the brain activity detection signal. The second matrix W can be expressed as formula (22): D p，q =diag([w p，q , w p，q ,…,w p，q ]) (twenty three)

[0103] Where (p = 1, 2, ..., N f ;q=1,2,…,N c ), w p，q The number is N s .

[0104] In the traditional third step template matching process, the original calculation of the correlation coefficient r can be expressed as formula (24):

[0105] Among them, t0, k and t k Represent the template of the k-th signal and the normalized template, which have the same size as z, r k is z and t k The correlation coefficient between .

[0106] For example, in at least one example of the embodiments of the present disclosure, the brain-computer interface signal processing method also includes: in response to time domain filtering and spatial filtering of the brain activity detection signal, approximating multiple parameters in the third matrix T as multiple templates after normalizing multiple stimulation task templates.

[0107] As shown in formula (26), since z and t k is the filtered signal, so the mean vector and is approximately equal to the zero vector, as shown in formula (25), since the numerator It can be approximated as z·t k , and r k The denominators are the same, so r k With z·t k Directly proportional.

[0108] Formula (24) can be rewritten as formula (27) to obtain the third matrix T: s = Tz (27)

[0109] Where s can be expressed as formula (28):

[0110] For example, a third matrix T can be obtained according to multiple stimulation task templates. The third matrix T can be expressed as formula (29):

[0111] FIG5 is a schematic diagram of an exemplary stimulation task template provided by at least one embodiment of the present disclosure. As shown in FIG5 , the template signals t1 t2…t 12 There are 12 instructions respectively, and the template signal of each instruction has the same scale as the brain activity detection signal. For example, the template signal has 8 channels and 100 time points.

[0112] The above introduces an exemplary method for obtaining the first matrix H, the second matrix W, and the third matrix T. However, this is not a limitation of the embodiments of the present disclosure. Those skilled in the art may obtain the first matrix H, the second matrix W, and the third matrix T through other methods.

[0113] Finally, the parameter matrix G for one-step decoding of the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, as shown in formula (30): G = TWH (30)

[0114] That is, the above three steps can be finally integrated into formula (31): s=Gx (31)

[0115] Then, in step S200, the parameter matrix G is mapped to the storage-computation integrated array. For example, s, G, and x in formula (31) are substituted into formula (1), that is, the parameter matrix G is mapped to the conductance values ​​of multiple storage-computation integrated units in the storage-computation integrated array, and x is used as the input voltage signal of multiple signal input terminals (that is, brain activity detection signal). The specific instructions for mapping the parameter matrix G to the storage-computation integrated array can be referred to the programming operation above, and will not be repeated here.

[0116] After mapping the parameter matrix G to the integrated storage and computing array, step S300 may be executed: using the integrated storage and computing array to which the parameter matrix G is mapped, a one-step decoding process is performed on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal.

[0117] For example, in at least one example of the embodiments of the present disclosure, a specific example of step S300 may include: inputting the brain activity detection signal into multiple signal input terminals of the storage and computing integrated array of the mapping parameter matrix G; controlling the storage and computing integrated array of the mapping parameter matrix G to perform multiplication and accumulation calculations on the brain activity detection signal, so as to obtain multiple analog output signals after the multiplication and accumulation calculations are performed from multiple signal output terminals of the storage and computing integrated array; receiving multiple digital output signals after analog-to-digital conversion of the multiple analog output signals, and obtaining decoding results corresponding to the brain activity detection signal.

[0118] FIG6 is a schematic diagram of mapping a parameter matrix for one-step decoding processing to mapping values ​​on a storage-computation integrated array provided by at least one embodiment of the present disclosure.

[0119] As shown in Figure 6, in an example, the number of signal channels Nc is 8, the number of sampling points Ns is 100, the number of stimulus categories Nf is 12, and a 2T2R structured storage and computing unit is used (each storage and computing unit includes two storage and computing devices). The size of the calculated parameter matrix G is 24×800, and the storage and computing array needs to be configured to use 24 rows and 800 columns of storage and computing units for calculation. The number of input voltage signals input to the storage and computing array is 100×8.

[0120] For example, the number of rows of the memory-computation integrated array in FIG1A is M=24, the number of columns is N=800, and each element G11~Gmn in the parameter matrix G has been mapped to the conductance value of each memory-computation integrated device of the memory-computation integrated array; the brain activity detection signals x1, x2...xn are applied to each bit line BL of the memory-computation integrated array. <1> BL <2> ...BL <n>On each word line WL <1> 、WL <2> ...WL <m>After applying the turn-on voltage to each transistor corresponding to the row, according to Ohm's law and Kirchhoff's current law, each source line SL <1> , SL <2> ...SL <m>Output multiple analog output signals, after analog-to-digital conversion, the multiple analog output signals are converted into multiple digital output signals, that is, corresponding to the decoding results of the brain activity detection signal.

[0121] Then, in step S400, based on the decoding results, control instructions for the target stimulation task corresponding to the brain activity detection signal are determined from the multiple stimulation task templates. For example, a threshold value can be set to select a stimulation task that best matches the decoding results from the multiple stimulation task templates as the target stimulation task, or the target stimulation task corresponding to the brain activity detection signal can be determined by other methods.

[0122] For example, the target stimulation task l corresponding to the brain activity detection signal can be determined from multiple stimulation task templates using formula (32): l = argmax(s) (32)

[0123] Finally, in step S500, a control instruction is sent to execute the task.

[0124] In an embodiment of the present disclosure, a control instruction may be sent to an external device so that the external device responds to the instruction and performs a related task, such as performing a display, moving a cursor, operating a robotic arm, etc.

[0125] The embodiments of the present disclosure can determine the parameter matrix G according to the above method, and configure the storage and computing integrated array according to the parameter matrix G. In actual application, the configured storage and computing integrated array can be directly used to perform one-step decoding processing on the detected EEG signal. Compared with the original decoding processing, it saves multiple programming operations, multiple verification operations (that is, multiple mapping processes) and multiple calculation operations on the storage and computing integrated array in multiple calculation steps in each application, reduces error accumulation, improves calculation accuracy, and reduces calculation amount and calculation delay.

[0126] Figure 7 is a schematic diagram comparing the computational complexity of a raw decoding process and a one-step decoding process provided by at least one embodiment of the present disclosure. As shown in Figure 7, the brain-computer interface signal processing method provided by at least one embodiment of the present disclosure can significantly reduce the computational complexity by utilizing a storage and computing array to perform the above-mentioned one-step decoding process on brain activity detection signals.

[0127] Furthermore, the embodiments of the present disclosure can utilize a storage and computing integrated array to directly process brain activity detection signals, without the need to first convert the brain activity detection signals into digital signals and then process the digital signals, thereby reducing the number of conversions between analog signals and digital signals during the decoding of brain activity detection signals, reducing the use of analog-to-digital converters and other hardware resources, and thereby reducing the power consumption of the circuit.

[0128] At least one embodiment of the present disclosure further provides a brain-computer interface system. Figure 8 is a schematic block diagram of a brain-computer interface system provided by at least one embodiment of the present disclosure. As shown in Figure 8 , the brain-computer interface system includes a signal receiving device 810 , a processing device 820 , and an output device 830 .

[0129] For example, the signal receiving device 810 is configured to receive a brain activity detection signal from a target subject.

[0130] For example, the processing device 820 includes a storage and computing integrated array 821, and is configured to: map the parameter matrix G into the storage and computing integrated array, wherein a first matrix H for time domain filtering of the brain activity detection signal, a second matrix W for spatial filtering of the brain activity detection signal, and a third matrix T for template matching of the brain activity detection signal are obtained according to a training model including multiple stimulation task templates; the parameter matrix G for one-step decoding processing of the brain activity detection signal is calculated according to the first matrix H, the second matrix W, and the third matrix T, G=TWH; use the storage and computing integrated array to which the parameter matrix G is mapped to perform one-step decoding processing on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal; and determine the control instructions of the target stimulation task corresponding to the brain activity detection signal from the multiple stimulation task templates according to the decoding result.

[0131] For example, the output device 830 is configured to send control instructions to perform a task.

[0132] The signal receiving device 810 is used to implement step S100 shown in Figure 2, the processing device 820 is used to implement steps S200 to S400 shown in Figure 2, and the output device 830 is used to implement step S500 shown in Figure 2. Therefore, the specific description of the signal receiving device 810, the processing device 820, and the output device 830 can refer to the relevant description of each step in the embodiment of the brain-computer interface signal processing method mentioned above.

[0133] For example, the brain-computer interface system may further include a signal acquisition device (not shown in FIG8 ) for acquiring detection signals of brain activity of the target object. The signal acquisition device may include EEG signal acquisition electrodes, a low-noise amplifier, a filter, an analog-to-digital converter, and the like. For example, the signal receiving device 810 may directly acquire EEG signals from the target object through the signal acquisition device, or may indirectly acquire EEG signals from the target object through a signal storage device. The embodiments of the present disclosure do not limit the method for acquiring brain activity detection signals. For example, the target object may be a brain, such as a human brain or an animal brain.

[0134] For example, as shown in FIG8 , the brain-computer interface system provided in at least one embodiment of the present disclosure further includes an effector 840. Effector 840 is coupled to output device 830 and is configured to be controlled by control instructions to perform corresponding operations. For example, effector 840 can be a robotic arm, a brain-controlled robot, etc., although the embodiments of the present disclosure are not limited thereto.

[0135] For example, the storage-computing integrated array may be a memristor array. For specific descriptions of the storage-computing integrated array, please refer to the relevant description above. The embodiments of the present disclosure do not limit the structure of the storage-computing integrated array and the structure and type of the storage-computing integrated unit.

[0136] The brain-computer interface system provided by at least one embodiment of the present disclosure has the advantages of low energy consumption and low overhead, and can operate more flexibly in wearable and implantable scenarios.

[0137] At least one embodiment of the present disclosure further provides an electronic device, comprising a memory and a processor, wherein the memory non-transiently stores computer-executable instructions, and the processor is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement a brain-computer interface signal processing method as in any of the above embodiments.

[0138] At least one embodiment of the present disclosure further provides a non-transitory computer-readable storage medium that can non-transitorily store one or more computer-executable instructions. For example, when the computer-executable instructions are executed by a processor, one or more steps of the brain-computer interface signal processing method described above can be performed.

[0139] For example, the non-transitory readable storage medium is implemented as a memory, such as a volatile memory and / or a non-volatile memory. In the above embodiments, the memory may be a volatile memory, such as a random access memory (RAM) and / or a cache. Non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, etc. The memory may also store various applications (code, instructions) and data, as well as various data used and / or generated by the applications.

[0140] Some embodiments of the present disclosure further provide an electronic device comprising the brain-computer interface system of any of the above embodiments or capable of executing the brain-computer interface signal processing method of any of the above embodiments.

[0141] FIG9 is a schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device 1000 shown in FIG9 is merely an example and should not limit the functionality or scope of use of the embodiments of the present disclosure.

[0142] For example, as shown in FIG9 , in some examples, the electronic device 1000 includes a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which may be the processing device 820 in FIG8 , which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the computer system are also stored in the RAM 1003. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0143] For example, the following components may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009, which may also include, for example, a network interface card such as a LAN card or modem. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or wired to exchange data, performing communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable storage medium 1011, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in the drive 1010 as needed, so that a computer program read therefrom can be installed into the storage device 1008 as needed. Although FIG. 9 shows the electronic device 1000 including various devices, it should be understood that it is not required to implement or include all of the devices shown, and more or fewer devices may be implemented or included instead.

[0144] For example, the electronic device 1000 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 1009 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communications may use any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0145] For example, the electronic device 1000 can be any device such as a mobile phone, tablet computer, laptop computer, e-book, game console, television, digital photo frame, navigator, etc., or it can be any combination of data processing devices and hardware, and the embodiments of the present disclosure are not limited to this.

[0146] Although the present disclosure has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications or improvements may be made based on the embodiments of the present disclosure. Therefore, such modifications or improvements, as long as they do not depart from the spirit of the present disclosure, are within the scope of protection claimed by the present disclosure.

[0147] Regarding this disclosure, the following points need to be explained:

[0148] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0149] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the thickness of layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale.

[0150] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0151] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.< / m> < / m> < / n> < / n> < / m> < / m> < / n> < / m> < / n> < / m>

Claims

1. A brain-computer interface signal processing method, comprising: receiving a brain activity detection signal from a target subject; Mapping a parameter matrix G into a storage-computation integrated array, wherein a first matrix H for performing time-domain filtering on the brain activity detection signal, a second matrix W for performing spatial filtering on the brain activity detection signal, and a third matrix T for performing template matching on the brain activity detection signal are obtained based on a training model including multiple stimulation task templates, and the parameter matrix G for performing one-step decoding processing on the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, where G=TWH; performing the one-step decoding process on the brain activity detection signal using the storage-computation-in-one array that maps the parameter matrix G to obtain a decoding result corresponding to the brain activity detection signal; determining, based on the decoding result, a control instruction of a target stimulation task corresponding to the brain activity detection signal from the plurality of stimulation task templates; and The control instruction is sent to execute the task.

2. The brain-computer interface signal processing method according to claim 1, wherein: Using the storage-computation-in-one array that maps the parameter matrix G to perform the one-step decoding process on the brain activity detection signal to obtain a decoding result corresponding to the brain activity detection signal, including: Inputting the brain activity detection signal into a plurality of signal input terminals of the storage and computing integrated array that maps the parameter matrix G; Controlling the integrated storage and computation array that maps the parameter matrix G to perform multiplication and accumulation calculations on the brain activity detection signals, so as to obtain a plurality of analog output signals after performing the multiplication and accumulation calculations from a plurality of signal output terminals of the integrated storage and computation array; A plurality of digital output signals are received after analog-to-digital conversion is performed on the plurality of analog output signals, and a decoding result corresponding to the brain activity detection signal is obtained.

3. The brain-computer interface signal processing method according to claim 1 or 2, further comprising: Obtaining the number of signal channels Nc and the number of sampling points Ns of the brain activity detection signal; Determine the number of stimulation categories Nf according to the number of the plurality of stimulation task templates; The number of storage and computing units used for calculation in the storage and computing array is determined to be Nc×Ns×Nf according to the number of stimulation categories Nf, the number of signal channels Nc and the number of sampling points Ns.

4. The brain-computer interface signal processing method according to claim 3, further comprising: The size of the test data matrix is determined to be P×Q according to the number of signal channels Nc and the number of sampling points Ns of the brain activity detection signal, where P=Nc×Ns and Q=1.

5. The brain-computer interface signal processing method according to any one of claims 1 to 4, wherein: Acquiring, according to a training model including a plurality of stimulation task templates, a first matrix H for performing time-domain filtering on the brain activity detection signal, a second matrix W for performing spatial filtering on the brain activity detection signal, and a third matrix T for performing template matching on the brain activity detection signal, including: Acquiring a time-domain filter response function for performing the time-domain filtering on the brain activity detection signal, a spatial-domain filter matrix for performing the spatial filtering on the brain activity detection signal, and the plurality of stimulation task templates; A first matrix H, a second matrix W, and a third matrix T with mutually matching dimensions are obtained according to the test data matrix and the time domain filter response function, the spatial domain filter matrix, and the multiple stimulation task templates.

6. The brain-computer interface signal processing method according to any one of claims 1 to 5, further comprising: In response to performing the time-domain filtering and the spatial filtering on the brain activity detection signal, a plurality of parameters in the third matrix T are approximated as a plurality of templates obtained by normalizing the plurality of stimulation task templates.

7. The brain-computer interface signal processing method according to any one of claims 1 to 6, wherein: The brain activity detection signal includes a steady-state visual evoked potential signal or a functional near-infrared spectroscopy signal.

8. The brain-computer interface signal processing method according to any one of claims 1 to 7, wherein: The training model includes a model trained according to an integrated task-related component analysis algorithm.

9. A brain-computer interface system, comprising a signal receiving device, a processing device and an output device, in, The signal receiving device is configured to receive a brain activity detection signal from a target subject; The processing device includes a storage and computing integrated array and is configured to: Mapping a parameter matrix G into a storage-computation integrated array, wherein a first matrix H for performing time-domain filtering on the brain activity detection signal, a second matrix W for performing spatial filtering on the brain activity detection signal, and a third matrix T for performing template matching on the brain activity detection signal are obtained based on a training model including multiple stimulation task templates, and the parameter matrix G for performing one-step decoding processing on the brain activity detection signal is calculated based on the first matrix H, the second matrix W, and the third matrix T, where G=TWH; performing the one-step decoding process on the brain activity detection signal using the storage-computation-in-one array that maps the parameter matrix G to obtain a decoding result corresponding to the brain activity detection signal; determining, based on the decoding result, a control instruction of a target stimulation task corresponding to the brain activity detection signal from the plurality of stimulation task templates; The output device is configured to send the control instruction to perform the task.

10. The brain-computer interface system according to claim 9, further comprising an effector, wherein: The effector is coupled to the output device and is configured to be controlled by the control instruction to perform a corresponding operation.

11. The brain-computer interface system according to claim 9 or 10, wherein: The storage-computing integrated array includes a memristor array.

Images