Storage and calculation integrated steady-state visual evoked electroencephalogram signal processing system

Abstract

The invention relates to the technical field of signal processing, in particular to a storage and calculation integrated steady-state visual evoked electroencephalogram signal processing system which comprises an electroencephalogram acquisition subsystem and a signal decoding subsystem. The electroencephalogram acquisition subsystem is used for preprocessing the received steady-state visual evoked electroencephalogram signals; the signal decoding subsystem comprises a storage and calculation integrated chip, the storage and calculation integrated chip runs a pre-trained zero calibration decoding algorithm based on combination of a convolutional neural network and a Transform model, carries out decoding operation on the preprocessed electroencephalogram signals, and inputs corresponding decoding results into an external control device. The processing algorithm performance can be improved, the system power consumption is reduced, and the system integration degree is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of signal processing, and more specifically, to a processing system for steady-state visual evoked potential electroencephalogram signals with in-memory computing. Background Art

[0002] With the development of brain-computer interface technology, non-invasive brain-computer interfaces have received attention and research due to their high safety and wide range. Among them, steady-state visual evoked potentials (SSVEP) signals have the advantages of high information transmission rate, good repeatability, and mature research. By decoding SSVEP signals, high-interaction and high-real-time device control have become the mainstream technical direction of non-invasive brain-computer interfaces.

[0003] Edge-side wearable deployment is the ultimate form of brain-computer interface devices in the fields of medical rehabilitation, virtual reality, etc. However, the combined effects of low decoding algorithm performance, high power consumption required by high-performance algorithms, and low integration of existing device forms have severely restricted its development. Existing SSVEP brain-computer interfaces adopt a trained decoding method based on correlation analysis. In the early stage of use, specific subjects need to perform specific tasks to provide a large amount of individualized data to form a corresponding labeled data set for training classification or regression models. On this basis, during use, the brain-computer interface analyzes the characteristics of the subject's electroencephalogram signals, optimizes the target using correlation analysis, and minimizes the prediction error to output the corresponding decoding result. This trained decoding scheme requires the collection of personal data from a large number of subjects, facing serious personal privacy and ethical issues. In addition, the correlation algorithm it relies on has poor performance and is difficult to correctly predict in scenarios such as across days and across devices, with low stability and it is difficult to form a mature industrial application solution.

[0004] In addition, the key application direction of brain-computer interface technology is edge-side wearable devices, and the zero-calibration decoding method has a large computing power requirement. Currently, the hardware system for SSVEP signal processing still uses the von Neumann architecture with separate memory and computing. During calculation, data and instructions are frequently transferred between the memory and the processor, resulting in serious power consumption overhead. Especially when processing deep learning algorithms with large computational requirements, the large power consumption limits the implementation of related technologies in edge-side devices.

[0005] Therefore, how to improve the performance of processing algorithms, reduce system power consumption, and enhance system integration is an urgent problem for those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a processing system for steady-state visual evoked potential electroencephalogram signals with in-memory computing, which can improve the performance of processing algorithms, reduce system power consumption, and enhance system integration.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A processing system for steady-state visually evoked electroencephalogram signals with in-memory computing, comprising: an electroencephalogram acquisition subsystem and a signal decoding subsystem;

[0009] The electroencephalogram acquisition subsystem is used for preprocessing the received steady-state visually evoked electroencephalogram signals;

[0010] The signal decoding subsystem includes an in-memory computing chip, and the in-memory computing chip runs a zero-calibration decoding algorithm based on the combination of a convolutional neural network and a Transformer model, performs decoding operations on the preprocessed electroencephalogram signals, and inputs the corresponding decoding results into an external control device.

[0011] Further, the electroencephalogram acquisition subsystem includes: an analog-to-digital converter, a programmable amplifier, a programmable master control chip, a filter, and a signal shaping circuit;

[0012] The analog-to-digital converter converts the received electroencephalogram signals in the analog domain into discrete digital values;

[0013] The programmable amplifier configures a gain value based on an input control signal and amplifies the electroencephalogram signals in digital value form according to the gain value;

[0014] The programmable master control chip programs the hardware communication interface to adapt to the interface requirements of different processing platforms;

[0015] The filter filters out low-frequency drift and high-frequency noise in the electroencephalogram signals output by the programmable master control chip;

[0016] The signal shaping circuit converts the signals output by the filter into a format that can be processed by the in-memory computing chip.

[0017] Further, the training process of the zero-calibration decoding algorithm based on the combination of a convolutional neural network and a Transformer model includes:

[0018] Taking the electroencephalogram signals preprocessed by the electroencephalogram acquisition subsystem as samples, and parallelly inputting them into a convolutional neural network and a Transformer model. The convolutional neural network extracts local temporal features and local spatial features of the electroencephalogram signals, and the Transformer model analyzes the global features of the electroencephalogram signals in the entire acquisition time window;

[0019] Adding the local spatio-temporal features output by the convolutional neural network and the global features output by the Transformer model to obtain fused features;

[0020] Input the fused features into the domain generalization module. By reweighting the training samples and using random Fourier features, eliminate the distribution shift between the source domain training data and the target domain test data, and remove the redundant correlation between the invariant discriminative information and the variable background information in the fused features.

[0021] Input the features processed by the domain generalization module into the classifier to obtain the classification result, that is, the frequency classification of the current visual stimulus received.

[0022] Furthermore, the memory-computation integrated chip includes one or more in-memory computing arrays; adjacent in-memory computing arrays are connected by neurons; each in-memory computing array performs matrix-vector multiplication calculations with changeable weights.

[0023] Furthermore, the working mode of the memory-computation integrated chip is configured into two modes: programming mode and operation mode; in the programming mode, the built-in peripheral control circuit controls the external control device to access the control interface of the memory-computation integrated chip through the communication circuit and write the pre-trained network weights into the in-memory computing array. After writing is completed, enter the operation mode; in the operation mode, receive the EEG signal data to be decoded and output the corresponding operation result to the external control device.

[0024] Furthermore, the in-memory computing array consists of two storage units. When writing the pre-trained network weights into one storage unit, the other storage unit performs multiplication operations with the input EEG signal data to be decoded, realizing the parallel operation of the programming mode and the operation mode.

[0025] Furthermore, the processing system also includes a power supply module; the memory-computation integrated chip is built with a power supply circuit, and the EEG acquisition subsystem also includes a power management chip, which is used to control the enabling of the power supply circuit for battery power supply or mains power supply, or control the enabling of the power supply module for self-power supply.

[0026] Furthermore, the EEG acquisition subsystem, the signal decoding subsystem, and the power supply module are all integrated on a single development board.

[0027] As can be seen from the above technical solutions, compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The present invention deploys the algorithm based on the memory-computation integrated chip, decodes and operates the EEG signals through the zero-calibration decoding algorithm combining the convolutional neural network and the Transformer model. While meeting the high recognition rate of the brain-computer interface across scenarios, the integration of the memory and the processor using the new hardware architecture ensures the overall low-power operation to meet the requirements of the edge-side brain-computer interface device.

[0029] 2. The present invention develops a high-precision electroencephalogram (EEG) acquisition subsystem. While effectively balancing precision, energy consumption, and area, it realizes internal customization through a programmable interface to adapt to the EEG signal requirements in various scenarios.

[0030] 3. The present invention integrates the EEG acquisition subsystem and the signal decoding subsystem onto a development board. The overall system balances the requirements of high precision and low power consumption and has strong integratability, showing great potential in the development of future brain-computer interface devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0032] Figure 1 It is a schematic structural diagram of the in-memory computing steady-state visually evoked EEG signal processing system provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0034] As Figure 1 shown, the embodiments of the present invention disclose an in-memory computing steady-state visually evoked EEG signal processing system, which is characterized by including: an EEG acquisition subsystem and a signal decoding subsystem;

[0035] The EEG acquisition subsystem is used to preprocess the received steady-state visually evoked EEG signals;

[0036] The signal decoding subsystem includes an in-memory computing chip. The in-memory computing chip runs a zero-calibration decoding algorithm pre-trained based on the combination of a convolutional neural network and a Transformer model, performs decoding operations on the preprocessed EEG signals, and inputs the corresponding decoding results into an external control device.

[0037] Starting from enhancing the quality of EEG acquisition, improving the performance of processing algorithms, reducing system power consumption, and strengthening system integration, the present invention designs a high-performance EEG acquisition subsystem to receive signals and perform preprocessing. With the memory-compute integrated chip as the core, high-performance algorithms are deployed on the memory-compute integrated chip to decode and output signals. Through the cooperation of software and hardware, the overall design is realized, thus achieving a high-performance, high-energy efficiency, and high-integration brain-computer interface module.

[0038] Next, the above two subsystems will be further described.

[0039] The EEG acquisition subsystem includes: an analog-to-digital converter, a programmable amplifier, a programmable master control chip, and a signal processing circuit.

[0040] The analog-to-digital converter converts the received EEG signals in the analog domain into discrete digital values.

[0041] The programmable amplifier configures the gain value based on the input control signal and amplifies the EEG signals in digital value form according to the gain value, featuring small size and low power consumption.

[0042] The programmable master control chip adopts a lightweight and high-energy efficiency solution, which is based on the end-side STM32 master control. It programs the hardware communication interface and transmits the sampled digital signals to the subsequent processing module through Bluetooth communication / WiFi communication, adapting to the interface requirements of different processing platforms.

[0043] The signal processing circuit consists of a filter and a signal shaping circuit. The filter filters out the low-frequency drift and high-frequency noise in the EEG signals output by the programmable master control chip, retaining the minimum fundamental frequency for processing and the 3 - 5 times maximum fundamental frequency harmonics.

[0044] The signal shaping circuit converts the signals output by the filter into a format that can be processed by the memory-compute integrated chip.

[0045] To improve the performance of the processing algorithm, the present invention uses a zero-calibration decoding algorithm based on the combination of convolutional neural network and Transformer model to decode EEG signals. Before the system works, this decoding algorithm needs to be trained. The specific training process includes:

[0046] Taking the EEG signals preprocessed by the EEG acquisition subsystem as samples, they are parallelly input into the convolutional neural network and the Transformer model. The convolutional neural network extracts the local temporal features and local spatial features of the EEG signals, and the Transformer model analyzes the global features of the EEG signals under the entire acquisition time window;

[0047] Adding the local spatio-temporal features output by the convolutional neural network and the global features output by the Transformer model to obtain the fused features;

[0048] Input the fused features into the domain generalization module. By reweighting the training samples and using random Fourier features, eliminate the distribution shift between the source domain training data and the target domain test data, and remove the redundant correlation between the invariant discriminative information and the variable background information in the fused features.

[0049] Input the features processed by the domain generalization module into the classifier to obtain the classification result, that is, the frequency classification of the current visual stimulus received.

[0050] After training, the network weights can be obtained. The present invention uses the data preprocessed by the electroencephalogram acquisition subsystem as samples for training to consider the additional noise / errors introduced by the hardware in the middle; and moderately prunes the model to achieve customized software and hardware training, improving the accuracy and deployment efficiency of the algorithm on the hardware.

[0051] To reduce the system power consumption, the present invention uses a memory-compute integrated chip as the hardware operation platform for the decoding process. The main component module of the memory-compute integrated chip is the in-memory computing array, which is designed based on special microelectronic devices and is a large-scale parallel computing array. According to the physical laws of the devices and Kirchhoff's laws, it can implement matrix-vector multiplication calculations with changeable weights. Matrix-vector multiplication accounts for more than 90% of the neural network calculations, and large-scale acceleration optimization can be performed on it.

[0052] The memory-compute integrated chip includes one or more in-memory computing arrays. The number of in-memory computing arrays can be configured according to specific scenarios. Each group of arrays is used to calculate a matrix-vector multiplication; adjacent two in-memory computing arrays are connected by neurons, and neurons refer to the activation functions between layers of the neural network.

[0053] The working mode of the memory-compute integrated chip is configured into two modes: programming mode and operation mode; in the programming mode, the built-in peripheral control circuit controls the external control device to access the control interface of the memory-compute integrated chip through the communication circuit and write the pre-trained network weights into the in-memory computing array. After writing is completed, enter the operation mode; in the operation mode, receive the electroencephalogram signal data to be decoded and output the corresponding operation result to the external control device. Among them, the peripheral control circuit is used for the scheduling of timing and communication, that is, controlling the scheduling inside the entire chip system, such as writing the processed neural network weights into the in-memory computing array through the peripheral control circuit.

[0054] For scenarios that require replacing the neural network weights, a coarse-grained in-memory computing array can be used. The in-memory computing array consists of two memory units. When the pre-trained network weights are written into one memory unit, the other memory unit performs a multiplication operation with the input EEG signal data to be decoded, achieving parallel operation of the programming mode and the operation mode. Parallel operation of the programming mode and the operation mode can be achieved by moderately sacrificing area, thereby improving the system parallelism and the overall operation speed.

[0055] In one embodiment, the processing system further includes a power supply module; the in-memory computing chip is built with a power supply circuit, and the EEG acquisition subsystem further includes a power management chip. The power management chip is used to control the activation of the power supply circuit for battery power supply or mains power supply, or to control the activation of the power supply module for self-powered power supply. This power supply method can adapt to the usage requirements of edge devices, adopt self-powered or self-powered-battery collaborative power supply, and improve the system's endurance. The self-powered method can use a photovoltaic solar panel to collect solar energy, a thermoelectric generator to collect thermal energy, an RF radio frequency energy collector to collect radio frequency energy, a piezoelectric converter or an electromagnetic generator to collect mechanical energy or other energy collection systems.

[0056] To enhance the system integration, the present invention integrates the EEG acquisition subsystem, the signal decoding subsystem, the power supply module, and the communication circuit on a single development board. By designing the overall signal chain and energy supply chain, it is adapted to the device form of the edge side.

[0057] Generally speaking, when the system of the present invention processes EEG signals, first, through the analog integrated front end, the signal is converted from the analog domain to the digital domain. Then, via the filter and shaping circuit, the signal frequency band screening and basic preprocessing are completed. During this process, the signal timing and communication scheduling of the programmable main control chip control module are controlled; the processed signal is input into the in-memory computing chip, and the decoding algorithm is run through the configured in-memory computing array, and the corresponding decoding result is input into the external control device through the communication circuit, thereby completing the real-time control of the external device. In addition, the external control device performs real-time information interaction with the subject, enabling the subject to adjust the control accuracy according to the received information, achieving efficient two-way interaction.

[0058] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0059] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A storage and computing integrated steady-state visual evoked electroencephalogram signal processing system, characterized in that: include: EEG acquisition subsystem and signal decoding subsystem; The EEG acquisition subsystem is used to pre-process the received steady-state visually evoked EEG signals; The signal decoding subsystem includes a storage and computing integrated chip, which runs a pre-trained zero calibration decoding algorithm based on a combination of a convolutional neural network and a Transformer model, performs decoding operations on the pre-processed EEG signals, and inputs the corresponding decoding results into an external control device.

2. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 1 is characterized in that: The EEG acquisition subsystem includes: an analog-to-digital converter, a programmable amplifier, a programmable main control chip, a filter and a signal shaping circuit; The analog-to-digital converter converts the received EEG signal in the analog domain into a discrete digital value; The programmable amplifier configures a gain value based on an input control signal, and amplifies the EEG signal in digital value form according to the gain value; The programmable main control chip programs the hardware communication interface to adapt to the interface requirements of different processing platforms; The filter filters low-frequency drift and high-frequency noise in the EEG signal output by the programmable main control chip; The signal shaping circuit converts the signal output by the filter into a format that can be processed by the integrated storage and computing chip.

3. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 1, characterized in that: The training process of the zero calibration decoding algorithm based on the combination of convolutional neural network and Transformer model includes: The EEG signals preprocessed by the EEG acquisition subsystem are used as samples and input into the convolutional neural network and the Transformer model in parallel, the local time features and local spatial features of the EEG signals are extracted by the convolutional neural network, and the global features of the EEG signals in the entire acquisition time window are analyzed by the Transformer model; Add the local spatiotemporal features output by the convolutional neural network and the global features output by the Transformer model to obtain the fused features; The fused features are input into the domain generalization module, which uses random Fourier features to eliminate the distribution shift between the source domain training data and the target domain test data by re-weighting the training samples, and removes the redundant correlation between the invariant discriminant information and the variable background information in the fused features; The features processed by the domain generalization module are input into the classifier to obtain the classification result, that is, the frequency classification of the current visual stimulation.

4. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 1, characterized in that: The integrated memory and computing chip includes one or more in-memory computing arrays; two adjacent in-memory computing arrays are connected via neurons; and each in-memory computing array performs a matrix-vector multiplication calculation with changeable weights.

5. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 4, characterized in that: The working mode of the integrated storage and computing chip is configured into two modes, programming mode and operation mode; in the programming mode, the built-in peripheral control circuit controls the external control device to access the control interface of the integrated storage and computing chip through the communication circuit, writes the pre-trained network weights to the internal computing array, and enters the operation mode after writing; in the operation mode, the EEG signal data to be decoded is received, and the corresponding operation results are output to the external control device.

6. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 5, characterized in that: The in-memory computing array consists of two storage units. When the pre-trained network weights are written into one storage unit, the other storage unit performs multiplication operation with the input EEG signal data to be decoded, thereby realizing parallel operation of the programming mode and the operation mode.

7. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 1, characterized in that: The processing system also includes a power supply module; the integrated storage and computing chip has a built-in power supply circuit, and the EEG acquisition subsystem also includes a power management chip, which is used to control the activation of the power supply circuit for battery power or AC power supply, or control the activation of the power supply module for self-power supply.

8. The storage and computing integrated steady-state visual evoked electroencephalogram signal processing system according to claim 1, characterized in that: The EEG acquisition subsystem, the signal decoding subsystem and the energy supply module are all integrated on a single development board.

Images