A hybrid brain-computer interface system based on ultrasonic haptics and spatial imagination

Abstract

A kind of mixed brain-computer interface system based on ultrasonic haptics and space imagination, including ultrasonic haptics phased array, electroencephalogram cap, electroencephalogram signal detection board and host computer;In ultrasonic haptics phased array task module, the corresponding relationship of input instruction, tactile perception instruction and output instruction is constructed, and time-frequency domain coding method is used to generate tactile perception instruction.Ultrasonic haptics phased array focuses multiple ultrasonic waves on human hand according to the order of tactile perception instruction, and constructs different shapes, so that human hand can perceive different shapes and tactile characteristics, and induce space imagination.Electroencephalogram cap positions the brain to collect the electroencephalogram signal of space imagination position induced by feeling touch, and transmits to electroencephalogram signal detection board.Host computer receives bioelectric signal, carries out channel separation, filtering denoising processing and intelligent decoding to original bioelectric signal, and generates output instruction in turn.The system of the application can control unmanned equipment with output instruction, and can also train and improve human cognitive ability.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and brain-computer interface technology in electronic and information engineering, and relates to a hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination. Background Technology

[0002] Brain-computer interface (BCI) is a real-time communication system that connects the brain to external devices. It enables direct control of external devices using brain signals, allowing for direct communication and interaction between consciousness and the external environment. BCI systems have broad application prospects in military, clinical, and everyday life applications. BCI can not only directly control external devices or muscle tissue by acquiring spontaneous firing of neurons in specific areas and decoding them into command signals, but it can also feed back responses to neural networks, enabling visual, auditory, and tactile sensations. Non-invasive electroencephalography (EEG) signals are the sum of postsynaptic potentials emitted by a group of neurons recorded by placing electrodes on the surface of the scalp or cerebral cortex. Currently, the brain models used in EEG-based brain-computer interfaces mainly include the P300 visual evoked potential proposed by Farwell and Donchin in 1988, steady-state evoked potentials proposed by Muller et al. (such as steady-state visual evoked potentials (SSVEP)), and event-related desynchronization / synchronization (ERD / ERS) generated by motor imagery (MI).

[0003] Significant progress has been made in paradigm design, brain signal processing algorithms, and control systems for single-modal BCI systems. Among these, visual stimulation modes are relatively mature in evoked stimulus BCI applications, but shortcomings remain. For example, P300-bit-based BCIs require repeated flashing, and prolonged repetitive flashing can affect the subject's brain signals. The number of control commands on SSVEP-based BCIs is affected by stimulation frequency and other factors, especially as the number of commands on the brain-computer interface increases, leading to a decrease in classification accuracy. Motor imagery-based BCIs require extensive practice from the subject, which can easily lead to fatigue, affecting concentration and brain signal quality. Furthermore, these BCI systems face challenges including low information transmission rates, multi-faceted / functional control, human-machine adaptability, robustness, and stability. One potential approach to addressing these challenges is to explore new paradigm-based BCI systems. Tactile paradigm stimulation presents a novel BCI system that evoked potentials without relying on visual stimulation. This invention proposes a BCI technology based on an ultrasonic tactile mode. Ultrasonic haptic technology uses an array of ultrasonic transducers to focus ultrasound onto the skin surface, inducing shear waves and thus producing a tactile sensation. Through ultrasonic haptic technology, the human hand can perceive different two-dimensional patterns and tactile characteristics. Therefore, while the human body receives tactile sensation, a corresponding spatial image is generated, and changes in neuronal electrical signals occur in the corresponding functional areas of the brain. These changes can be analyzed by multi-channel EEG signals detected by brain-computer interfaces. Therefore, this technology can be used for BCI-controlled unmanned equipment and can also be used to train and enhance human cognitive abilities.

[0004] Currently, few researchers, both domestically and internationally, focus on the application of tactile stimulation-induced EEG signals in BCI systems. Research on ultrasonic tactile sensing is relatively recent, with most studies directly applying it to human-computer interaction, and no research has yet found that utilizes it in BCI technology. Therefore, this invention proposes to construct an integrated system for ultrasonic tactile testing and EEG signal detection to achieve a hybrid brain-computer interface technology based on both tactile perception and spatial imagination. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art. The present invention provides a hybrid brain-computer interface system based on ultrasonic tactile and spatial imagination, realizes BCI technology based on ultrasonic tactile paradigm stimulation, designs perceptual instructions based on ultrasonic tactile paradigm, and classifies tactile perceptual instructions according to the tactile characteristics constructed by the temporal control technology of ultrasonic tactile phased array and the spatial imagination induced by tactile patterns.

[0006] The present invention adopts the following technical solution: a hybrid brain-computer interface system based on ultrasonic tactile sensing and spatial imagination, including an ultrasonic tactile phased array, an EEG cap, an EEG signal detection board, and a main control computer; the ultrasonic tactile phased array is connected to the main control computer via USB, the EEG cap is worn on the head, the EEG cap is connected to the analog front-end module of the EEG signal detection board, and the EEG signal detection board transmits data to the main control computer via wireless communication;

[0007] The main control computer includes an ultrasonic haptic phased array task module, an EEG signal receiving module, a channel data separation module, a filtering and noise reduction module, and an intelligent decoding module. During operation, the main control computer inputs a task file containing input instructions and distributes the task file to the ultrasonic haptic phased array task module. The task module parses the input instructions, generates haptic perception instructions, and sequentially sends these instructions to the ultrasonic haptic phased array slave computer. The slave computer then sequentially emits several ultrasonic waves according to the order of the haptic perception instructions, focusing them onto the user's hand and constructing different shapes to make the hand... The system senses different shapes and tactile characteristics; the EEG cap locates and collects EEG signals that the brain senses touch and generates spatial imagination, and transmits them to the analog front-end module of the EEG signal detection board. The EEG signal detection board completes the acquisition, amplification, A / D conversion, data packaging, and wireless transmission of the EEG signals; the EEG signal receiving module of the main control computer receives the bioelectric signals acquired and converted by the EEG signal detection board. The channel data separation module, filtering and noise reduction module, and intelligent decoding module respectively perform channel separation, filtering and noise reduction, and intelligent decoding on the original bioelectric signals, and generate corresponding output control commands in sequence according to the intelligent decoding results.

[0008] Furthermore, the ultrasonic tactile phased array includes M×N transducers. Controlled by a main computer, the ultrasonic tactile phased array emits M×N channels of 40kHz ultrasonic waves with different phase delays. These waves can be modulated with 1-1kHz signals and focused at any point. A time-controlled mechanism activates and deactivates the ultrasonic waves at a set frequency, allowing the human hand to experience ultrasonic vibrations. The ultrasonic tactile phased array controls the focal point to move along a given two-dimensional shape at a set speed and several designated positions, allowing the human hand to experience tactile perception of a two-dimensional shape simulated by several discrete focal points. According to the task file input instructions, the ultrasonic tactile phased array's lower-level computer emits focused ultrasonic waves at a specified height using a time-controlled mechanism to construct characters or regular shapes. When the human hand is placed at the specified height, it can perceive the constructed characters or regular shapes and generate a spatial image. The ultrasonic tactile phased array executes the tasks according to the order of the task file input instructions, where M and N are positive integers.

[0009] Furthermore, the EEG signal detection board includes an analog front-end module, a main control module, a power management module, and a Wi-Fi radio frequency module. Raw EEG signals are acquired by bioelectrodes on the EEG cap using a single-ended input method. After passing through a passive resistor-capacitor low-pass filter, the EEG ADC chip in the analog front-end module amplifies and samples the signal, and transmits the converted digital signal to the main control module MCU. The main control module MCU sends the packaged EEG data to the Wi-Fi radio frequency module via a UART serial port. The data is then wirelessly transmitted to the main control computer by the Wi-Fi radio frequency module.

[0010] Furthermore, the system software of the EEG signal detection board implements data acquisition and logic control functions, realizes the initialization of various peripherals of the MCU, external interrupts of the MCU, synchronous sampling, and ADC chip register configuration. It reads and packages the EEG data through SPI and sends it to the Wi-Fi radio frequency module from the UART interface, thus completing the logic control function of the entire system.

[0011] Furthermore, the task file includes a permutation and combination of one or more input instructions; the input instructions are represented by Arabic numerals, with one Arabic numeral representing one input instruction; the ultrasonic haptic phased array task module has the function of defining the input instructions as corresponding characters or regular shapes, where the characters are Chinese characters or English letters, and the corresponding characters or regular shapes are called haptic perception instructions. The task module uses a time-frequency domain encoding method to generate haptic perception instructions; the time-frequency domain encoding method is based on the modulation frequency of the emitted ultrasonic waves and the focal scanning time interval for constructing the two-dimensional shape; the ultrasonic haptic phased array task module has the function of defining haptic perception instructions as corresponding output instructions. During operation, the ultrasonic haptic phased array task module automatically identifies the input instructions in the task file, converts the input instructions into haptic perception instructions of characters or regular shapes, and sends them to the ultrasonic haptic phased array lower-level computer.

[0012] Furthermore, the EEG signal receiving module initiates a TCP server to receive and verify the raw wireless data transmitted from the EEG signal detection board, and then inputs the raw wireless data to the channel data separation module.

[0013] Furthermore, the channel data separation module divides the multi-channel data into their respective data channels, converts the original 16-bit raw data into decimal amplitude data, and then inputs the decimal amplitude data into the filtering and noise reduction module.

[0014] Furthermore, the filtering and denoising module performs noise reduction processing on the data through bandpass filtering and power frequency dip digital filter, and then inputs it into the intelligent decoding module to obtain the parsed output command.

[0015] Furthermore, the intelligent decoding module employs a classification model to filter key features that are more sensitive to tactile / spatial imagination from the collected multi-channel data for instruction classification. For tactile information of EEG signals detected based on the ultrasound tactile evoked paradigm, it uses feature extraction and classification methods based on canonical correlation analysis (CCA) and deep neural network model training. For spatial imagination EEG signals detected based on the ultrasound tactile evoked paradigm, it uses feature extraction based on one-to-one and one-to-many co-spatial patterns (CSP) and classification modes based on SVM. The module uses average weighting and prior weighting signal fusion algorithms to perform signal fusion on the classification results based on EEG signal frequency information and spatial imagination EEG signals, thus realizing the brain-computer interface decoding algorithm for input task control instructions.

[0016] Furthermore, in the ultrasonic tactile phased array task module, an experimental paradigm is designed based on the resonant frequencies of the individual characteristics of the human hand and fingers.

[0017] The beneficial technical effects of this invention compared to the prior art are:

[0018] (1) Current non-invasive BCI technologies mainly use P300 and SSVEP, which require flashing stimulation of the human eye at different frequencies, which is not a natural and comfortable method. However, the ultrasonic-induced tactile sensation method provided by this invention does not stimulate the human eye and has better comfort.

[0019] (2) The ultrasonic tactile technology provided by the present invention can control ultrasonic waves to start and stop at a controllable frequency, so that the human hand can obtain ultrasonic vibration. It can also control the focus to move along a given two-dimensional shape at a certain speed in several set positions to form a time-controlled frequency and induce spatial imagination. This information is hidden in the detected raw EEG signal. The method of extracting and classifying brain-computer signal features based on spatial imagination and frequency stimulation can be adopted. The information is large, which can improve the reliability and accuracy of BCI control and reduce a lot of training.

[0020] (3) Through the testing of different groups of people by the system of the present invention, the tactile perception and spatial imagination ability of different subjects can be detected. For people with declining perception and cognitive abilities, the system can be used for training and improvement. Attached Figure Description

[0021] Figure 1 This is a system composition block diagram of the present invention;

[0022] 1. Ultrasonic tactile phased array; 2. EEG cap; 3. EEG signal detection board; 4. Main control computer.

[0023] Figure 2 The ultrasonic tactile phased array device used in this invention;

[0024] Figure 3 This is a diagram of the functional divisions of the cerebral cortex used in this invention (black boxes indicate the somatosensory cortex);

[0025] Figure 4 The above are Broadman brain region maps used in this invention, wherein (a) is a Broadman region map of the lateral surface of the cerebral hemisphere and (b) is a Broadman region map of the medial surface of the cerebral hemisphere.

[0026] Figure 5 This is a schematic diagram of the 10-20 international standard lead system used in this invention, (a) is a side view, and (b) is a top view;

[0027] Figure 6 This is a schematic diagram of the electrode distribution of the 32-channel EEG cap provided by the present invention;

[0028] Figure 7 This is a schematic diagram of the right leg drive circuit of the ADC chip in the EEG signal detection board of the present invention;

[0029] Figure 8 This is a flowchart of the embedded software of the EEG signal detection board of the present invention.

[0030] Figure 9 This is a schematic diagram of the implementation framework for separating the EEG signal channels in this invention;

[0031] Figure 10 The diagram shows the bandpass filter of the present invention, wherein (a) is the amplitude response transfer function diagram and (b) is the phase response transfer function diagram;

[0032] Figure 11 The diagram shows the power frequency notch filter of the present invention, wherein (a) is the amplitude response transfer function diagram and (b) is the phase response transfer function diagram. Detailed Implementation

[0033] The following is in conjunction with the appendix Figure 1-11 The present application will be further described in detail with reference to specific embodiments:

[0034] Reference Figure 1 A hybrid brain-computer interface system based on ultrasonic tactile sensing and spatial imagination includes an ultrasonic tactile phased array, an EEG cap, an EEG signal detection board, and a main control computer, wherein:

[0035] (1) Ultrasonic tactile phased array

[0036] The ultrasonic tactile phased array device consists of three interconnected circuit boards. The first board is an ultrasonic transducer array, composed of 252 40kHz ultrasonic transducers. The second board is an ultrasonic phased array driver board, whose main function is to amplify the waveform to drive the transducers. The third board is a control board, including a CPU and an FPGA. The CPU is responsible for communicating with the external host, and communication between the CPU and the FPGA uses PCIE / Local Bus. The FPGA module calculates the phase delay of all transducers in real time based on the target position and the distance between the transducers, and controls the beamformer to generate a 40kHz rectangular wave. It can also be modulated with 1-1kHz pulse signals (1Hz steps) to generate ultrasonic vibration waves of different frequencies to simulate different tactile sensations. Human tactile perception has a temporal resolution of only a few milliseconds (reportedly, the precise value may be between 2 and 40ms). Therefore, this invention constructs tactile perception of two-dimensional graphics through temporal control technology. When designing a two-dimensional graphic (such as a regular circle or the character A), if the speed of the focus along the trajectory is faster than the temporal resolution of touch, the user will perceive the resulting stimulus as a single, smooth tactile pattern, constructing a spatial imagination of the two-dimensional graphic. The ultrasonic phased array is equipped with a host computer software development library. The dynamic library interface functions define all functions of the tactile controller, including focus setting / acquisition, drive signal setting / acquisition, modulation signal setting / acquisition, area enable setting / acquisition, global configuration setting / acquisition, sending configuration to the phased array / setting the sent configuration response callback interface, trigger control / setting the trigger control callback interface, and acquiring temperature / acquiring temperature callback interface, etc. An appearance diagram of the ultrasonic phased array device is shown below. Figure 2 As shown.

[0037] Table 1 defines the task module instructions of this invention.

[0038]

[0039] (2) EEG cap and tactile sensing area

[0040] The electrical activity of the cerebral cortex is spontaneous, and its potential changes over time. EEG signal acquisition can be categorized into invasive and non-invasive methods based on the location of the measuring electrodes within or outside the cerebral cortex. Since invasive methods can damage the brain, this invention employs a non-invasive approach for EEG acquisition. There are many options for the number of EEG measurement channels; generally, more channels result in a richer information output by the EEG. Common channel numbers include 8, 16, 20, 24, 32, and 64. Different channel numbers determine the fixed positions of the electrodes. In EEG measurement, there is an internationally standardized electrode placement rule, known as the "International 10-20 System."

[0041] Measuring brainwaves essentially involves measuring the potential difference between two (or more) locations in the brain. For a single EEG signal, the potential difference between two electrodes needs to be measured. One electrode serves as the reference electrode, i.e., the zero-potential reference point. The potential difference between these two points is the absolute potential of the other electrode. In actual measurements, the reference point is usually chosen at locations such as the earlobe or the tip of the nose. The electrode placed at the zero-potential point is called the irrelevant electrode, and the corresponding electrode located on the scalp is called the active electrode.

[0042] The brain's sensory center for touch is primarily located in the postcentral gyrus of the preparative lobe, between the central and postcentral sulci. Figure 3 As shown. Tactile stimulation activates the primary somatosensory cortex (S1) and the secondary somatosensory cortex (S2). In the Broadman area, S1 includes regions 1, 2, 3a, and 3b, while S2 is located in region 40. Current research on the brain's processing of tactile information mainly focuses on S1, such as... Figure 4 As shown, (a) is the Broadman map of the lateral surface of the cerebral hemisphere, and (b) is the Broadman map of the medial surface of the cerebral hemisphere.

[0043] The 10-20 system electrode placement method is the standard electrode placement method prescribed by the International Society for Electroencephalography (ESEG), and includes 21 electrodes, such as... Figure 5 As shown, (a) is a side view and (b) is a top view. For a 16-channel EEG cap, EEG signals can be acquired at locations such as F3, Fz, F4, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, and O2.

[0044] The 32-channel EEG cap offers finer segmentation than the 10-20 international standard lead system, adding 17 electrodes: Fpz, FC5, FC1, FC2, FC6, T7, T8, M1, M2, CP5, CP1, CP2, CP6, P7, P8, POz, and Oz. It lacks 6 electrodes: A1, A2, T3, T4, T5, and T6. When performing tactile stimulation, the focus should be on studying 11 electrodes: C3, Cz, C4, CP5, CP1, CP2, CP6, P3, Pz, P4, and POz. Figure 6 As shown. Similar to the visual and auditory systems, touch also has a personal characteristic resonant frequency, to which the human nervous system is most sensitive to stimulation. The personal characteristic resonant frequency of the palm is approximately 21 Hz. Applying mechanical sinusoidal vibrations to the skin at a constant frequency or repeatedly activating its afferent nerves can induce neuronal activity in the S1 cortex. In the EEG spectrum, the stimulation frequency and its harmonic frequencies will produce obvious peaks, decreasing sequentially. Generally, reliable steady-state somatosensory evoked potentials can only be generated at stimulation frequencies below 50–100 Hz, and modulating high-frequency signals with low frequencies can also generate corresponding low-frequency steady-state somatosensory evoked potentials in the EEG signal.

[0045] (3) Electroencephalogram (EEG) signal detection board

[0046] The main functions of the EEG detection board-level hardware circuit are to acquire, amplify, convert (A / D) data, package data, and transmit it wirelessly. The hardware circuit supports multi-channel synchronous sampling, 5G / 2.4G dual-band wireless transmission, and is powered by dry cell batteries. It includes an analog front-end module, a main control module, a power management module, and a Wi-Fi radio frequency module.

[0047] The hardware circuit works as follows: First, the bioelectrodes acquire raw EEG signals using a single-ended input method. After passive RC low-pass filtering, the ADC chip amplifies and samples the signal, then transmits the converted digital signal to the MCU. The MCU sends the packaged EEG data to the Wi-Fi module via the UART serial port, and the data is then wirelessly transmitted to the host computer via the Wi-Fi radio frequency module.

[0048] 1. Main control module

[0049] The EEG acquisition system has certain requirements for the processing speed, storage space, communication interface, portability, and stability of the main control module, and therefore uses a microprocessor as the main control chip. The EEG acquisition system has high requirements for power supply ripple; if the power supply circuit's filtering effect is not ideal, voltage fluctuations may occur, introducing more high-frequency noise coupled into the analog part of the EEG acquisition circuit. Therefore, the EEG signal detection board uses a low dropout regulator (LDO) to power the main control chip. To further suppress high-frequency noise, filter capacitors are connected in parallel at the power input and output pins. In addition to the power supply circuit, the microprocessor also requires a JTAG programming interface, an external crystal oscillator, and a reset button to function properly.

[0050] 2. Simulated front-end module

[0051] The design of the analog front-end module of the EEG acquisition system takes into account the extremely small voltage signals of EEG, typically only at the microvolt level. Furthermore, because the subject's body acts as an antenna, it is susceptible to electromagnetic interference, such as 50Hz household power frequency interference. This interference may mask the body's biological signals, making them difficult to measure. The right-leg driver circuit is essentially a negative feedback circuit; its function is to reduce common-mode interference and improve the common-mode rejection ratio. The right-leg driver can suppress the 50 / 60Hz common-mode variation in the human body without affecting the body's physiological electrical signals. It can simultaneously eliminate common-mode and differential-mode interference caused by the power line and has real-time dynamic adjustment capabilities. The principle of the right-leg drive circuit is: by acquiring the common-mode interference from the electrodes, amplifying and inverting the common-mode variation signal, and then connecting it back to the body, the negative feedback eliminates the common-mode interference. The right-leg drive circuit based on an ADC chip is shown below. Figure 7As shown, the ADC chip has 8 channels. The common-mode signals transmitted from each channel are superimposed after passing through the corresponding equivalent resistor. The superimposed common-mode signals from all channels are then uniformly passed through an inverting amplifier bias to reverse their polarity. Afterward, just like a single channel, the inverted signal is fed back to the subject, thereby eliminating some of the influence of power frequency interference.

[0052] The ADC chip has a programmable amplifier with 7 gain options (1, 2, 4, 6, 8, 12, 24). The gain can be adjusted by modifying the register. After connecting the right leg drive electrode, the differential electrode input can be maintained at tens of mV. Therefore, when actually collecting EEG data, the maximum gain of 24 times can be used.

[0053] The ADC chip integrates eight low-noise, synchronous sampling 24-bit Σ-Δ ADCs. Its main function is to convert the acquired EEG analog signals into digital signals that the MCU can recognize. Since a single ADC chip only has eight electrode channels, multiple ADC chips need to be cascaded to expand the number of channels. The ADC chip supports DAIZY-CHAIN ​​topology mode to expand the system channels. This DAIZY-CHAIN ​​mode is enabled by setting the DAIZY_EN bit in the CONFIG1 register.

[0054] 3. Wireless Local Area Network Transmission System

[0055] In the design of EEG signal transmission systems, the Wi-Fi transmission system plays a crucial role in accurately, stably, and promptly transmitting the data stream output from the microprocessor's main control unit to the serial port (UART). This data is then wirelessly transmitted to the main control computer software program using a WLAN communication protocol based on the 2.4GHz / 5.8GHz frequency band.

[0056] 4. System Software

[0057] The system software primarily performs the following functions: initialization of MCU peripherals, configuration of external interrupts, configuration of ADC chip registers, reading and packaging EEG data via SPI, and sending it to the Wi-Fi module through the UART interface. Ultimately, it completes the logic control functions for the entire system. Its overall flowchart is as follows: Figure 8 As shown.

[0058] (4) Main control computer

[0059] The main control computer is designed with the following five modules: ultrasonic tactile phased array task module, EEG signal receiving module, channel data separation module, filtering and noise reduction module, and intelligent decoding module.

[0060] 1. Ultrasonic haptic phased array task module

[0061] The input task file consists of one or more input instructions arranged in combination. Input instructions are represented by Arabic numerals, with one Arabic numeral representing one input instruction. The ultrasonic haptic phased array task module has the function of defining input instructions as corresponding characters or regular shapes; the characters can be Chinese characters or English letters. The corresponding characters or regular shapes are called haptic sensing instructions. The task module also has the function of defining haptic sensing instructions as corresponding output instructions. Table 1 shows the task module instruction definitions. During operation, the ultrasonic haptic phased array task module automatically identifies the input instructions in the task file, converts the input instructions into haptic sensing instructions of characters or regular shapes, and sends them to the ultrasonic haptic phased array lower-level computer.

[0062] The input task file consists of one or more arbitrary combinations of input commands. For example, if the input command is 154263, the generated tactile sensing commands are triangle, up, P, circle, down, and L. Finally, the output commands, after intelligent decoding by the system, are takeoff, up, hover, circle, down, and landing.

[0063] The task module uses a time-frequency domain coding method to generate tactile perception commands. This method encodes signals based on the modulation frequency of the emitted ultrasonic waves and the focal scanning time interval for constructing the two-dimensional shape. The ultrasonic tactile phased array system provides a modulation frequency range of 1-1kHz, with the optimal frequency designed based on the individual characteristic resonance frequencies (17-31Hz) of the human hand and fingers, as well as tactile characteristics. This invention preferably uses frequencies between 1-50Hz for encoding, as shown in the following frequency encoding examples: triangle with a modulation frequency of 5Hz, circle with a modulation frequency of 15Hz, "L" with a modulation frequency of 22Hz, "P" with a modulation frequency of 25Hz, "up" with a modulation frequency of 30Hz, and "down" with a modulation frequency of 35Hz.

[0064] The time-based control technology uses n focal points for time-division focusing. The focal point scanning time interval is encoded as another tactile sensing characteristic. Examples of time interval encoding are as follows: triangle: 3 focal points, 10ms scanning time interval; circle: 12 focal points, 12ms scanning time interval; "L": 15 focal points, 14ms scanning time interval; "P": 15 focal points, 16ms scanning time interval; "Up": 20 focal points, 18ms scanning time interval; "Down": 20 focal points, 20ms scanning time interval. When a tactile sensing command is sent to the ultrasonic tactile phased array lower-level computer, regular shapes, letters, and Chinese characters can be fitted by the n focal points. The ultrasonic tactile phased array sequentially emits ultrasonic waves to these n focal points and quickly switches between them, thus forming a two-dimensional graphic tactile sensing effect.

[0065] 2. EEG signal receiving module

[0066] To ensure the stability and accuracy of raw EEG data transmission, TCP was selected as the data interaction protocol between the EEG acquisition system and the host computer. The main control computer was preset as the server (TCP_SEVER), and the EEG acquisition system was preset as the client (TCP_CLIENT), thus establishing a complete client / server architecture. In reliable TCP transmission, relying solely on the reliability of the TCP protocol itself is far from sufficient. Designers need to encapsulate the raw data into a complete data frame before transmission. This data frame has a fixed size and contains one or more verification frame bits to check the correctness of the identification. At the receiving end, a corresponding length of receive window can be set, and the corresponding frame verification bits can be verified to determine the correctness of the raw test data carried in the data frame. Furthermore, during long-term, high-speed wireless data transmission, this data frame processing ensures that accidental transmission errors are identified and discarded by the receiving end before causing catastrophic consequences, greatly improving the robustness of the wireless transmission process.

[0067] 3. Channel data separation module

[0068] This implementation uses embedded C code to separate multi-channel data and convert it into computationally readable data from byte arrays. The core processing idea consists of two parts: regular code (for cases where the frame verification bit is at the beginning of the array) and emergency (unexpected) code (for cases where the frame verification bit is anywhere in the middle). The channel separation implementation framework diagram is shown below. Figure 9 As shown.

[0069] 4. Filtering and noise reduction module

[0070] Noise reduction is achieved through bandpass and power frequency dip digital filters. EEG signals are continuous bioelectrical signals in the time domain, ranging from a few μV to several hundred μV. Even when only bioelectrical signals from the head are collected, they are mixed with a large number of electrooculogram (EOG), electrocardiogram (ECG), and body movement signals. These non-EEG signals are collectively referred to as interference signals, which have a significant impact on subsequent EEG time-domain and frequency-domain feature analysis. At the same time, even if the system is not directly connected to the power system (national standard 220V / 50Hz), it may still be subject to spatial radio frequency interference generated by the power system during system operation. The resulting noise interference will submerge the original EEG data. The best way to solve these two problems is to use a digital filter in the host computer software to filter the transmitted raw data.

[0071] Power frequency interference is usually the most serious problem in bio-electroelectric data acquisition systems. This is because the effective EEG signal we acquire is typically only on the order of tens of μV, while power system interference can often reach the mV level. This results in the required effective signal being submerged in power frequency interference, making data analysis impossible. To ensure that the final displayed waveform is meaningful, a digital bandpass (bandstop) filter and a 50Hz notch filter must be added beforehand.

[0072] This module uses a Butterworth bandpass filter with a low cutoff frequency (Fs) of 2Hz and a high cutoff frequency (Fh) that can be set by the experimenter according to their needs (default 45Hz). The passband gain is 1x, and the filter order is 10. Figure 10 (a) is the amplitude function of the designed Butterworth bandpass filter, such as Figure 10 (b) is the phase function of the designed Butterworth bandpass filter.

[0073] For power frequency interference, IIR (In-line Resonance) notch filters (IIR_NOTCH) are required in the design. A notch filter is a filter that rapidly attenuates the input signal at a specific frequency, effectively blocking that frequency signal from passing through. Notch filters are a type of band-stop filter, but their stopband is very narrow, and the filter order must be at least third order. Essentially, a notch filter is a high-order Butterworth band-stop filter with a low cutoff frequency (Fs) of 48Hz and a high cutoff frequency (Fh) of 52Hz. The IIR filter order is 10. Figure 11 (a) is the amplitude function of the designed power frequency notch filter. Figure 11 (b) shows the phase function of the designed power frequency notch filter. It can be seen that the gain coefficient is close to 0 at 50Hz, which is sufficient to reduce the impact of power frequency interference on the final result.

[0074] 5. Intelligent decoding module

[0075] The intelligent decoding module uses a classification model to filter out key features that are more sensitive to tactile / spatial imagination from the collected multi-channel data for instruction classification.

[0076] For tactile information from EEG signals detected using the ultrasound-induced tactile paradigm, feature extraction and classification methods based on Canonical Correlation Analysis (CCA) and deep neural network model training, such as EEGNet, Transformer, and RFW-4DCRNN, can be employed. For spatial imagery EEG signals detected using the ultrasound-induced tactile paradigm, feature extraction based on one-to-one and one-to-many co-spatial patterns (CSP) and classification based on Support Vector Machine (SVM) are used. A signal fusion algorithm combining average weighting and prior weighting is applied to the classification results based on tactile information and spatial imagery EEG signals to implement a brain-computer interface decoding algorithm for input task control commands.

[0077] Feature extraction and classification of CCA:

[0078] For two sets of one-dimensional data Where m is the feature dimension, the correlation coefficient ρ can be used to represent the degree of correlation between the two sets of data, and its calculation formula is shown in equation (1):

[0079]

[0080] Where Cov(x,y) is the covariance of x and y, and D(x) and D(y) are the variances of x and y.

[0081] When the data to be analyzed is multidimensional rather than one-dimensional, the CCA method can be used to calculate the correlation between two sets of multidimensional data. For two sets of high-dimensional data... Where m is the feature dimension, p, q are the sample dimensions, and let vectors be... Make

[0082] U = a T X (2)

[0083] V = b T Y (3)

[0084] Where U and V are linear combinations of samples of X and Y. Defined as canonical variables of X and Y. In essence, vectors a and b project X and Y from a higher dimension to one dimension, and the resulting one-dimensional vectors U and V are the canonical variables.

[0085] Substituting equations (2) and (3) into equation (1), and calculating the correlation coefficients of the typical variables, we obtain expression (4):

[0086]

[0087] Therefore, the problem becomes finding vectors a and b that maximize the correlation coefficient ρ. In this case, the correlation coefficient ρ of the canonical variables represents the maximum correlation between the original data X and Y.

[0088] To avoid infinite solutions, constraints are set.

[0089] a T Cov(X,X)a=b T Cov(Y,Y)b=1 (5)

[0090] Substituting equation (5) into equation (4), the problem is further simplified to:

[0091]

[0092] sta T Cov(X,X)a=b T Cov(Y,Y)b=1 (6)

[0093] Applying the Lagrange multiplier method, we can obtain

[0094]

[0095] Taking the partial derivatives of vectors a and b and setting them to zero yields the following result.

[0096]

[0097]

[0098] Rearrange equations (8) and (9) and multiply them by a on the left respectively. T and b T We can obtain:

[0099] θ=λ=a T Cov(X,Y)b=b T Cov(Y,X)a (10)

[0100] Substituting into equation (4), we can obtain

[0101] ρ=θ=λ (11)

[0102] Rearrange equations (8) and (9) and multiply them by Cov(X,X) on the left respectively. -1 and Cov(Y,Y) -1 We can obtain:

[0103] Cov(X,X) -1 Cov(X,Y)b=θa (12)

[0104] Cov(Y,Y) -1 Cov(Y,X)a=λb (13)

[0105] Substituting equation (13) into equation (12) yields

[0106]

[0107] From equation (11), we can obtain

[0108] Cov(X,X) -1 Cov(X,Y)Cov(Y,Y) -1 Cov(Y,X)a=θ 2 a (15)

[0109] From equation (15), we can see that θ 2 and 'a' represent Cov(X,X) respectively. -1 Cov(X,Y)Cov(Y,Y) -1 The eigenvalues ​​and eigenvectors of Cov(Y,X) can be obtained, and b can be calculated from a. The correlation coefficient can be obtained by taking the square root of the eigenvalues ​​using equation (11).

[0110] Thus, the problem is solved; the maximum correlation coefficient between the canonical variables U and V is the matrix Cov(X,X). -1 Cov(X,Y)Cov(Y,Y) -1 The square root of the largest eigenvalue of Cov(Y,X) represents the maximum correlation coefficient, indicating the greatest degree of correlation between the original data X and Y.

[0111] EEGNet is a compact convolutional neural network (CNN) with strong generalization capabilities, capable of learning a wide variety of interpretable features from a range of BCI tasks. CNNs extract local features from the input data through convolutional layers. Different convolutional kernels yield different feature representations; through continuous iteration and refinement, effective feature representations are strengthened, thus achieving the goal of feature extraction. The convolution calculation formula for a convolutional layer is as follows:

[0112] x j(L) =f(∑ i∈Mj x i(L-1) *k ij(L) +b j(L) (16)

[0113] Where, x j(L) Let M be the j-th feature map in the L-th convolutional layer; * denotes convolution operation; M j It is the input feature set; b j(L) f is the j-th bias value in the L-th convolutional layer; f() is the activation function.

[0114] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A hybrid brain-computer interface system based on ultrasonic tactile sensing and spatial imagination, characterized in that: It includes an ultrasonic tactile phased array, an EEG cap, an EEG signal detection board, and a main control computer; the ultrasonic tactile phased array is connected to the main control computer via USB, the EEG cap is worn on the head, the EEG cap is connected to the analog front-end module of the EEG signal detection board, and the EEG signal detection board transmits data to the main control computer via wireless communication. The main control computer includes an ultrasonic haptic phased array task module, an EEG signal receiving module, a channel data separation module, a filtering and noise reduction module, and an intelligent decoding module. During operation, the main control computer inputs a task file containing input instructions and distributes the task file to the ultrasonic haptic phased array task module. The task module parses the input instructions, generates haptic perception instructions, and sequentially sends these instructions to the ultrasonic haptic phased array slave computer. The slave computer then sequentially emits several ultrasonic waves according to the order of the haptic perception instructions, focusing them onto the user's hand and constructing different shapes to make the hand... The system senses different shapes and tactile characteristics; the EEG cap locates and collects EEG signals that the brain senses touch and generates spatial imagination, and transmits them to the analog front-end module of the EEG signal detection board. The EEG signal detection board completes the acquisition, amplification, A / D conversion, data packaging, and wireless transmission of the EEG signals; the EEG signal receiving module of the main control computer receives the bioelectric signals acquired and converted by the EEG signal detection board. The channel data separation module, filtering and noise reduction module, and intelligent decoding module perform channel separation, filtering and noise reduction, and intelligent decoding on the original bioelectric signals, respectively, and generate corresponding output control commands according to the intelligent decoding results. The ultrasonic tactile phased array includes M×N transducers. Controlled by a main computer, the array emits M×N channels of 40kHz ultrasonic waves with different phase delays. These waves can be modulated with 1-1kHz signals and focused at any point. A time-controlled mechanism activates and deactivates the ultrasonic waves at a set frequency, allowing the hand to experience ultrasonic vibrations. The array controls the focal point to move along a given two-dimensional shape at a set speed and several designated positions, providing the hand with tactile perception of a two-dimensional shape simulated by several discrete focal points. Based on the task file input instructions, the lower-level ultrasonic tactile phased array emits focused ultrasonic waves at a specified height using a time-controlled mechanism to construct characters or regular shapes. When the hand is placed at the specified height, it can perceive the constructed characters or regular shapes and generate a spatial image. The ultrasonic tactile phased array executes the tasks according to the order of input instructions, where M and N are positive integers.

2. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 1, characterized in that: The EEG signal detection board includes an analog front-end module, a main control module, a power management module, and a Wi-Fi radio frequency module. Raw EEG signals are acquired by bioelectrodes on the EEG cap using a single-ended input method. After being filtered by a passive resistor-capacitor low-pass filter, the EEG ADC chip in the analog front-end module amplifies and samples the signal, and transmits the converted digital signal to the main control module MCU. The main control module MCU sends the packaged EEG data to the Wi-Fi radio frequency module via a UART serial port. The data is then wirelessly transmitted to the main control computer by the Wi-Fi radio frequency module.

3. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 2, characterized in that: The system software of the EEG signal detection board implements data acquisition and logic control functions, including initialization of MCU peripherals, MCU external interrupts and synchronous sampling, ADC chip register configuration, reading and packaging EEG data via SPI, and sending it from the UART interface to the Wi-Fi radio frequency module, thus completing the logic control function of the entire system.

4. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 3, characterized in that: The task file includes a combination of one or more input instructions; the input instructions are represented by Arabic numerals, with one Arabic numeral representing one input instruction; the ultrasonic haptic phased array task module has the function of defining the input instructions as corresponding characters or regular shapes, where the characters are Chinese characters or English letters, and the corresponding characters or regular shapes are called haptic sensing instructions. The task module uses a time-frequency domain encoding method to generate haptic sensing instructions; the time-frequency domain encoding method is based on the modulation frequency of the emitted ultrasonic waves and the focal scanning time interval for constructing the two-dimensional shape; the ultrasonic haptic phased array task module has the function of defining haptic sensing instructions as corresponding output instructions. During operation, the ultrasonic haptic phased array task module automatically identifies the input instructions in the task file, converts the input instructions into haptic sensing instructions of characters or regular shapes, and sends them to the ultrasonic haptic phased array lower-level computer.

5. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 4, characterized in that: The EEG signal receiving module starts the TCP server to receive and verify the raw wireless data transmitted by the EEG signal detection board, and then inputs the raw wireless data to the channel data separation module.

6. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 5, characterized in that: The channel data separation module divides the multi-channel data into their respective data channels, converts the original 16-bit raw data into decimal amplitude data, and then inputs the decimal amplitude data into the filtering and noise reduction module.

7. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 6, characterized in that: The filtering and denoising module processes the data through bandpass filtering and power frequency dip digital filter to reduce noise, and then inputs the data into the intelligent decoding module to obtain the parsed output command.

8. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 7, characterized in that: The intelligent decoding module uses a classification model to filter out key features that are more sensitive to tactile / spatial imagination from the collected multi-channel data for instruction classification. For tactile information of EEG signals detected based on the ultrasound tactile evoked paradigm, it uses feature extraction and classification methods based on canonical correlation analysis (CCA) and deep neural network model training. For spatial imagination EEG signals detected based on the ultrasound tactile evoked paradigm, it uses feature extraction based on one-to-one and one-to-many co-spatial patterns (CSP) and classification mode based on SVM. A signal fusion algorithm combining average weighting and prior weighting is used to implement a brain-computer interface decoding algorithm for input task control commands, based on classification results based on EEG signal frequency information and classification results based on spatial imagination EEG signal information.

9. The hybrid brain-computer interface system based on ultrasonic tactile sensation and spatial imagination according to claim 4, characterized in that: In the ultrasonic tactile phased array task module, an experimental paradigm is designed based on the resonant frequencies of the individual characteristics of the human hand and fingers.

Images