Multi-mode ear behind brain wave signal collection multi-modal fusion purification brain control terminal interaction system and method

Abstract

This invention discloses a brain-controlled terminal interaction system and method for multimodal fusion and purification of postauricular EEG signals, belonging to the field of brain-computer interfaces and wearable electronic devices. The system includes a main unit housing that conforms to the mastoid process region of the temporal bone behind the ear, EEG acquisition electrodes and a bone conduction vibration sensor disposed inside the housing, a signal conditioning module, a microcontroller, a wireless communication module, and a feedback stimulation module. The microcontroller performs frequency domain cross-correlation calculations on the EEG and vibration signals to generate a cancellation coefficient matrix, performs adaptive artifact suppression on the EEG signals, and outputs purified EEG signals. The purified EEG signals are sent to an external terminal via the wireless communication module for pattern decoding, generate control commands, and are returned via a wireless channel. This invention selects the mastoid process region of the temporal bone behind the ear as the acquisition site, which has a high signal-to-noise ratio and is comfortable to wear. Bone conduction vibration reference noise reduction improves the accuracy of remote control decoding, and closed-loop feedback stimulation realizes integrated EEG remote control of perception, decision-making, and intervention.

Description

Technical Field

[0001] This invention relates to brain-computer interfaces and wearable electronic devices, and more particularly to a brain-controlled terminal interaction system and method for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition. Background Technology

[0002] Brain-computer interface (BCI) technology enables direct information interaction between the human brain and external devices by collecting and decoding human brain electrical signals, and has broad application prospects in fields such as medical rehabilitation, intelligent control, and assisted communication. Traditional EEG acquisition devices are based on multi-channel wet electrodes. Although they have high signal quality, they require the application of conductive gel, involve cumbersome preparation, and cannot be worn for extended periods, making it difficult to meet the needs of remote control interaction in daily life.

[0003] In recent years, wearable EEG acquisition devices have gradually become a research hotspot. For example, Chinese patent application CN202210158400.X discloses a wearable single-channel universal in-ear EEG sensor, which acquires EEG signals in the ear canal by embedding powder-sintered electrodes in the base of a universal earplug. However, this scheme only acquires a single EEG signal and lacks effective means to suppress motion artifacts and electromyographic interference, resulting in a significant decrease in signal quality during facial and head movements. Chinese patent application CN110353674A discloses an earphone-type EEG signal acquisition device, which achieves wireless EEG acquisition through electrodes on the ear hook. However, this device uses electrodes in the temporalis muscle region to acquire EEG signals. The temporalis muscle region has frequent electromyographic activity, resulting in high superposition of EEG and electromyographic signals and insufficient signal-to-noise ratio.

[0004] Regarding multimodal physiological signal fusion, Chinese patent application CN202111101955.2 discloses a wearable system based on multimodal physiological data. This system collects EEG, heartbeat, and pulse signals via a smart EEG cap, smart headphones, and a smart bracelet, respectively, and performs feature-weighted fusion for mood disorder detection. However, this system employs a split-device architecture, resulting in low time synchronization accuracy between devices and lacking closed-loop feedback control. Chinese patent application CN202511586746.X discloses an ear EEG acquisition and sleep monitoring device based on conductive leather. It integrates conductive leather electrodes into the earplug to achieve EEG signal acquisition and audio-based sleep intervention. However, the acquisition site is inside the ear canal, causing discomfort with prolonged wear, and it lacks a reference noise reduction mechanism for bone conduction vibration artifacts.

[0005] Chinese patent application CN202510601365.8 discloses a low-channel EEG amplification circuit and brain-computer interface device for EEG signals behind the ear. It collects EEG signals in the hairless area behind the ear using dry electrodes and uses high common-mode rejection ratio differential amplification and right leg drive feedback technology to achieve high-fidelity signal acquisition and low-power wireless transmission. However, it does not involve artifact reference noise reduction methods based on multimodal signals, nor does it involve closed-loop feedback stimulation function based on EEG remote control.

[0006] In summary, existing wearable EEG remote control devices have the following shortcomings: First, the EEG acquisition sites are mostly selected inside the ear canal or in the temporalis muscle region. The former is uncomfortable to wear, and the latter causes serious electromyographic interference. Second, there is a lack of real-time reference noise reduction methods for motion artifacts such as bone conduction vibration, resulting in a significant decrease in decoding accuracy during users' daily activities. Third, there is a lack of closed-loop link between acquisition, decoding and feedback, making it impossible to achieve adaptive stimulation intervention. Summary of the Invention

[0007] Purpose of the invention: The purpose of this invention is to provide a brain-controlled terminal interaction system and method for multimodal fusion and purification of multi-mode postauricular EEG signal acquisition. By selecting the mastoid region of the temporal bone behind the ear as the EEG acquisition site, and combining it with a bone conduction vibration sensor, multimodal signal acquisition and reference noise reduction are achieved. A closed-loop remote control link of acquisition-decoding-feedback is constructed through wireless communication.

[0008] Technical Solution: A brain-controlled terminal interaction system and method for multimodal fusion and purification of postauricular brainwave signals, comprising: a main unit housing, the inner side of which has a skin-contact surface that conforms to the mastoid region of the temporal bone behind the ear when worn; brainwave acquisition electrodes, disposed on the skin-contact surface of the inner side of the main unit housing, including at least one pair of differential dry electrodes, for contacting the hairless skin area behind the ear to acquire raw brainwave signals; a multimodal sensor group, integrated inside the main unit housing, including a bone conduction vibration sensor, for conforming to the mastoid region of the temporal bone to acquire mechanical vibration signals of the skull generated by facial muscle activity and vascular pulsation; and a signal conditioning module, disposed inside the main unit housing, including a front-end differential amplifier, an analog filter, and an analog-to-digital converter. The front-end differential amplifier is connected to the brainwave acquisition electrodes and the bone conduction vibration sensor respectively, performing common-mode suppression amplification and anti-aliasing filtering on the analog signals. The device outputs digital signals; a microcontroller, located inside the main unit housing and connected to the signal conditioning module, receives digitized EEG signals and digitized vibration signals, performs cross-correlation calculations on the vibration signal spectrum and the EEG signal spectrum to generate a cancellation coefficient matrix, and adaptively suppresses artifact components in the EEG signal based on the cancellation coefficient matrix to output purified EEG signals; a wireless communication module, connected to the microcontroller, sends the purified EEG signals to an external terminal for pattern decoding to generate control commands; a feedback stimulation module includes a boost circuit, a constant current output circuit, and output electrodes located on the skin-contact surface inside the main unit housing. The boost circuit converts the low-voltage DC power supply of the built-in power supply into a high-voltage DC power supply for stimulation. The constant current output circuit adjusts the output current parameters according to the decoding commands returned by the external terminal, and applies microcurrent stimulation signals to the skin behind the user's ear through the output electrodes.

[0009] Furthermore, the bone conduction vibration sensor is installed in the central area of ​​the skin-contact surface inside the main unit housing, with its vibration-sensitive surface facing the mastoid bone surface of the temporal bone. When worn, it directly contacts the mastoid bone surface of the temporal bone or indirectly contacts it through an elastic thermally conductive silicone pad. The bone conduction vibration sensor is a piezoelectric accelerometer or a MEMS accelerometer, with a frequency response range covering 20Hz to 500Hz.

[0010] Furthermore, when the microcontroller performs cross-correlation operations, it performs Fast Fourier Transform on the digitized EEG signal x(n) and the digitized vibration signal v(n) to obtain X(k) and V(k), respectively, and calculates the normalized cross-correlation coefficient. Frequency components exceeding a preset threshold are marked as artifact-related components, and the EEG signal is weighted and filtered in the frequency domain using ρ(k) as the weight.

[0011] Furthermore, the boost circuit of the feedback stimulation module uses a switched capacitor voltage multiplier rectifier topology to boost the low-voltage DC to the high-voltage DC required for stimulation. The constant current output circuit adopts a Howland current pump structure, and sets the output current amplitude through a digital-to-analog converter according to the digital control word output by the microcontroller. The output current range is 0 to 500 μA. It also includes a time-division multiplexing switch, which disconnects the electrical connection between the output electrode and the skin during EEG acquisition and closes during stimulation output.

[0012] Furthermore, the wireless communication module supports Bluetooth Low Energy protocol or StarFlash protocol, and transmits the purified EEG signal through Bluetooth Low Energy data channel or StarFlash basic access mode; the wireless communication module is also compatible with HarmonyOS soft bus protocol, realizing device discovery and distributed data flow with HarmonyOS operating system terminal devices.

[0013] Furthermore, the multimodal sensor group also includes at least one of a heart rate sensor, a blood oxygen sensor, and a skin conductance sensor; the microcontroller performs time-synchronized fusion of at least one of the collected heart rate signal, blood oxygen signal, and skin conductance signal with the purified EEG signal to generate a multimodal physiological signal feature vector, which is then sent to the external terminal through the wireless communication module to assist in user intent decoding.

[0014] Furthermore, the outer side of the main unit housing is made of medical-grade titanium alloy or carbon fiber reinforced polymer, and the inner skin-contact surface is provided with a medical-grade silicone buffer layer, with a total weight not exceeding 15g; the outer side of the main unit housing is provided with a magnetic fixing structure, including a permanent magnet embedded in the housing and an elastic buckle; the housing is also provided with a power management module, including a lithium polymer battery and a power management chip, to provide independent isolated power supply for each module.

[0015] This invention also provides a brain-controlled terminal interaction method for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition using the above system, comprising the following steps: Step 1: The EEG acquisition electrodes, which are placed on the skin-contacting surface inside the main unit housing, contact the skin of the mastoid region of the temporal bone behind the ear to acquire raw EEG signals at a set sampling frequency. The EEG acquisition electrodes are at least a pair of differential dry electrodes. The raw EEG signals are amplified by the front-end differential amplifier in the signal conditioning module with common-mode rejection, then filtered by the analog filter for anti-aliasing, and finally converted into digital EEG signals by the analog-to-digital converter. Step 2: Simultaneously, a bone conduction vibration sensor integrated inside the main housing is attached to the mastoid region of the temporal bone to collect mechanical vibration signals generated by facial muscle activity and blood vessel pulsation of the skull. After amplification, filtering and analog-to-digital conversion by the signal conditioning module, a digital vibration signal is output. Step 3: The microcontroller performs fast Fourier transform on the digitized EEG signal and the digitized vibration signal to obtain the EEG spectrum and vibration spectrum, respectively, calculates the cross-correlation coefficient between the two and normalizes them to generate a frequency-related cancellation coefficient matrix. Step four: The microcontroller uses an adaptive filtering algorithm based on the cancellation coefficient matrix to suppress artifact components related to the vibration signal in the digitized EEG signal and outputs a purified EEG signal. Step 5: The purified EEG signal is sent to an external terminal via the wireless communication module; Step six: The external terminal uses a pre-trained classification model to perform pattern recognition and decoding on the purified EEG signal to generate control commands; Step 7: The control command is returned to the host housing via the wireless communication module. The feedback stimulation module adjusts the output current parameters according to the control command and applies the microcurrent stimulation signal to the user's skin behind the ear through the output electrode to achieve closed-loop remote control feedback.

[0016] Furthermore, in step six, the external terminal uses a hybrid model of convolutional neural network-long short-term memory network for pattern recognition. The convolutional neural network extracts the time-frequency spatial features of the purified EEG signal, the long short-term memory network captures the temporal dependency relationship, and the output layer uses a Softmax classifier to map to a set of control instructions. When the maximum probability value is lower than the preset confidence threshold, the control instructions are rejected and a re-acquisition instruction is sent to the host casing. Beneficial effects

[0017] (1) The mastoid process region of the temporal bone behind the ear was selected as the EEG acquisition site. This region is free of hair and the electrode contact is stable. Compared with the temporalis muscle region, the interference of electromyographic artifacts is significantly reduced, and high signal-to-noise ratio EEG signals can be obtained.

[0018] (2) The integrated bone conduction vibration sensor is used as a reference source for electromyography artifacts. The correlation between the EEG signal and the vibration signal is calculated in real time through frequency domain cross-correlation operation to generate a cancellation coefficient matrix, thereby achieving accurate reference noise reduction for bone conduction motion artifacts.

[0019] (3) A complete closed-loop remote control link was constructed, which includes EEG acquisition, signal purification, external decoding, instruction return, and feedback stimulation. The external terminal can generate adaptive control instructions based on the user's real-time brain state and perform microcurrent stimulation intervention through the feedback stimulation module.

[0020] (4) A multi-modal sensor group is used to synchronously collect physiological signals such as heart rate, blood oxygen, and skin conductance. Multi-source signal fusion assists the external terminal in decoding the user's intentions, thereby improving the classification accuracy of remote control commands and the robustness of the system. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall structure of the brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to the present invention.

[0022] Figure 2 This is a schematic diagram of the signal processing and closed-loop remote control link of the system of the present invention.

[0023] Figure 3 This is a schematic diagram of the electrode and sensor layout on the skin-contact surface inside the main housing of the present invention.

[0024] In the picture: 1. Main unit housing; 2. EEG acquisition electrodes; 3. Bone conduction vibration sensor; 4. Signal conditioning module; 5. Microcontroller; 6. Wireless communication module; 7. Feedback stimulation module; 8. Power management module; 9. Output electrodes; 10. Heart rate sensor; 11. Electrodermal conduction sensor; 12. Magnetic fixation structure. Detailed Implementation

[0025] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Example 1

[0026] like Figures 1 to 3 As shown, this embodiment provides a brain-controlled terminal interaction system and method for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition, including a host housing 1, EEG acquisition electrodes 2, bone conduction vibration sensor 3, signal conditioning module 4, microcontroller 5, wireless communication module 6, and feedback stimulation module 7.

[0027] The main unit housing 1 adopts an ergonomic curved surface design. Its outer curved surface is made of medical-grade titanium alloy, and the inner skin-contact surface is covered with a 1.5mm thick medical-grade silicone cushioning layer. The overall dimensions of the main unit housing 1 are approximately 35mm × 25mm × 12mm, and its total weight is 12g. The curvature of the main unit housing 1 is designed to fit the anatomical structure between the mastoid process of the temporal bone and the depression at the root of the zygomatic arch behind the ear, allowing the housing to naturally embed into this depression area when worn, achieving stable mechanical positioning. A neodymium iron boron permanent magnet is embedded inside the main unit housing 1 as part of the magnetic fixation structure 12. Together with the external elastic silicone buckle, the main unit housing 1 is firmly fixed behind the user's ear through the synergistic effect of magnetic attraction and elasticity.

[0028] The EEG acquisition electrode 2 is disposed on the inner skin-contact surface of the main unit housing 1, and includes a pair of Ag / AgCl differential dry electrodes, with a spacing of 15 mm between each pair of electrodes. When worn, the EEG acquisition electrode 2 contacts the hairless skin surface of the mastoid process region of the temporal bone behind the ear, acquiring the raw EEG signals from this region. Due to the thin scalp and sparse hair in the mastoid process region of the temporal bone behind the ear, the contact impedance between the electrode and the skin is low and stable, enabling the acquisition of EEG signals with a high signal-to-noise ratio.

[0029] The bone conduction vibration sensor 3 employs a MEMS triaxial accelerometer, model ADXL345, integrated into the central area of ​​the skin-contact surface inside the main housing 1, with its vibration-sensitive surface facing the mastoid process of the temporal bone. When worn, the bone conduction vibration sensor 3 indirectly contacts the mastoid process of the temporal bone through a 0.5mm thick elastic thermally conductive silicone pad, acquiring mechanical vibration signals transmitted through the skull. The bone conduction vibration sensor 3 has a sensitivity frequency response range covering 20Hz to 500Hz, a sampling frequency of 1000Hz, and can capture micro-vibrations of the bones caused by facial muscle activities such as the masticatory muscles and corrugator supercilii, as well as bone vibrations caused by the pulsation of the superficial temporal artery.

[0030] The signal conditioning module 4 is located inside the main unit housing 1 and includes a front-end differential amplifier, an analog filter, and an analog-to-digital converter (ADC). The front-end differential amplifier uses an INA333 instrumentation amplifier with a common-mode rejection ratio (CMRR) exceeding 120dB, an input impedance greater than 10GΩ, and a gain of 1000. The analog filter is a fourth-order Butterworth low-pass filter with a cutoff frequency of 250Hz for anti-aliasing filtering. The ADC uses a 24-bit Σ-Δ ADC, model ADS1292R, with a sampling frequency of 500Hz, converting the amplified and filtered analog EEG signal into a digital signal. The signal conditioning module 4 also includes a right leg drive circuit, which feeds back the body's common-mode voltage to a reference potential via the right leg drive electrode, further improving the common-mode rejection effect.

[0031] Microcontroller 5 uses a Nordic nRF52840 system-on-a-chip, with a built-in ARM Cortex-M4F processor, a main frequency of 64MHz, and a hardware floating-point unit. Microcontroller 5 is connected to signal conditioning module 4, wireless communication module 6, and feedback stimulation module 7. Microcontroller 5 receives the digitized EEG signals output from signal conditioning module 4 and the digitized vibration signals output from bone conduction vibration sensor 3, and performs the following signal processing steps: (a) After applying Hanning windows to the digitized EEG signal x(n) and the digitized vibration signal v(n) with a length of N=512 sampling points, a fast Fourier transform is performed to obtain the frequency domain representations X(k) and V(k), with a frequency resolution of 500Hz / 512≈0.98Hz; (b) Calculate the cross-power spectral density Sxv (k)=X(k)·V*(k), where V*(k) is the complex conjugate of V(k); (c) Calculate the normalized cross-correlation coefficient ρ(k) = |S xv (k)| / √(S xx (k)·S vv (k)), where S xx (k)=|X(k)| 2 For the electroencephalogram (EEG) power spectral density, S vv (k)=|V(k)| 2 The vibrational self-power spectral density; (d) Remove ρ(k) from the preset threshold ρ th The frequency component with a value of 0.6 is marked as the pseudo-trace correlation component, and ρ(k) is used as the weight of each frequency component in the cancellation coefficient matrix. (e) Weighted filtering of the digitized EEG signal in the frequency domain: Y(k)=X(k)·[1-α·ρ(k)], where α is the inhibition strength coefficient, and in this embodiment α=0.8; (f) Perform an inverse fast Fourier transform on Y(k) to obtain the time-domain purified EEG signal y(n).

[0032] The complete execution cycle of the above signal processing is 20ms, which meets the timeliness requirements of real-time remote control.

[0033] The wireless communication module 6 is integrated within the nRF52840 chip and supports the Bluetooth Low Energy 5.2 protocol. The microcontroller 5 transmits the purified EEG signal at a sampling rate of 256Hz via the Bluetooth Low Energy connection-oriented data channel to an external terminal (smartphone or dedicated receiver), achieving a data transmission rate of 2Mbps and an end-to-end latency of less than 50ms. Upon receiving the purified EEG signal, the external terminal runs a pre-trained convolutional neural network-long short-term memory hybrid model for pattern recognition and decoding.

[0034] The feedback stimulation module 7 is located inside the main unit housing 1 and includes a switched capacitor voltage multiplier rectifier boost circuit, a constant current output circuit with a Howland current pump structure, and a pair of stainless steel output electrodes 9. The boost circuit boosts the 3.3V low-voltage DC voltage of the lithium polymer battery to ±12V bipolar DC voltage. The constant current output circuit sets the amplitude of the output current according to the digital control word output by the microcontroller 5 through the 12-bit digital-to-analog converter DAC8568. The output current range is 0 to 500μA, with a resolution of 1μA. The output electrodes 9 are located on the skin-contact surface inside the main unit housing 1, outside the EEG acquisition electrodes 2, and contact the skin behind the ear when worn. The feedback stimulation module 7 also includes a time-division multiplexing circuit composed of an analog switch ADG1414. During the EEG acquisition phase, the time-division multiplexing switch disconnects the electrical connection between the output electrodes 9 and the skin to avoid interference from the stimulation current to the EEG signal acquisition; during the stimulation output phase, the time-division multiplexing switch closes the electrical connection between the output electrodes 9 and the skin, outputting a microcurrent stimulation signal. The switching period between acquisition and stimulation is 100ms, with a duty cycle of 50%.

[0035] The power management module 8 is located inside the main unit housing 1 and includes a 120mAh lithium polymer battery and a low-power power management chip, BQ25180. The power management module 8 provides independent, isolated power supply channels for the signal conditioning module 4, microcontroller 5, wireless communication module 6, and feedback stimulation module 7, with power noise coupling of each channel below -80dB. In typical operating mode, the system has a continuous operating time of 8 hours and a standby time of up to 72 hours.

[0036] In this embodiment, the system's workflow is as follows: The user wears the main unit housing 1 behind the ear, with the EEG acquisition electrode 2 and bone conduction vibration sensor 3 contacting the skin behind the ear and the mastoid bone surface of the temporal bone, respectively. After the system starts, the EEG acquisition electrode 2 continuously acquires raw EEG signals, while the bone conduction vibration sensor 3 simultaneously acquires mechanical vibration signals from the skull. The microcontroller 5 performs frequency domain cross-correlation calculations and adaptive filtering on the two signals, outputting a purified EEG signal. The purified EEG signal is sent to an external terminal via Bluetooth Low Energy, where the hybrid model decodes the EEG features into control commands. When the user's intention is to "turn on the device," the external terminal sends the corresponding decoding command back to the main unit housing 1, and the feedback stimulation module 7 outputs a 100μA microcurrent stimulation signal applied to the skin behind the ear. Simultaneously, the external terminal performs the device-turning operation, completing the closed-loop remote control. Example 2

[0037] This embodiment extends the multimodal sensor group and wireless communication protocol based on Embodiment 1.

[0038] In addition to the bone conduction vibration sensor 3, the multimodal sensor group also includes a heart rate sensor 10 and a skin conductance sensor 11. The heart rate sensor 10 is a photoplethysmography (PPG) sensor, model MAX30102, located on the skin-contacting surface inside the main unit housing 1. It includes a green LED (wavelength 530nm) and a photodetector. When worn, the green LED emits light that illuminates the skin tissue behind the ear, and the photodetector detects the periodic changes in the reflected light intensity to collect the heart rate signal. The skin conductance sensor 11 includes a pair of stainless steel dry electrodes spaced 8mm apart, located on the skin-contacting surface inside the main unit housing 1. When worn, it contacts the skin behind the ear to collect signals indicating changes in skin conductivity. The skin conductance sensor 11 is powered by a 0.5V DC voltage applied through a constant voltage source, and the change in current flowing through the skin reflects the skin conductivity level.

[0039] Microcontroller 5 performs time-synchronized fusion of heart rate signals, skin conductance signals, and purified EEG signals. Specifically, microcontroller 5 uses the sampling time point of the purified EEG signal as a reference to perform linear interpolation resampling on the heart rate signal and skin conductance signal, aligning the three at the same time point. Then, it extracts the time-domain and frequency-domain features of each signal: for the purified EEG signal, it extracts the power spectral density features of alpha waves (8-13Hz), beta waves (13-30Hz), and theta waves (4-8Hz); for the heart rate signal, it extracts heart rate value and heart rate variability features (RMSSD and SDNN); and for the skin conductance signal, it extracts skin conductance level and skin conductance response features. These features are concatenated into a 18-dimensional multimodal physiological signal feature vector, which is then transmitted to an external terminal via wireless communication module 6.

[0040] After receiving multimodal physiological signal feature vectors, the external terminal inputs them into a pre-trained graph attention network classification model. This model uses EEG features, heart rate features, and skin conductance features as graph nodes, learning the association weights between features through a graph attention mechanism, and outputting the probability distribution of control commands. When multimodal fusion is used, the classification accuracy of control commands reaches 92.3%, an improvement of 6.6 percentage points compared to 85.7% when only EEG signals are used.

[0041] In this embodiment, the wireless communication module 6 uses the NearLink protocol instead of the Bluetooth Low Energy protocol. The NearLink protocol supports two access modes: Basic Access Mode (SE, SparkLink Enhanced) and SLE (SparkLink Low Energy) low-power access mode. This embodiment uses the SLE basic access mode as the main data transmission channel between the host housing 1 and the external terminal. In the SLE basic access mode, the wireless connection establishment process between the host housing 1 and the external terminal is as follows: The host housing 1, as a slave device, sends an access request response frame at a periodic broadcast interval on a specified SparkLink frequency, with the broadcast interval configured to be 10ms; the external terminal, as a master device, scans for the broadcast and sends a connection request frame (CONNECT_REQ), which carries the address and connection parameters of the master device; after both parties complete the link layer handshake, an SLE connection channel is established, and the total time from receiving the broadcast to the completion of the connection establishment is less than 5ms. The SLE basic access mode adopts a point-to-point communication architecture, supports a maximum payload of 251 bytes, and the air time of a single packet transmission is less than 1ms, meeting the real-time transmission requirements of EEG signal data packets.

[0042] In the SLE basic access mode, the StarScan protocol uses a Time Division Multiple Access (TDMA) mechanism for time slot allocation. The master device (external terminal) divides each superframe into multiple time slots. Uplink time slots are allocated to the slave device (host housing 1) for sending cleaned EEG data packets, while downlink time slots are allocated to the master device for sending control commands and acknowledgment frames (ACKs). Each time slot lasts 125 μs, the superframe period is 1 ms, and each superframe contains 8 time slots, with 2 uplink time slots (Slot #2 and Slot #5) and 1 downlink time slot (Slot #7) allocated to this system. The physical layer uses GFSK (Gaussian Frequency Shift Keying) modulation in the 2.4 GHz ISM band (2400 MHz to 2483.5 MHz), with a modulation index of 0.5, a symbol rate of 1 Msps, and an over-the-air transmission rate of 2 Mbps. GFSK modulation smooths the digital baseband signal using a Gaussian filter, achieving a 3dB bandwidth of 0.5MHz, effectively reducing spectral sidelobe leakage and minimizing interference from adjacent channels. Channel selection employs a frequency-hopping mechanism, switching the operating channel in each superframe according to a pseudo-random sequence with a 1MHz hopping interval to enhance anti-interference capabilities.

[0043] The StarSpark protocol's data packet structure comprises four parts: frame header, timestamp, data payload, and CRC checksum. The frame header is 4 bytes long and includes a preamble (an 80-bit alternating sequence of 0s and 1s used for receiver clock synchronization), an access address (a 32-bit unique identifier for the current connection), and packet header control information (PCI, containing packet type identifier, sequence number, length indicator, and flow control flag). The PCI field identifies the packet type as a data frame, acknowledgment frame, or control frame. The timestamp field is 4 bytes long with a precision of 1 μs, recording the precise transmission time of the data packet at the sending end. The receiving end uses this timestamp to achieve microsecond-level time synchronization between the host casing 1 and the external terminal, ensuring that the time alignment accuracy of multimodal signal acquisition is better than 10 μs. The data payload is a maximum of 251 bytes. In this embodiment, each data packet carries 16-bit purified EEG signal data from 125 sampling points (125 × 2 = 250 bytes), transmitting approximately 2048 data packets per second at a sampling rate of 256 Hz. The CRC check uses a 32-bit CRC-32 checksum (polynomial 0x04C11DB7) to verify the integrity of the frame header and data payload. If the check fails, the receiver requests retransmission in the next downlink time slot, with a maximum of 3 retransmissions per packet. Based on the above TDMA time slot allocation and packet structure, the end-to-end transmission delay from the completion of EEG signal acquisition in the host casing 1 to the external terminal's reception and decoding is less than 20ms (of which the air interface transmission delay is less than 5ms, the protocol stack processing delay is less than 10ms, and the application layer queuing delay is less than 5ms), meeting the timeliness requirements of real-time remote control.

[0044] When the external terminal is running the HarmonyOS operating system, the wireless communication module 6 is also compatible with the HarmonyOS Distributed Soft Bus protocol, which enables automatic device discovery and distributed data transfer. The device discovery process of HarmonyOS SoftBus is as follows: After the host shell 1 connects to the HarmonyOS ecosystem, it first registers its device capability description (Device Capability Descriptor) through the SLE GATT service of the Starlink. This description includes the device type identifier (brainwave_wearable), a list of supported sensor types (EEG, HR, EDA, IMU), data output format, and sampling rate parameters. Subsequently, the host shell 1 sends a CoAP (Constrained Application Protocol) multicast message to the local area network through the HarmonyOS SoftBus device discovery agent. The destination address of the broadcast message is the IPv6 multicast address of the HarmonyOS device group, and the message body adopts CBOR encoding format, containing the device ID, device name, and device capability description summary. After receiving the CoAP broadcast message, the HarmonyOS SoftBus Daemon running on nearby HarmonyOS devices (such as HarmonyOS phones, tablets, smart screens, etc.) parses the device capability description and matches it with the application requirements registered on the device. After a successful match, the host shell 1 publishes its complete device profile to the HarmonyOS SoftBus Distributed Device Management service. In the Service, the device profile includes information such as device hardware version, firmware version, sensor calibration parameters, and data encryption method. Through the CoAP broadcast and profile publishing mechanism described above, users can automatically discover and identify the host casing 1 and HarmonyOS devices without manual pairing, with a device discovery latency of less than 2 seconds.

[0045] After device discovery is completed, the connection establishment and data flow process of HarmonyOS soft bus is as follows: The application on the external terminal initiates a device connection request through the DiscoverDevice interface of HarmonyOS soft bus. The request parameters carry the target device's Profile ID and the expected service type (DataStream). After receiving the connection request, host shell 1 creates a GATT data service channel on the established StarSpark SLE link. This channel defines two characteristics: the uplink characteristic (UUID: 0xEEG01) is used by host shell 1 to send purified EEG signals and multimodal physiological signal data to the external terminal, and the downlink characteristic (UUID: 0xEEG02) is used by the external terminal to send control commands and parameter configuration commands to host shell 1. After the connection is established, the external terminal sends a data acquisition start command to the host casing 1 via the SendMsg interface of the HarmonyOS soft bus. The microcontroller 5 of the host casing 1 encapsulates the purified EEG signals and multimodal physiological signals into standardized data packets according to the HarmonyOS IDL (Interface Definition Language) interface specification. The data packets include a data header (containing a timestamp, data type identifier, and data length), a serialized signal data body, and a checksum, and are transmitted back to the external terminal via the SendMsg interface through the SLE GATT channel. At the distributed data management level, the HarmonyOS soft bus maps the sensor data of the host casing 1 into distributed data objects. Applications on the external terminal can obtain purified EEG signals and multimodal physiological signals in real time by subscribing to the onChange callback event of the data object, without needing to worry about the underlying communication protocol and data transmission details. Distributed data objects also support cross-device migration. When a user switches between a HarmonyOS phone and a HarmonyOS tablet, the data object automatically migrates to the new device, and the EEG remote control task is seamlessly continued. Through the device discovery, connection establishment, and data transfer mechanisms of the HarmonyOS soft bus, seamless collaboration between the host shell 1 and HarmonyOS ecosystem devices is achieved, allowing users to freely transfer EEG remote control tasks between multiple HarmonyOS devices.

[0046] The other structures and functions of this embodiment are the same as those of Embodiment 1. Example 3

[0047] Based on Example 1, this embodiment improves the structure and control method of the feedback stimulation module 7 and provides a multi-scenario remote control application solution.

[0048] The boost circuit of feedback stimulation module 7 adopts a three-stage switched capacitor voltage multiplier rectifier topology. Each stage includes a flying capacitor and two switching transistors with a switching frequency of 100kHz. It sequentially doubles the 3.3V input voltage to 6.6V, 9.9V, and 13.2V, and outputs a ±12V bipolar DC voltage after linear regulation, with a voltage ripple of less than 50mV. The constant current output circuit adopts a modified Howland current pump structure, including an operational amplifier OPA4227 and four precision matching resistors (matching accuracy 0.01%). The output current is set by a 12-bit digital control word output from microcontroller 5 via a digital-to-analog converter. The output current waveform supports three modes: square wave, biphase asymmetrical square wave, and sine wave, with a frequency range of 1Hz to 100Hz and a pulse width range of 50μs to 500ms.

[0049] When decoding control commands, the external terminal generates different stimulation parameters based on the type of user intent. Specifically, when the control command is of the "attention enhancement" type, the external terminal sends the following parameter configuration to the feedback stimulation module 7: stimulation frequency 20Hz (corresponding to beta wave enhancement), output current 200μA, stimulation duration 30 seconds, and stimulation waveform is a biphasic asymmetrical square wave. When the control command is of the "relaxation induction" type, the stimulation parameters are configured as follows: stimulation frequency 10Hz (corresponding to alpha wave enhancement), output current 100μA, stimulation duration 60 seconds, and stimulation waveform is a square wave. When the control command is of the "device control" type, the external terminal directly drives the connected external device to perform the corresponding operation (such as light switch, volume adjustment, etc.), without activating the feedback stimulation module 7.

[0050] In this embodiment, the pre-trained classification model of the external terminal adopts a hybrid architecture of convolutional neural network (CNN) and long short-term memory (LSTM). The CNN part includes three convolutional layers and two pooling layers, with kernel sizes of (1,5), (1,3), and (1,3), respectively. Batch normalization and ReLU activation functions are applied after each layer. The LSTM part includes two layers of LSTM units, each containing 128 hidden units. The input of the hybrid model is a 5-second purified EEG signal segment (sampling rate 256Hz, 1280 sampling points), and the output is the probability distribution of 16 types of control commands. When the maximum probability value is lower than the preset confidence threshold of 0.75, the hybrid model refuses to output control commands and sends a re-acquisition command to the host casing 1. After receiving the re-acquisition command, the host casing 1 extends the duration of the next acquisition window to 10 seconds to improve signal quality and decoding accuracy.

[0051] The application scenarios of this embodiment include: (a) Smart home remote control: Users imagine different movement tasks (such as imagining the lights turning on with the left hand and the curtains closing with the right hand), and the system decodes the EEG pattern into corresponding device control commands to drive smart home devices to perform operations; (b) Attention assistance: The system monitors the user's attention state in real time. When it detects a decrease in attention (increased theta wave power and decreased alpha wave power), it automatically starts 20Hz microcurrent stimulation to enhance attention; (c) Rehabilitation training assistance: In the upper limb rehabilitation training of stroke patients, the system detects the patient's motor intention EEG signal, decodes it, drives the exoskeleton device to assist upper limb movement, and outputs feedback stimulation to promote neuroplasticity.

[0052] The other structures and functions of this embodiment are the same as those of Embodiment 1. Example 4

[0053] Based on Example 1, this embodiment improves the artifact suppression algorithm of the microcontroller 5 by using time-domain adaptive filtering instead of frequency-domain weighted filtering.

[0054] The microcontroller 5 employs the Normalized Least Mean Square (NLMS) algorithm for adaptive artifact suppression of the digitized EEG signal. Specifically, the digitized vibration signal v(n) is used as the reference input signal, and the digitized EEG signal x(n) is used as the main input signal. The output of the adaptive filter is y(n) = x(n) - w. T w(n)·v(n), where w(n) is a filter weight vector of length L=64. The iterative update formula for the filter weight vector is: w(n+1)=w(n)+μ·e(n)·v(n) / [v T [(n)·v(n)+δ], where e(n)=y(n) is the error signal, μ is the step size factor (μ=0.005 in this embodiment), and δ is the regularization parameter with a value of 10. -6 This is used to prevent division by zero operations.

[0055] Compared with the frequency domain weighted filtering method in Example 1, the time domain adaptive filtering method has the following advantages: (a) lower computational cost, requiring only L+1=65 multiply-accumulate operations per iteration, and can complete the filtering of 512 sampling points within 1ms on a 64MHz microcontroller 5; (b) able to track the time-varying changes in artifact characteristics, and the filter weights can be adaptively adjusted when the user's facial muscle activity pattern changes, maintaining a continuous noise reduction effect; (c) no frequency domain transformation and inverse transformation are required, avoiding the spectral leakage problem caused by the window function.

[0056] In this embodiment, the microcontroller 5 also includes an adaptive step size control mechanism. When the short-time energy of the detected digital vibration signal v(n) exceeds a preset threshold (indicating that the user is engaging in significant facial muscle activity with strong artifacts), the step size factor μ automatically increases to 0.01, accelerating the convergence speed of the filter weights; when the short-time energy of v(n) is below the threshold (indicating that the user is in a quiet state with weak artifacts), the step size factor μ decreases to 0.001, reducing the steady-state offset noise of the filter. The short-time energy is calculated using a sliding window of 200ms, updated every 100ms.

[0057] The other structures and functions of this embodiment are the same as those of Embodiment 1.

[0058] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A brain-controlled terminal interaction system and method for multimodal fusion and purification of postauricular brainwave signals, characterized in that, include: The main unit housing has a skin-contact surface formed on its inner side, which fits the mastoid region of the temporal bone behind the ear when worn. The system includes: an EEG acquisition electrode, located on the skin-contact surface inside the main housing, comprising at least one pair of differential dry electrodes for acquiring raw EEG signals by contacting the hairless skin area behind the ear; a multimodal sensor group, integrated inside the main housing, including a bone conduction vibration sensor for acquiring mechanical vibration signals generated by facial muscle activity and vascular pulsation in the mastoid process region of the temporal bone; a signal conditioning module, located inside the main housing, comprising a front-end differential amplifier, an analog filter, and an analog-to-digital converter; the front-end differential amplifier being connected to the EEG acquisition electrode and the bone conduction vibration sensor respectively, performing common-mode suppression amplification and anti-aliasing filtering on the analog signal, and the analog-to-digital converter outputting a digital signal; and a microcontroller, located inside the main housing and connected to the signal conditioning module, receiving digitized EEG signals and digitized vibration signals, performing cross-correlation calculations on the vibration signal spectrum and the EEG signal spectrum to generate a cancellation coefficient matrix, and adaptively suppressing artifact components in the EEG signal based on the cancellation coefficient matrix to output a purified EEG signal. A wireless communication module, connected to the microcontroller, sends the purified EEG signal to an external terminal for pattern decoding to generate control commands. The feedback stimulation module includes a boost circuit, a constant current output circuit, and an output electrode disposed on the skin-contact surface inside the main unit housing. The boost circuit converts the low-voltage DC power supply of the built-in power supply into a high-voltage DC power supply for stimulation. The constant current output circuit adjusts the output current parameters according to the decoding instructions returned by the external terminal, and applies a microcurrent stimulation signal to the skin behind the user's ear through the output electrode.

2. The brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 1, characterized in that, The bone conduction vibration sensor is installed in the central area of ​​the skin-contact surface inside the main unit housing, with its vibration-sensitive surface facing the mastoid bone surface of the temporal bone. When worn, it is in direct contact with the mastoid bone surface of the temporal bone or indirect contact through an elastic thermally conductive silicone pad. The bone conduction vibration sensor is a piezoelectric accelerometer or a MEMS accelerometer, and its frequency response range covers 20Hz to 500Hz.

3. The brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 1, characterized in that, When the microcontroller performs cross-correlation operations, it performs Fast Fourier Transform on the digitized EEG signal x(n) and the digitized vibration signal v(n) to obtain X(k) and V(k), respectively, and calculates the normalized cross-correlation coefficient. Frequency components exceeding a preset threshold are marked as artifact-related components, and the EEG signal is weighted and filtered in the frequency domain using ρ(k) as the weight.

4. The brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 1, characterized in that, The boost circuit of the feedback stimulation module uses a switched capacitor voltage multiplier rectifier topology to boost the low-voltage DC to the high-voltage DC required for stimulation. The constant current output circuit adopts a Howland current pump structure. The output current amplitude is set by a digital-to-analog converter according to the digital control word output by the microcontroller. The output current range is 0 to 500 μA. It also includes a time-division multiplexing switch, which disconnects the electrical connection between the output electrode and the skin during EEG acquisition and closes during stimulation output.

5. The brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 1, characterized in that, The wireless communication module supports Bluetooth Low Energy protocol or StarFlash protocol, and transmits the purified EEG signal through Bluetooth Low Energy data channel or StarFlash basic access mode; the wireless communication module is also compatible with HarmonyOS soft bus protocol, realizing device discovery and distributed data flow with HarmonyOS operating system terminal devices.

6. The brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 1, characterized in that, The multimodal sensor group also includes at least one of a heart rate sensor, a blood oxygen sensor, and a skin conductance sensor; the microcontroller performs time-synchronized fusion of at least one of the collected heart rate signal, blood oxygen signal, and skin conductance signal with the purified EEG signal to generate a multimodal physiological signal feature vector, which is then sent to the external terminal through the wireless communication module to assist in user intent decoding.

7. The brain-controlled terminal interaction system for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 1, characterized in that, The outer side of the main unit casing is made of medical-grade titanium alloy or carbon fiber reinforced polymer, and the inner skin-contact surface is provided with a medical-grade silicone buffer layer, with a total weight not exceeding 15g; the outer side of the main unit casing is provided with a magnetic fixing structure, including a permanent magnet embedded in the casing and an elastic buckle; the casing is also provided with a power management module, including a lithium polymer battery and a power management chip, to provide independent isolated power supply for each module.

8. A brain-controlled terminal interaction method for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition using the system described in any one of claims 1 to 7, characterized in that, Includes the following steps: Step 1: The EEG acquisition electrodes, which are placed on the skin-contacting surface inside the main unit housing, contact the skin of the mastoid region of the temporal bone behind the ear to acquire raw EEG signals at a set sampling frequency. The EEG acquisition electrodes are at least a pair of differential dry electrodes. The raw EEG signals are amplified by the front-end differential amplifier in the signal conditioning module with common-mode rejection, then filtered by the analog filter for anti-aliasing, and finally converted into digital EEG signals by the analog-to-digital converter. Step 2: Simultaneously, a bone conduction vibration sensor integrated inside the main housing is attached to the mastoid region of the temporal bone to collect mechanical vibration signals generated by facial muscle activity and blood vessel pulsation of the skull. After amplification, filtering and analog-to-digital conversion by the signal conditioning module, a digital vibration signal is output. Step 3: The microcontroller performs fast Fourier transform on the digitized EEG signal and the digitized vibration signal to obtain the EEG spectrum and vibration spectrum, respectively, calculates the cross-correlation coefficient between the two and normalizes them to generate a frequency-related cancellation coefficient matrix. Step four: The microcontroller uses an adaptive filtering algorithm based on the cancellation coefficient matrix to suppress artifact components related to the vibration signal in the digitized EEG signal and outputs a purified EEG signal. Step 5: The purified EEG signal is sent to an external terminal via the wireless communication module; Step six: The external terminal uses a pre-trained classification model to perform pattern recognition and decoding on the purified EEG signal to generate control commands; Step 7: The control command is returned to the host housing via the wireless communication module. The feedback stimulation module adjusts the output current parameters according to the control command and applies the microcurrent stimulation signal to the user's skin behind the ear through the output electrode to achieve closed-loop remote control feedback.

9. The brain-controlled terminal interaction method for multimodal fusion and purification of multi-mode postauricular brainwave signal acquisition according to claim 8, characterized in that, In step six, the external terminal uses a hybrid model of convolutional neural network and long short-term memory network for pattern recognition. The convolutional neural network extracts the time-frequency spatial features of the purified EEG signal, the long short-term memory network captures the temporal dependency relationship, and the output layer uses a Softmax classifier to map the control command set. When the maximum probability value is lower than the preset confidence threshold, the control command is rejected and a re-acquisition command is sent to the host casing.

Images