Neural fusion interaction method and system based on multi-modal perception

Abstract

The invention discloses a neural fusion interaction method and system based on multi-modal perception, and the method comprises the steps: building a space-time synchronous triggering mechanism of staring focus staying + electroencephalogram P300 potential through the hardware cooperation of a glasses leg electrode and an eye movement camera, and solving the problems of high interaction delay and high false triggering rate in the prior art; a dynamic energy distribution strategy is adopted, and energy consumption is reduced by 70% through pulse type electroencephalogram collection; and realizing cognitive load adaptive interface control by combining theta wave power spectrum density analysis. The system realizes zero-manual precise operation in the fields of intelligent glasses, barrier-free interaction and the like, the endurance time is greater than or equal to 8 hours, and the cognitive load of a user is reduced by 40%. The method is suitable for intelligent glasses, brain-computer interface equipment and barrier-free interaction equipment.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and specifically to a multimodal interaction system that integrates electroencephalogram (EEG) signals and eye tracking, which is suitable for smart glasses, brain-computer interface devices, and barrier-free interaction devices. Background Art

[0002] Defects of existing technology: (1) Patent CN113253850A: A multi-task collaborative operation method based on eye tracking and EEG signals (Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, 2021): This patent proposes multi-task collaborative operation based on eye tracking and EEG signals, but relies on external computing devices for signal processing, which cannot meet the real-time requirements of wearable devices. The hardware integration problem of signal acquisition and interactive control is not solved; (2) Patent CN118942146A: Method and system for controlling drones with mind control based on brain imagination and eye tracking (Northwestern Polytechnical University, 2024): The drone is controlled by “imagination + eye movement”, but the user needs to actively recite the command words, which has a heavy interaction load, and EEG recognition relies on pre-trained models, which has poor individual adaptability. (3) Existing game testing solutions rely on facial expression analysis, with an error rate of >40%; SteamVR performance monitoring can only obtain hardware data such as frame rate. This invention is the first to achieve millisecond-level event-physiological response alignment, accurately quantifying subjective experience; (4) Existing relaxation devices (such as the Calm headband) need to be worn separately and have no visual coordination, resulting in user compliance of less than 50%; Patent CN119185727A is a mood disorder treatment auxiliary system and device based on virtual reality technology, which includes a head-mounted display, a VR handle, a storage base for placing the head-mounted display and the VR handle, and a cleaning structure provided on the storage base for synchronously cleaning the head-mounted display and the VR handle. The device is bulky and induces motion sickness, while the present invention achieves the same therapeutic effect on standard AI glasses; (5) Common technical defects. Spatiotemporal asynchrony: There is no millisecond-level synchronization mechanism between eye movement coordinates and EEG signals, resulting in a high false trigger rate; Energy consumption contradiction: Continuous EEG sampling leads to insufficient battery life of wearable devices (e.g., Emotiv head-mounted devices have a battery life of <4 hours); Cognitive overload: There is a lack of a neural feedback adaptive mechanism, and complex scenarios can easily cause user fatigue.

[0003] The innovative value of this invention is reflected in three aspects (1) Technological breakthrough: The first “gaze focus + EEG P300 potential” spatiotemporal synchronization triggering mechanism, achieving zero manual interaction; (2) Energy efficiency innovation: The dynamic energy distribution system increases the EEG monitoring endurance to ≥8h; (3) Cognitive optimization: Dynamically adjust information complexity through the θ wave power spectrum density, reducing cognitive load by 40%. Summary of the Invention

[0004] This invention discloses a neural fusion interaction method and system based on multimodal perception. Through hardware collaboration between temple electrodes and an eye-tracking camera, a spatiotemporal synchronization triggering mechanism for "gaze focus retention + EEG P300 potential" is established, addressing the high interaction latency and false triggering rate issues inherent in existing technologies. A dynamic energy allocation strategy reduces energy consumption by 70% through pulsed EEG acquisition. Furthermore, the system integrates theta-wave power spectral density analysis to achieve adaptive interface control for cognitive load. The system architecture comprises an 8-channel dry electrode array embedded in the temples, an integrated camera eye-tracking module, a dual-core processor: a neural signal pre-processing MCU (on the temples) and a main AI control chip (on the frame); and an energy system: a main battery (to drive AI tasks) and a thermoelectric / kinetic energy harvesting module (dedicated to EEG monitoring).

[0005] Figure 1 The core hardware architecture innovation of this invention is revealed. The temple electrode array (A) utilizes a micro-conical dry electrode design, tilted 15°-25° to fit the temple area. A biocompatible coating ensures long-term wear comfort, and the 3mm electrode spacing meets the simplified 10-20 system standard. The neural signal preprocessing MCU (B) is directly integrated into the temple, performing real-time amplification and 50Hz power frequency filtering, transmitting processed eigenvalues ​​(not raw data) to the main control AI chip (D), reducing data transmission energy consumption by 60%. The dynamic power manager (F) innovatively implements a dual-channel power supply: the thermoelectric conversion module (E) utilizes the temperature difference between the human body and the environment (ΔT ≥ 2°C) to generate 3-5mW of power, and the piezoelectric generator (G) captures walking kinetic energy (outputting 4-7mW at a cadence > 1Hz). Both are specifically powered by the EEG monitoring module; the main battery (H) is only engaged for high-power AI tasks. This architecture enables EEG monitoring to last for over 8 hours, a 300% improvement over traditional solutions (such as the NeuroSky headset), while keeping the overall weight ≤45g (comparable to consumer-grade AI glasses).

[0006] This invention proposes an original space-time synchronous interaction mechanism ( Figure 2When the user gazes at an interface element (such as an AR menu option) for longer than the threshold time T1 (default 200ms), the eye movement module records the gaze start time t1 and sends the coordinates (steps 1-2). During the gaze period, if the user generates a cognitive response due to target prominence (such as double blink confirmation), a P300 event-related potential (ERP) will be induced within 300ms±150ms after the stimulus (step 3). The EEG module detects P300 characteristics in real time: an increase of ≥30% in energy in the δ band (1-4Hz) and a decrease of ≥15% in energy in the θ band (4-8Hz) (step 4). The fusion processor verifies that the EEG event timestamp t2 satisfies the physiologically valid window of Δt=t2-t1∈[50ms,800ms] (step 5), eliminating random signal interference. After verification, the command execution is triggered (step 6), and immediate visual feedback is provided through the AR display (step 7). This mechanism compresses the interaction delay to <300ms (traditional eye movement + voice solution >800ms), reduces the false trigger rate from the industry average of 35% to below 8%, and maintains a 92% recognition accuracy rate in a 90dB noisy environment.

[0007] This invention adopts the core technology of dynamic energy management ( Figure 3 The dynamic energy management system includes five power consumption states: 1) Standby mode (10Hz EEG sampling rate) consumes only 2mW and is independently powered by the energy harvesting module; 2) Enters the eye movement activation state after detecting the gaze focus (power consumption 15mW), and the eye movement camera increases to a 120Hz sampling rate; 3) When the gaze lasts ≥T1, the EEG monitoring is awakened (power consumption 18mW), and the sampling rate is increased to 250Hz to capture P300 features; 4) Command trigger state (power consumption 50mW) performs basic operations; 5) The AI ​​high power consumption state (300-800mW) is enabled only for high-level tasks (such as AR navigation and large model inference), and is powered by the main battery. The key innovation lies in the state association constraint: the EEG monitoring module is awakened only after eye movement activation to avoid continuous high power consumption (the traditional solution has a resident power consumption of ≥50mW); the AI ​​high power consumption state is maintained for a maximum of 20 seconds before forced frequency reduction to prevent overheating. Actual measured data shows that this strategy reduces the average daily power consumption to 900mWh (traditional solutions ≥2000mWh). In a typical scenario of 30 operations per hour, the battery life is extended from 4 hours to 11 hours, completely solving the "battery anxiety" pain point of wearable devices.

[0008] Cognitive load adaptation systems such as Figure 4As shown, by continuously monitoring the power spectral density of theta waves (4-8Hz) in the prefrontal cortex (A)—the energy in this frequency band is positively correlated with cognitive load—the system assesses the user's mental state in real time. When theta wave power exceeds 40% of the baseline value for three consecutive seconds (B), it is identified as a high cognitive load state, automatically triggering three levels of optimization: 1) collapsing non-core AR information layers (e.g., hiding background data overlays), reducing visual elements by 70% (D); 2) increasing the spacing between interactive icons from 8mm to 12mm (in line with Fitts's law optimal value), reducing the visual search burden (E); and 3) activating the voice summary function, converting text information into speech output at a rate of 120 words per minute (F). When theta wave power returns to within ±10% of the baseline (H), the full interface is immediately restored (I). This mechanism has been clinically validated: in neurosurgery navigation scenarios, it reduces the error rate of doctors during high-load phases (such as vascular anastomosis) by 62%; in education, it extends students' attention span during continuous learning by 40%. The innovation lies in establishing a θ wave power-interface complexity mapping function: every 10% increase in power corresponds to a 15% reduction in information density, achieving true neural feedback closed-loop control and improving user experience satisfaction by 48% compared to traditional fixed interface solutions.

[0009] The relaxation therapy subsystem of this invention: When the θ / α index is detected to be greater than 2.5 and the blink frequency is greater than 25 times / minute, AR is used to guide the gaze focus and match the α wave induction sound frequency, increasing the visual fatigue recovery rate to 79% within 10 minutes (an increase of 108% compared to traditional solutions). Figure 1 : System hardware architecture (temple electrode distribution, dual-core processor link).

[0010] Figure 2 : Flowchart of spatiotemporal synchronization mechanism (eye movement-EEG time window matching algorithm).

[0011] Figure 3 : Dynamic energy allocation logic (switching threshold between main battery and energy collection module).

[0012] Figure 4 : Example of cognitive load adaptive interface simplification (full mode vs. simplified mode).

[0013] Figure 5 Schematic diagram of the relaxation therapy subsystem. Illustration: Closed-loop control logic of the healing process—focusing attention through visual tasks and regulating brainwave states through acoustic resonance.

[0014] Figure 6 : Schematic diagram of the test process.

[0015] Figure 7Game experience analysis flow chart. Steps AE in the figure: aligning game events with physiological data in time and space (reusing the mechanism of claim 1); Step F: diagnosing experience defects based on the mapping function of claim 6; Step GI: outputting optimization instructions that meet game development specifications. DETAILED DESCRIPTION

[0016] Example 1: Smart glasses interactive control: (1) Hardware configuration. Temple electrodes: titanium alloy micro-cone array (tilt angle 20°±5°), contact pressure 0.3N / cm²; eye tracking camera: 500Hz sampling rate, field of view 60°; energy module: frame solid-state battery (380Wh / L) + temple piezoelectric generator (walking kinetic energy → 5mW). Command trigger process: (2) The user stares at the AR menu option "Play Music" for 200ms; (3) The P300 potential (appearing 300ms after gaze onset) was detected simultaneously; (4) System verification Δt=300ms∈[50,800ms] → Execute the play command; Cognitive protection example: Theta wave power increases by 45% (lasts 4 seconds) → non-core AR information layers are automatically folded; theta waves return to normal → the fully functional interface is restored.

[0017] Example 2: Text input for ALS patients. Eye tracking of virtual keyboard letters + P300 potential confirmation → Input speed reaches 35 characters per minute (70% improvement over a pure eye tracking system);

[0018] Example 3: Neurofeedback relaxation therapy system: (1) The technical principle is based on the cognitive load monitoring framework of claim 6, and the reverse application of the theta wave control mechanism: high theta wave power (cognitive load) → triggering a relaxation protocol (rather than interface simplification), and achieving brain wave state switching (anxiety → relaxation) by guiding the user to generate high alpha waves (8-12Hz); (2) Hardware reuse configuration. Biosignal acquisition: The temple electrodes monitor the power ratio (θ / α index) of theta waves (frontal lobe) and alpha waves (occipital lobe) in real time, and the eye movement camera tracks the blink frequency (fatigue index: >20 times / minute); feedback actuator: the AR display module projects dynamic optical flow scenes, and the bone conduction headphones output dual-channel beats (resonating with the target alpha wave) The healing process is as follows Figure 5 shown.

[0019] Example 4: Gaming experience evaluation system based on neural fusion: (1) Technical Principle: Based on the spatiotemporal synchronization mechanism of claim 1 and the neural load mapping of claim 6, a quantitative model of gaming experience is constructed: real reaction capture: EEG signal → emotional valence (alpha wave asymmetry), eye tracking → attention allocation hotspot, skin conductance (reusing temple electrodes) → excitement; game event marking: event triggers are implanted in the game through the SDK (such as the appearance of the boss / plot turning point), and the event-physiological response timeline is established (accuracy ±50ms); (2) Hardware configuration (reusing the original system) (Table 1): Table 1: Game experience evaluation system hardware configuration table Module Game scene function Temple electrode array Frontal alpha wave (emotion) + skin conductance (arousal) Eye-tracking camera Gaze dwell rate / scan path analysis AR display module Real-time overlay of neural heatmaps bone conduction headphones Play test tone .

[0020] The test process is as follows Figure 6 As shown in the figure, steps AE: aligning game events with physiological data in time and space (reusing the mechanism of claim 1); step F: diagnosing experience defects based on the mapping function of claim 6; and step GI: outputting optimization instructions that meet game development specifications.

[0021] Game experience analysis flow chart Figure 7 In the figure, steps AE: aligning game events with physiological data in time and space (reusing the mechanism of claim 1); step F: diagnosing experience defects based on the mapping function of claim 6; and step GI: outputting optimization instructions that comply with game development specifications.

[0022] Example of test data results (Table 2): Table 2: Comparison of measured data Test scenario Traditional questionnaire feedback Solution of the present invention Optimization effectiveness improvement Horror Game Jump Scare 63% of players reported that it was "not exciting enough" 82% of players were detected to have skin conductance increases of <2μS Player retention after scare intensity adjustment +35% Open World Guided Missions Average task time: 8 minutes Detected 47% of players distracted >40% of the area After the guide mark is optimized, the task completion rate is increased by 58%. Competitive Game Balance Win rate 45%-55% The losing side's theta wave load was detected to be 230% higher than the winning side's. After adjusting the character skills, the balance degree reaches 49%-51% .

Claims

1. A neural fusion interaction method, characterized in that include: a) Obtaining the user’s EEG signals through the temple electrodes of the wearable device; b) Capture eye movement tracks using the device’s built-in camera; c) When the gaze focus stays for ≥ T1 and a specific EEG pattern P is detected simultaneously, the target command is triggered; d) The offset Δt between the detection time window of the specific EEG pattern P and the gaze start time satisfies the following: 50ms ≤ Δt ≤ 800ms.

2. The method according to claim 1, wherein: (1) The EEG signal processing adopts a pulse acquisition mode, including: (2) The default is low power standby mode (sampling rate ≤ 10Hz); (3) When the eye movement coordinates remain ≥T1, wake up the EEG monitoring to the full sampling rate (≥250Hz).

3. The method according to claim 1, wherein: (1) The specific EEG pattern P is the event-related potential P300, which is identified based on the change in frequency band energy: (2) Within the 300-500 ms time window, the energy of the delta band (1-4 Hz) increases by ≥30%; (3) The energy in the θ band (4-8 Hz) decreases by ≥15%.

4. A neural fusion interactive system, characterized in that include: (1) Temple electrode module: dry electrode array with biocompatible coating; (2) Eye tracking module: infrared camera and micro-electromechanical galvanometer system; (3) a processing unit configured to execute the method according to any one of claims 1 to 3; (4) Energy module: a dynamic parallel circuit of the main battery and the energy collection device (thermoelectric converter / piezoelectric generator).

5. The system according to claim 4, characterized in that: (1) The switching logic of the energy module includes: (2) When the EEG monitoring power consumption is greater than the energy harvesting output, the main battery compensation power supply is enabled; (3) When the device is in a stationary state for more than 5 minutes, turn off the main battery power supply link.

6. The system according to claim 4, characterized in that: (1) The processing unit calculates the power spectrum density of the frontal lobe theta wave in real time and executes: (2) If the theta wave power continues to rise by 40% above the baseline for 3 seconds, the information flow simplification protocol is initiated; (3) If the theta wave power falls back to the range of ±10% of the baseline, the standard information flow is restored.

Images