A multi-paradigm collaboration-based electroencephalogram signal interaction and decoding method and system
By combining visual stimulation paradigms and edge processing technology, the robustness of EEG signal recognition is enhanced, solving the problems of low recognition accuracy and high power consumption on mobile devices under complex lighting conditions, and achieving low-latency and low-power EEG interaction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHENGDU WABO TECHNOLOGY CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-06-05
Smart Images
Figure CN122152117A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of brain-computer interface and extended reality interaction technology. More specifically, it relates to a method and system for brain signal interaction and decoding using a hybrid visual stimulation paradigm in resource-constrained and complex lighting environments. Background Technology
[0002] With the popularization of augmented reality and mixed reality technologies, brain-computer interface (SCVI) technology based on steady-state visual evoked potentials (SSVEP) has gradually become an important interaction method for wearable display devices due to its lack of training requirements, large number of commands, and high information transmission rate. Traditional SSVEP systems typically display high-contrast black-and-white or color flashing squares on a computer screen to induce a synchronous response in the user's occipital cortex, which is then decoded using algorithms such as canonical correlation analysis.
[0003] However, directly transferring existing brain-computer interface technology to mobile devices such as portable AR glasses faces significant technical challenges. The first is the problem of visual interference in complex lighting environments. AR glasses typically employ optical see-through solutions, where the displayed image is superimposed on the ambient light. In outdoor bright light or complex background scenes, the contrast of the displayed image decreases significantly. In this "weakly modulated" environment, the traditional SSVEP visual stimulation paradigm elicits extremely weak EEG characteristics, which are easily drowned out by environmental noise, leading to a substantial drop in recognition accuracy.
[0004] Secondly, there is the efficiency loss caused by fixed time windows. Existing technologies mostly use data windows of fixed length for signal interception and analysis. For users or times with good signal quality, this results in unnecessary waiting time; while for situations with poor signal quality, a fixed duration may not be sufficient to accumulate enough feature information, leading to recognition failure. This rigid mechanism limits the real-time smoothness of human-computer interaction.
[0005] Finally, there are the limitations of mobile computing power and power consumption. High-performance EEG decoding algorithms are typically computationally complex, making it difficult to run in real-time on the low-power mobile processors built into AR glasses. Furthermore, transmitting high-sampling-rate raw EEG data from all channels wirelessly to the cloud or mobile phone for processing in real time would consume significant wireless bandwidth and cause a surge in power consumption of the acquisition device, shortening battery life. Therefore, there is an urgent need for an EEG interaction method and system that can maintain high robustness in low-light AR display environments while also achieving low latency and low power consumption. Summary of the Invention
[0006] The purpose of this invention is to provide a method and system for EEG signal interaction and decoding based on multi-paradigm collaboration, so as to solve the problems of low recognition rate, poor interaction efficiency in fixed time window and high power consumption of mobile data transmission in the existing technology in outdoor low light environment.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multi-paradigm collaborative EEG signal interaction and decoding method, comprising the following steps: First, generating a mixed visual stimulus signal. Based on the refresh cycle of the display terminal, a fundamental frequency brightness modulation sequence is generated using sampling sinusoidal coding technology. During real-time monitoring of the phase of this sequence, when the phase is detected to be within a preset peak range, a transient enhancement signal is superimposed on the fundamental frequency signal. This design utilizes the visual masking effect of the human eye, i.e., the human eye's sensitivity to small increments decreases against a high-brightness background, thereby enhancing the retinal potential response intensity by increasing the absolute luminous flux without increasing the user's visual flicker perception.
[0008] Secondly, edge-side signal preprocessing is performed. Through hardware circuitry integrated into the acquisition device, data from inactive channels is physically disconnected or logically discarded at the underlying level based on channel masking instructions from the host computer, retaining only the core channel data relevant to the task. Simultaneously, the edge computing unit performs hardware-level bandpass filtering and notch filtering on the retained data to remove EMG artifacts and power frequency interference, thereby significantly reducing the data bandwidth usage for wireless transmission.
[0009] Next, dual-stream feature extraction and signal-to-noise ratio (SNR) estimation are performed. Frequency domain and time domain analyses are performed in parallel on the preprocessed data to extract SSVEP frequency features and P300 waveform features, respectively. Simultaneously, the SNR of the current signal is calculated in real time and normalized to the range of 0 to 1 using a nonlinear mapping function.
[0010] Finally, dynamic decision-making based on hybrid weights is employed. The normalized signal-to-noise ratio is used as the dynamic confidence weight to weight and fuse the matching degrees in the frequency and time domains. When the fused decision metric meets the preset convergence condition, the system immediately terminates data acquisition and outputs the results without waiting for a fixed period.
[0011] In addition, the present invention also provides an EEG interaction system for implementing the above method, comprising a head-mounted display device, a distributed EEG acquisition module integrating a low-power MCU, and a host computer processing terminal.
[0012] Compared with existing technologies, this invention has the following advantages: First, it has extremely strong environmental adaptability. By using a hybrid stimulation paradigm of "sampling sine wave + peak superposition", it effectively enhances the characteristic intensity of induced EEG signals in the low-contrast environment of AR perspective mode, thereby improving the robustness of target recognition.
[0013] Second, interaction efficiency is significantly improved. A dynamic stopping strategy based on hybrid SNR is introduced, enabling adaptive adjustment of the recognition time according to signal quality. Compared to the fixed time window method, this significantly shortens the average command response time and improves the interactive experience.
[0014] Third, system-level low power consumption and low bandwidth. By using edge-side channel masking and hardware filtering, invalid data is intercepted before it reaches the transmission link, significantly reducing the load and power consumption of wireless communication and extending device battery life. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0016] Figure 1 This is a schematic diagram of the overall architecture of the AR brain-computer interface system based on multi-paradigm collaboration provided in an embodiment of the present invention.
[0017] Figure 2 This is a flowchart of the EEG signal interaction and decoding method provided in the embodiments of the present invention.
[0018] Figure 3 This is a waveform diagram of the hybrid stimulation paradigm of "sampling sine wave + peak superposition" in an embodiment of the present invention.
[0019] Figure 4 This is a hardware preprocessing logic block diagram of the distributed acquisition module in this embodiment of the invention.
[0020] Figure 5 This is a schematic diagram of the dynamic stop decision curve based on hybrid SNR in an embodiment of the present invention.
[0021] Figure labeling: 100 - Visual presentation module (AR glasses); 200 - Biosensor module; 300 - Processing terminal Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0023] Example 1: System Hardware Architecture like Figure 1 As shown, the present invention provides a brainwave interaction system for resource-constrained environments, which mainly includes a head-mounted augmented reality display device 100, a distributed brainwave acquisition module 200, and a host computer processing terminal 300.
[0024] In this embodiment, the head-mounted display device 100 employs an optical see-through head-mounted display or smart glasses. The device is equipped with a high refresh rate display module for presenting smooth sequences of mixed visual stimuli. A distributed EEG acquisition module 200 is physically integrated into the wearing structure of the head-mounted device and includes several dry electrode contacts corresponding to the occipital and parietal regions of the cerebral cortex.
[0025] To address the conflict between power consumption and bandwidth in mobile devices, module 200 employs a heterogeneous edge computing architecture. For example... Figure 4 As shown, the architecture includes a low-power main control microcontroller and a coprocessor unit. Physical layer mask: A multiplexer is placed at the front end of the analog-to-digital converter. When the system is in "standby monitoring" mode, the MCU controls the MUX to close only the wake-up channel and disconnect other channels to minimize analog front-end power consumption. Data link layer mask: When the system is in "multi-target recognition" mode, the ADC collects data from all channels, but during the data packaging stage, the MCU logically discards data frames from non-task-related channels according to preset task configuration instructions, retaining only the data from the core active channels. Hardware-level filtering: The retained digital signals are processed in parallel by the coprocessor unit. Using finite impulse response (FIR) or infinite impulse response (IR) filtering algorithms, bandpass filtering of a preset frequency band is performed to remove EMG artifacts and DC drift, while 50Hz / 60Hz power frequency notch filtering is performed simultaneously.
[0026] The preprocessed data is transmitted to the host computer processing terminal 300 via a wireless communication protocol. The host computer processing terminal 300 receives the clean preprocessed data and runs the advanced decoding algorithm described below. The host computer processing terminal 300 can be the computing unit of the head-mounted device itself, or a paired mobile computing terminal (such as a smartphone or tablet). It is responsible for receiving the preprocessed EEG data and running the decoding algorithm described below.
[0027] Example 2: Mixed Visual Stimulus Generation Method like Figure 3 As shown, to address the feature overlay problem in low-light and complex background environments outdoors, this embodiment employs a hybrid encoding method of "sampling sine wave + peak overlay". This method is implemented by synthesizing image frames in real time within the rendering pipeline of the display terminal (e.g., the micro-display of AR glasses).
[0028] The first stage: The system generates a reference brightness sequence based on the screen refresh rate of the display terminal and the target stimulus frequency. To avoid high-order harmonic interference caused by traditional square wave stimulation, this embodiment uses sampling sine coding technology to generate smooth brightness changes. Specifically, for each rendered frame, the system calculates its corresponding sine phase value and maps the corresponding pixel grayscale value based on the set average brightness reference and modulation amplitude.
[0029] It is worth noting that an inverse gamma correction mechanism is introduced before the grayscale value output. Since the brightness response of existing display screens is typically non-linear, directly outputting sinusoidal grayscale values would distort the actual display brightness. By pre-executing the inverse gamma correction function, the system ensures that the actual light intensity waveform entering the human eye is a standard sine wave, thereby significantly improving the spectral purity of the SSVEP signal.
[0030] Phase Two: Instantaneous Signal Superposition for Peak Phase Locking During the continuous rendering of the aforementioned sine wave, the system monitors the current phase state in real time before each frame refresh. The system sets a preset trigger detection interval centered on the sine wave peak, which includes a certain phase tolerance range. When the system detects that the phase of the current frame falls within this peak interval, the rendering engine will trigger an instantaneous enhancement mechanism.
[0031] Specifically, the instantaneous enhancement mechanism overlays a visual abrupt change layer on top of the existing sinusoidal brightness texture. This abrupt change layer can be a striking border appearing around the target, a geometric shape appearing at the center of the target, or a momentary inversion of the target's color tone. This instantaneous enhancement signal lasts for several refresh frames and then immediately disappears, reverting to a pure sinusoidal display.
[0032] Phase Three: Physiological Optimization Based on Visual Masking Effect. This embodiment chooses to superimpose the signal at the "peak" of the sine wave based on the physiological principle of Weber's Law. At the peak, the background brightness of the stimulus target reaches its highest value. At this moment, the human eye's perception threshold for brightness increments also increases. By superimposing a transient signal at this moment, this invention achieves a "subthreshold enhancement" effect: the user subjectively perceives only a smooth, breathing light-like flicker, but the absolute luminous flux received by the retina is superimposed with additional pulse energy at the peak. This physical energy mutation is sufficient to induce significant P300 event-related potentials in the parietal cortex of the brain, thereby upgrading the simple SSVEP paradigm to a "SSVEP+P300" dual-feature synergistic paradigm without compromising user experience.
[0033] Example 3: Dynamic Decoding Algorithm Based on Hybrid SNR like Figure 2 and Figure 5 As shown, the decoding process does not rely on a fixed time window, but uses a sliding window for real-time decision-making.
[0034] First, signal preprocessing and signal-to-noise ratio (SNR) estimation are performed. The received EEG signal is bandpass filtered through a preset frequency band. The system calculates the estimated SNR value for the current window in real time. To prevent high SNR from dominating the decision, this invention introduces a nonlinear function to normalize the original SNR. This mapping function includes a slope adjustment coefficient and a center offset parameter, which can smoothly compress the wide-range fluctuating original SNR value into a closed interval of 0 to 1, serving as a confidence index of the current signal quality.
[0035] Secondly, dual-stream feature extraction is performed. For the SSVEP stream, a frequency domain correlation analysis algorithm is used to calculate the correlation coefficient between the signal and the reference signal to obtain the first matching degree. For the P300 stream, the time-domain waveform within a specific time window after the superposition time is extracted and matched with the standard P300 template to obtain the second matching degree.
[0036] Next, dynamic fusion decision-making is performed. The system calculates a hybrid decision index using a weighted summation method: the normalized signal-to-noise ratio (SNR) is used as the weight for the first matching degree, while "1 minus the normalized SNR" is used as the weight for the second matching degree. That is, when the ambient noise is low, the system mainly relies on frequency domain features to achieve rapid identification; when the ambient interference is strong, the system automatically reduces the frequency domain weight and increases the time domain weight, utilizing the noise resistance of the time domain waveform to maintain the recognition rate.
[0037] Finally, the Bayesian termination is executed. The aforementioned hybrid decision index is input into the classifier to calculate the posterior probability of the target belonging to each candidate category. When the maximum posterior probability exceeds a preset threshold for several consecutive time windows, the system is considered to have converged and an instruction is immediately output. This method enables the system to respond quickly when the signal quality is good, while extending the acquisition time to ensure accuracy when the signal quality is poor.
[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for EEG signal interaction and decoding based on multi-paradigm collaboration, characterized in that, Includes the following steps: S1: Generate a fundamental frequency brightness modulation sequence with a specific frequency and phase according to the refresh cycle of the display terminal; The phase change of the fundamental frequency brightness modulation sequence is monitored in real time. When the phase is detected to be in a preset peak range, an instantaneous enhancement signal is superimposed on the fundamental frequency brightness modulation sequence to form a hybrid visual stimulus sequence and output to the display terminal. S2: The raw EEG signals generated by the user in response to the mixed visual stimulation sequence are acquired through the bioelectric signal acquisition unit; according to the preset task configuration instructions, channel masking is performed at the bottom layer of the signal acquisition link to retain only the signal data of the active channel, and the signal data is subjected to hardware-level frequency domain filtering to output preprocessed data; S3: Perform frequency domain feature extraction and time domain feature extraction on the preprocessed data in parallel, calculate the first matching degree corresponding to the fundamental frequency brightness modulation sequence and the second matching degree corresponding to the instantaneous enhancement signal, respectively; at the same time, calculate the signal-to-noise ratio estimate of the signal within the current time window in real time, and normalize the signal-to-noise ratio estimate. S4: Using the normalized signal-to-noise ratio estimate as a dynamic weight, the first matching degree and the second matching degree are weighted and fused to generate a real-time decision index; when the real-time decision index meets the preset convergence condition, the signal acquisition is terminated and the target recognition result is output.
2. The method according to claim 1, characterized in that, The step of generating the fundamental frequency luminance modulation sequence specifically includes: Based on the screen refresh rate of the display terminal frequency of target stimulus Calculate the discrete phase points corresponding to each refresh frame; utilize sampling sine coding technology to adjust the grayscale values of screen pixels. Fitting a sine waveform, the grayscale value Satisfying the formula: in, Average brightness For modulation amplitude, The initial phase; and the grayscale value Gamma correction is applied before output to linearize the brightness response of the screen display.
3. The method according to claim 1, characterized in that, The step of superimposing an instantaneous enhancement signal onto the fundamental frequency brightness modulation sequence when the phase is detected to be within a preset peak interval specifically includes: setting the peak interval as the phase. The range, of which The preset deviation value is used; when the phase detected in real time falls within the peak range, the background brightness of the current baseband signal is calculated. Set the brightness increment of the instantaneous enhancement signal. , making the ratio Within a closed interval of 0.05 to 0.15, and the duration of the instantaneous enhancement signal is set to 1 to 3 display frame cycles; the brightness increment is... It is superimposed onto the corresponding pixel position of the display terminal.
4. The method according to claim 1, characterized in that, The steps of performing channel masking and real-time frequency domain filtering are executed using a heterogeneous edge computing architecture. Specifically, they include: using a main control microcontroller (MCU) to intercept data packets at the data link layer and discarding digital frames from inactive channels; and using a digital signal processing core (DSP) or embedded neural processing unit (NPU) to perform finite impulse response (FIR) filtering or infinite impulse response (IIR) filtering on the retained active channel data in parallel to remove electromyographic artifacts and power frequency interference.
5. The method according to claim 4, characterized in that, The step of performing hardware-level frequency domain filtering on the signal data includes: using the digital signal processing core built into the edge computing unit to perform bandpass filtering on the signal in a preset frequency band to remove electromyographic artifacts and DC drift; and simultaneously performing power frequency notch filtering to suppress power line interference.
6. The method according to claim 1, characterized in that, The step of calculating the signal-to-noise ratio estimate and performing normalization mapping specifically includes: calculating the ratio of the power spectral density of the EEG signal at the target stimulation frequency to the power spectral density of adjacent frequency bands within the current time window to obtain the original signal-to-noise ratio. Using the Sigmoid function or hyperbolic tangent function as a nonlinear mapping kernel, the original signal-to-noise ratio is mapped to... The normalized signal-to-noise ratio is obtained from the interval. .
7. The method according to claim 6, characterized in that, The specific calculation formula for generating the real-time decision index $Q(t)$ in step S4 is as follows: ,in The first matching degree is calculated using a frequency domain correlation analysis algorithm; The second matching degree is calculated using a time-domain template matching or discriminant analysis algorithm; α and β are preset confidence adjustment factors.
8. The method according to claim 1, characterized in that, The aforementioned convergence condition refers to: calculating the posterior probability of the target belonging to each candidate category based on the current real-time decision index; when the maximum posterior probability is continuous... The sliding time window exceeds a preset probability threshold. When the time is right, it is considered convergent.
9. A brain-computer interface system implementing the method of any one of claims 1 to 8, characterized in that, include: A head-mounted display device is configured to present the mixed visual stimulus sequence; a distributed EEG acquisition module is physically integrated into the contact surface of the head-mounted display device, adopts a heterogeneous computing architecture, and includes a main control microcontroller and a coprocessing unit (DSP or NPU). The main control microcontroller is used to perform channel masking filtering, and the coprocessing unit is used to perform hardware-level frequency domain filtering; and a host computer processing terminal is established with the distributed EEG acquisition module through wireless communication, configured to receive the preprocessed data and perform the dual-stream feature extraction and dynamic decision-making.