Brain-computer device based on augmented reality and brain networking system

Abstract

The invention provides a brain-computer device and a brain networking system based on augmented reality. The brain-computer device comprises an electroencephalogram signal collecting and processing device and augmented reality equipment. The electroencephalogram signal collecting and processing device is used for collecting and processing electroencephalogram signals; the augmented reality device is connected with the electroencephalogram signal collecting and processing device and used for displaying electroencephalogram signal visual stimulation, recognizing instructions according to the electroencephalogram signals and displaying corresponding operations. According to the brain-computer device based on augmented reality and the brain networking system, a single-person wearable brain-computer interface or a multi-person networking brain-computer system can be realized based on the augmented reality technology, and the requirements of different application scenes are met.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interfaces, and in particular to a brain-computer device and a brain networking system based on augmented reality. Background Art

[0002] Brain-machine interface (BMI) technology establishes direct communication and control between the human brain and other electronic communication devices, such as computers and mobile phones. This technology allows people to directly control machines or electronic devices by directly interacting with computers using brain activity signals, freeing their hands. BMIs can be broadly categorized into two types: active and passive.

[0003] Active brain-computer interfaces are exemplified by the widely studied motor imagery, where the user's brain actively imagines the actions they wish to perform. These imaginary signals are collected and analyzed to decode the user's intentions, which are then displayed on an electronic device. However, this approach has significant drawbacks: the considerable background noise in the human brain reduces the signal-to-noise ratio of the imaginary signal, and the low accuracy of the discrimination limits the technology's application scenarios.

[0004] Passive brain-computer interfaces are represented by solutions based on event-related potentials (ERP) and steady-state visual evoked potential (SSVEP), among which the SSVEP signal has the highest signal-to-noise ratio. Therefore, brain-computer interface technology based on SSVEP is recognized by a large number of researchers and users. SSVEP means that when the human eye is looking at a fixed-frequency stimulus, the visual cortex of the human brain will produce a continuous and stable response related to the stimulus (a signal with the same frequency or a multiple of the stimulus frequency). Compared with brain-computer interface technologies based on other signals (such as P300 and motor imagery), brain-computer interfaces based on SSVEP have significant advantages such as higher information transmission rate, higher signal-to-noise ratio, simpler system and experimental design, and less training required.

[0005] Currently, SSVEP-based brain-computer interface (BCI) technology, a widely studied area of ​​research, primarily presents visual stimulation via a flat-panel display. Each circular target on the screen represents a different command (using typing as an example), and each circular target flickers at a different frequency. The system compares the user's EEG waveform with the corresponding frequency waveform, then matches the waveform to the corresponding frequency. Once the frequency is detected, the system executes the command corresponding to the circular target (printing the corresponding letter or symbol or performing the corresponding operation). Current mainstream SSVEP research relies on stable visual stimulation to induce a passive brain response, thereby controlling electronic devices to respond to the corresponding flickering stimulus. Experimental subjects' responses indicate that prolonged viewing of these large flickering areas can cause significant visual fatigue. Furthermore, prolonged visual flickering stimulation is also a contributing factor to epilepsy. Therefore, this type of SSVEP-based visual stimulation is only a transitional approach to passive BCIs.

[0006] Existing EEG signal analysis algorithms for passive visual stimulation like SSVEP are represented by Canonical Correlation Analysis (CCA) and Task-Related Component Analysis (TRCA). CCA is one of the algorithms that may be used in the end, as it does not require prior training before using a brain-computer interface device. While the TRCA algorithm requires a small amount of pre-training, its high recognition accuracy suggests that this approach also has great potential for future application. Summary of the Invention

[0007] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a brain-computer device and brain networking system based on augmented reality, which can realize a single-person wearable brain-computer interface or a multi-person networked brain-computer system based on augmented reality technology to meet the needs of different application scenarios.

[0008] In a first aspect, the present invention provides a brain-computer device based on augmented reality, comprising an EEG signal acquisition and processing device and an augmented reality device; the EEG signal acquisition and processing device is used to acquire and process EEG signals; the augmented reality device is connected to the EEG signal acquisition and processing device, and is used to display EEG signal visual stimulation, and recognize instructions according to the EEG signals and display corresponding operations.

[0009] In an implementation of the first aspect, the EEG signal acquisition and processing device includes an EEG signal acquisition device and an EEG signal processing device;

[0010] The EEG signal acquisition device includes electrodes and an electrode cap, which are used to acquire the original EEG signals generated by the subject;

[0011] The EEG signal processing device is used to amplify and digitize the original EEG signal to obtain the EEG signal.

[0012] In an implementation of the first aspect, the electroencephalogram signal processing device communicates with the augmented reality device via a wired or wireless manner.

[0013] In an implementation of the first aspect, the visual stimulation is SSVEP visual stimulation or P300 visual stimulation.

[0014] In an implementation of the first aspect, the visual stimulation includes visual stimulation for indicating instructions and visual stimulation for confirming instructions, so as to generate electroencephalogram signals of instruction selection and instruction confirmation.

[0015] In a second aspect, the present invention provides an augmented reality-based brain networking system, comprising a plurality of the aforementioned augmented reality-based brain-computer devices and wireless ad hoc network communication devices corresponding one-to-one to the augmented display-based brain-computer devices;

[0016] The wireless ad hoc network communication device is arranged on the corresponding augmented reality device, and is used to realize wireless communication between the augmented reality devices.

[0017] In an implementation manner of the second aspect, the wireless ad hoc network communication device adopts a Zigbee module or a Bluetooth module.

[0018] In an implementation manner of the second aspect, the wireless ad hoc network communication device is connected to the augmented reality device through a USB interface.

[0019] In an implementation of the second aspect, the augmented reality device is further used to display device status information, communication information, and instruction recognition results.

[0020] In an implementation of the second aspect, the augmented reality devices are provided with respective identity identification information, and the wireless ad hoc network communication device transmits instructions to the corresponding augmented reality device or transmits instructions to each augmented display device according to the identity identification information.

[0021] As described above, the augmented reality-based brain-computer interface and brain networking system of the present invention have the following beneficial effects:

[0022] (1) A single-person wearable brain-computer interface (BCI) device can be constructed based on an AR all-in-one machine and a BCI, and a multi-person BCI system can be realized based on multiple single-person wearable BCI devices and wireless communication modules.

[0023] (2) Expanding the application of brain-computer interface from desktop to mobile terminal, and from single-person experimental detection to multi-terminal system interconnection, and providing an integrated system interface solution and algorithm local deployment solution, which enhances the portability and practicality of brain-computer interface technology and broadens the application scenarios, such as meeting the silent communication needs in specific scenarios such as special operations;

[0024] (3) It does not need to rely on external networks or servers to complete its functions, and can independently perform real-time signal acquisition, target command recognition, and communication interconnection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 FIG2 is a schematic structural diagram of an augmented reality-based brain-computer interface device according to an embodiment of the present invention;

[0026] Figure 2 FIG2 is a schematic diagram showing a framework of an augmented reality-based brain-computer system according to an embodiment of the present invention;

[0027] Figure 3 Schematic diagram of the interface of an augmented display device in an augmented reality-based brain-computer system of the present invention in one embodiment. DETAILED DESCRIPTION

[0028] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0029] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0030] The development of augmented reality (AR) technology has greatly promoted the progress of human-computer interaction systems, and its combination with brain-computer interaction technology has also received widespread attention. By providing visual stimulation, EEG analysis and processing, and outputting external commands in a wearable manner, users can unexpectedly obtain new interaction paths and methods through gestures, voice, and keystrokes, thereby enabling a concealed silent communication method in specific scenarios without interfering with other actions that users may need to perform with both hands. In terms of mobile communication protocols, Zigbee is a low-power wireless communication protocol designed to provide wireless communication and self-organizing networking functions for low-power devices. It is widely used in many fields, including smart homes, industrial automation, and the Internet of Things. The present invention can combine AR, Zigbee, and brain-computer interfaces by providing an integrated solution for AR and brain-computer interaction systems, thereby realizing a multi-person brain networking system.

[0031] Among them, the present invention provides a highly integrated and practical multi-person brain-computer interaction network system that uses a wearable AR all-in-one headset to present visual stimulation, and uses a local interface and processor to solve the real-time collection and recognition analysis of EEG signals, and uses a Zigbee module to realize multi-terminal self-organizing network communication in outdoor sports scenarios, and finally realizes a brain network system with silent communication function, effectively improving the practicality of brain-computer communication technology.

[0032] like Figure 1 As shown, in one embodiment, the brain-computer device based on augmented reality of the present invention includes an electroencephalogram signal acquisition and processing device 1 and an augmented reality device 2.

[0033] The EEG signal acquisition and processing device 1 is used to acquire and process EEG signals.

[0034] In one embodiment, the EEG signal acquisition and processing device 1 includes an EEG signal acquisition device 11 and an EEG signal processing device 12. The EEG signal acquisition device 11 includes electrodes and an electrode cap, which are used to collect the original EEG signals generated by the subject when watching the SSVEP visual stimulation. The EEG signal acquisition device 11 mainly collects EEG signals from the user's occipital lobe-related positions, and conductive gel is used in the electrodes to reduce resistance. The number of electrodes corresponds to the number of data channels of the acquisition system, and three options are available: 3 channels, 8 channels, and 16 channels. The EEG signal processing device 12 is connected to the EEG signal acquisition device 11, and is used to amplify and digitize the original EEG signals to obtain the EEG signals. The EEG signal processing device 12 includes an EEG signal acquisition and processing board and a data interface expansion board, which are used for EEG signal processing and data expansion, respectively, and communicates with the augmented reality device 2 via a wired or wireless method. In particular, the circuit board portion of the EEG signal processing device 12 is designed to be wearable and can be installed in a close-fitting storage space such as an arm bag or waist bag, or can be hung on a helmet or an AR all-in-one device.

[0035] The augmented reality device 2 is connected to the EEG signal acquisition and processing device 1 and is used to display EEG signal visual stimulation, recognize instructions based on the EEG signal, and display corresponding operations.

[0036] In one embodiment, the SSVEP visual stimulation and interactive interface are presented on the augmented reality device 2 via a graphical program. The SSVEP visual stimulation flickers at a frequency of 8-15 Hz. Based on the desired number of targets, a frequency with equal intervals within the range is selected, and a sinusoidal waveform of the corresponding frequency is generated and sampled. The sampled values ​​are then used to modulate the brightness of the stimulation unit to achieve SSVEP visual stimulation. During use, the SSVEP visual stimulation maintains a 2-second flickering-2-second non-flickering state cycle. The user needs to switch their gaze to the desired target instruction during the non-flickering period, wait for the switch to the flickering period, and then maintain their gaze until the flickering period ends and the next non-flicker period begins, thereby completing the selection of a single SSVEP instruction. During system operation, the EEG signal acquisition and processing device 1 continuously collects real-time EEG signals and transmits them to the augmented reality device 2. The program running in the augmented reality device 2 intercepts the EEG signals during each flickering period and performs recognition and analysis using canonical correlation analysis or task-related analysis algorithms. After the calculation is completed, the real-time analysis results are displayed on the augmented reality interface, allowing the user to observe the current instruction status.

[0037] It should be noted that the augmented reality device used in the present invention only needs to meet the AR perspective display function. It can be monocular or binocular AR, or it can be external glasses or an AR all-in-one machine. The AR software involved in the present invention integrates all required functions and can be easily adapted to different platforms, such as Windows, Android, iOS, Linux and other desktop or mobile platforms. In addition, the visual stimulation can also use P300 visual stimulation.

[0038] In order to meet the needs of silent communication in specific environments such as battlefields, in addition to locally integrating the brain-computer device based on augmented reality, the present invention also designs a wireless ad hoc network communication device, which mainly uses Zigbee technology to achieve multi-terminal ad hoc network mobile communication at the outdoor level of 100 meters, connecting the interactive information of multiple brain-computer devices based on augmented reality to realize the practical construction of the brain network system.

[0039] like Figure 2 As shown, in one embodiment, the augmented reality-based brain networking system of the present invention includes multiple aforementioned augmented reality-based brain-computer devices and wireless ad hoc network communication devices corresponding one-to-one to the augmented display-based brain-computer devices.

[0040] The wireless ad hoc network communication device is arranged on the corresponding augmented reality device, and is used to realize wireless communication between the augmented reality devices.

[0041] In one embodiment, the wireless ad hoc network communication device utilizes a Zigbee module or a Bluetooth module. The Zigbee module will be used as an example for illustration. The wireless ad hoc network communication device utilizes a universal USB interface protocol to serve as a slave module for the augmented reality device. Its specific operating mode can be configured through either firmware hardware configuration or host software configuration. It receives information or instructions from the augmented reality device and asynchronously transmits them to other devices. It also asynchronously receives instructions from other devices and uploads them to the augmented reality device for information display. The Zigbee module can be configured in coordinator or terminal node mode. In practice, at least one coordinator and one terminal node are required to implement multicast networking. Once established, other nodes can directly join. The Zigbee module, configured through firmware hardware configuration, enables plug-and-play. Software configuration can also be used to set identity flags for different augmented reality devices, determine whether certain instructions are visible to a specific augmented reality device, or enable on-demand communication between augmented reality devices. Furthermore, multicast communication can also be performed between augmented reality devices. That is, instructions issued by one augmented reality device can be received by all other augmented reality devices.

[0042] In the present invention, the workflow of the augmented reality (AR)-based brain-computer system primarily consists of two phases: initialization and command communication. The initialization phase primarily involves the deployment of each AR device and coordinator node, and module power-up. The AR module verifies that the underlying EEG signal acquisition and processing device and Zigbee module are connected and functioning properly. If all downstream modules are online, EEG signal streaming and networking begin. The command system then formally begins operation through mechanisms such as delayed automatic wakeup or gesture-activated wakeup. During the command communication phase, each target corresponds to a command. The user acquires the command status within a certain period by gazing at the flashing SSVEP visual stimulus. To avoid errors and other unexpected situations, the present invention also provides a command confirmation mechanism, which uses two states—command content and command confirmation—to form a complete command-issuing operation. These two states correspond to two SSVEP targets. The user first identifies the first command content, confirms the correct identification through real-time feedback, then transitions to the target to confirm the command. The second command completes the confirmation and sends the current command, ensuring that the other designated AR devices receive the correct command content. In scenarios where high accuracy is required, a command confirmation mechanism can be set up. In low-latency scenarios, a direct command output mode can be set up to meet possible continuous multi-command communication needs.

[0043] It should be noted that, in addition to displaying visual stimuli, the augmented reality device 2 is also used for displaying device status information, communication information, and command recognition results. It has binocular display function, computing and processing function, and wireless communication function. It can also accept projection from external devices (such as computers, etc.), and can directly receive real-time collected EEG signals and perform computational analysis.

[0044] The following is a detailed description of the augmented reality-based brain networking system of the present invention. Figure 3As shown, in this embodiment, the augmented reality headset uses a Microsoft HoloLens 2 with a resolution of 2048×1080 and a field of view of 52°. Its core processor is a Qualcomm Snapdragon 850, running Universal Windows Platform applications. It has multiple wireless interfaces and a full-function wired USB-C port, and is powered by a built-in lithium battery. Electrodes are either gel or dry, secured directly to the augmented reality headset using custom fabric straps and Velcro. These electrodes include two reference electrodes corresponding to the frontal region of the human brain and eight signal electrodes located on the occipital region. A total of 10 electrodes are connected to the EEG signal generation device. The EEG signal processing device is a SARI-BCI system, consisting of an 8-lead signal processing board based on the ADS1299 processor and a wireless communication board based on the ESP8266 processor. It supports a sampling rate of 250Hz-1000Hz and 24-bit analog-to-digital conversion accuracy. Signal communication with the augmented reality headset is accomplished via Wi-Fi or a USB wired connection. In a specific implementation, the augmented reality head-mounted device runs a graphics program to present the visual stimulation, and receives the processed EEG signal data sent by the sari-BCI system, runs the corresponding signal processing algorithm and outputs the analysis results, which are then displayed through text on the interface. The Zigbee module is connected to the augmented reality head-mounted device in a wired manner, and can be connected to the USB interface of HoloLen2 through an interface expansion dock, and becomes a lower module of the augmented reality head-mounted device through a serial port protocol. The Zigbee module is constructed using a CC2530 radio frequency chip and a CH343 serial port chip, and can use a PCB antenna or a rod antenna. The augmented reality-based brain-computer device of the present invention only requires the user to wear an augmented reality head-mounted device and connect the Zigbee module to the corresponding interface to complete the deployment. The EEG signal processing device can also be placed in a hanging manner or an arm bag manner. For people who use the system for the first time, the display will be adjusted according to the person's actual pupil distance data through a calibration program to achieve the best use effect.

[0045] The AR-based brain-networking system consists of multiple connected AR-based brain-computer interfaces (BCIs) for individual users. Its workflow is primarily divided into two phases: initialization and command communication. During the initialization phase, each AR-based BCI device in the system is powered on. The AR device launches its core program, powers on its lower-level modules, and verifies their status via the serial port. Automated procedures then perform necessary initial configurations, primarily verifying that the BCI system is properly connected and the EEG signal stream is of good quality, and that the Zigbee module is operating normally and has entered the network. After initialization, both lower-level modules will display "Online" on the AR screen. If any issues arise, the module will display "Reconnecting" or "Offline," and the exception handling process will automatically begin. If initialization is successful, the command communication phase begins. After a default delay of 5 seconds, visual stimuli are presented, along with the current local command status, sent and received content, and network membership information.

[0046] In one embodiment, considering a team's silent communication multi-person brain network, the commander's interface is designed as follows Figure 3 As shown, it mainly includes the status of two lower modules, BCI (brain-computer interface) and Zigbee, 6 visual stimulation targets corresponding to different commands, the status of sending and receiving commands, and the status of network members. When the user selects a command, the current status will change to the number of the corresponding command after one SSVEP recognition. After the second recognition selection is confirmed, the sending area will show the actual command sent. During the overall operation, if feedback from the augmented reality device is received, the reception will display the current status. If the command is the same as the command sent by the team leader, the member status in the upper right corner will be updated, that is, whether the reply task is completed. The receiving and sending processes are both asynchronous on multiple terminals. When multiple reply commands are received at the same time, the format and font size will be automatically adjusted in the receiving area to accommodate them. The main status information will be synchronously recorded in the member status area in the upper right corner.

[0047] The present invention integrates all signal processing processes into a single augmented reality-based brain-computer device, without the need for any external server support. Considering the recognition effect and computing power limitations, the canonical correlation analysis (CCA) method is selected for recognition. The recognition algorithm is implemented by a C++ program and deployed to the UWP platform through a dynamic link library, and is directly called by the graphics program of the augmented reality device. Among them, the signal processing process mainly includes three stages. The first stage is mainly the real-time acquisition of EEG signals. The front end is amplified and digitized by hardware and uploaded to the augmented reality device, and continuously refreshed in a ring buffer corresponding to an 11000 millisecond sample data size. The second stage is the reception and preprocessing of the signal. In this stage, the subject needs to keep staring at a target for 2 seconds. The signal acquisition and processing program is based on the moment when the graphics program flashes. The latest 2000 milliseconds of data in the ring buffer are intercepted by 2130 milliseconds, and a 5-100Hz bandpass filter and a 48-52Hz notch process are performed. This can effectively suppress high-frequency components and power frequency interference. The preprocessed EEG signal enters the third stage, algorithm recognition, where it undergoes correlation analysis with a reference signal constructed using a preset SSVEP frequency template. The target with the highest correlation coefficient is the target currently being gazed at. The algorithm recognition operation primarily occurs during a period of non-blinking following the blinking period. Upon completion, the corresponding text area on the interface is displayed in real time.

[0048] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A brain-computer interface device based on augmented reality, characterized in that: Including EEG signal acquisition and processing device and augmented reality equipment; The EEG signal acquisition and processing device is used to acquire and process EEG signals; The augmented reality device is connected to the EEG signal acquisition and processing device, and is used to display EEG signal visual stimulation, and recognize instructions according to the EEG signal and display corresponding operations.

2. The augmented reality-based brain-computer device according to claim 1, characterized in that The EEG signal acquisition and processing device includes an EEG signal acquisition device and an EEG signal processing device; The EEG signal acquisition device includes electrodes and an electrode cap, which are used to acquire the original EEG signals generated by the subject; The EEG signal processing device is used to amplify and digitize the original EEG signal to obtain the EEG signal.

3. The augmented reality-based brain-computer device according to claim 2, characterized in that The electroencephalogram signal processing device communicates with the augmented reality device via a wired or wireless manner.

4. The augmented reality-based brain-computer device according to claim 1, characterized in that The visual stimulation is SSVEP visual stimulation or P300 visual stimulation.

5. The augmented reality-based brain-computer device according to claim 1, characterized in that The visual stimulation includes visual stimulation for indicating instructions and visual stimulation for confirming instructions, so as to generate electroencephalographic signals of instruction selection and instruction confirmation.

6. A brain networking system based on augmented reality, characterized in that: A method comprising: comprising a plurality of brain-computer devices based on augmented reality according to any one of claims 1 to 5 and wireless ad hoc network communication devices corresponding one-to-one to the brain-computer devices based on augmented display; The wireless ad hoc network communication device is arranged on the corresponding augmented reality device, and is used to realize wireless communication between the augmented reality devices.

7. The augmented reality-based brain networking system according to claim 6, characterized in that: The wireless ad hoc network communication device adopts a Zigbee module or a Bluetooth module.

8. The augmented reality-based brain networking system according to claim 6, characterized in that: The wireless ad hoc network communication device is connected to the augmented reality device via a USB interface.

9. The augmented reality-based brain networking system according to claim 6, characterized in that: The augmented reality device is also used to display device status information, communication information, and command recognition results.

10. The augmented reality-based brain networking system according to claim 6, characterized in that: The augmented reality devices are provided with respective identity identification information, and the wireless ad hoc network communication device transmits instructions to the corresponding augmented reality devices or transmits instructions to each augmented display device according to the identity identification information.

Images