Multi-sensing fused environment-adaptive ar-bci multimodal stimulation system

Abstract

The application discloses a multi-sensing fusion environment adaptive AR-BCI multi-modal stimulation system, and relates to the technical field of brain-computer interface and augmented reality cross, which comprises an electroencephalogram acquisition module, an augmented reality module and an external operation module; the augmented reality module comprises an ambient light sensor, an inertial measurement unit, a camera, a microphone, a visual stimulation unit, an auditory stimulation unit, a tactile stimulation unit and an adaptive control sub-module. The application realizes comprehensive and instant perception of ambient light intensity, background color distribution, ambient noise spectrum and user head movement amplitude, enables visual stimulation brightness and color matching parameters, auditory stimulation gain and frequency parameters and tactile stimulation intensity and rhythm parameters to be millisecond-level real-time adaptive adjusted according to multi-source sensing data, does not depend on the external operation module, realizes low-delay closed-loop control, and ensures the overall robustness of the system when a single stimulation mode is disturbed by the environment.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface and augmented reality technology, and in particular to an environment-adaptive AR-BCI multimodal stimulation system with multi-sensor fusion. Background Technology

[0002] Brain-computer interface (BCI) technology can convert human brain neural activity into control commands and has been widely used in medical rehabilitation, human-computer interaction, and assisted communication. Augmented reality (AR) technology can overlay virtual information onto the real world, providing users with an immersive experience. Combining BCI and AR technologies can enable hands-free human-computer interaction, which has significant application value.

[0003] Traditional BCI systems typically consist of a separate electroencephalogram (EEG) acquisition device and an external display, with visual stimuli presented via a computer screen or a separate LED array. This technology has the following drawbacks: (1) The computing devices are independent of each other and are not portable. In traditional systems, EEG acquisition, signal processing and stimulation are completed on different devices, which rely on wired or wireless connections. The overall size is huge and it is difficult to meet the needs of mobile use.

[0004] (2) Fixed visual stimulus intensity and poor environmental adaptability. The current system's visual stimulus brightness, contrast and other parameters are set during production and cannot be dynamically adjusted according to ambient lighting conditions. In bright environments (such as outdoor sunlight), the visibility of fixed-intensity visual stimuli is greatly reduced, resulting in weak or even absent brainwave responses such as steady-state visual evoked potentials, and system failure.

[0005] (3) The stimulation modality is singular and the stability is insufficient. Traditional BCI systems mainly rely on visual stimulation evoked potentials and lack the integration and utilization of multimodal stimulation methods such as auditory, tactile and motor imagery. The system stability and flexibility of use are limited in complex environments.

[0006] While current technologies include systems combining BCI and AR, such as brain-computer interface systems for augmented reality that detect biosignals and provide sensory feedback to form a closed loop, they do not address the issue of multi-environment adaptation. Stimulation parameters cannot be adjusted in real-time according to ambient light, noise, and other conditions. Furthermore, existing AR-BCI systems have shortcomings in sensor fusion, failing to fully utilize multi-source sensor data from ambient light sensors, cameras, microphones, etc., for joint optimization of stimulation parameters. Summary of the Invention

[0007] The purpose of this invention is to provide an environment-adaptive AR-BCI multimodal stimulation system with multi-sensor fusion, which solves the technical problems of poor environmental adaptability, single stimulation modality, high response delay and insufficient ease of operation of current AR-BCI systems, and realizes full-illumination environment adaptation, multimodal stimulation synergistic enhancement, local low-latency real-time control and fully automatic optimization of stimulation parameters.

[0008] To achieve the above objectives, the present invention provides a multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system, comprising: The EEG acquisition module, mounted on a wearable device, is used to collect the user's EEG signals. Augmented reality module, a head-mounted display device, is used to provide multimodal stimulation and collect environmental sensing data; An external computing module is bidirectionally connected to the augmented reality module and also communicates with the EEG acquisition module to receive EEG signals, which are used to execute EEG signal processing and sensor fusion algorithms. The augmented reality module includes: an environmental sensing sub-module, a multimodal stimulation sub-module, and a local processing unit; The environmental sensing sub-module includes: an ambient light sensor, an inertial measurement unit, a camera, and a microphone; The multimodal stimulus sub-module includes: a visual stimulation unit, an auditory stimulation unit, and a tactile stimulation unit; The local processing unit includes an adaptive control submodule; the adaptive control submodule automatically adjusts the brightness and color parameters of the visual stimulation unit, the gain and frequency parameters of the auditory stimulation unit, and the intensity and rhythm parameters of the tactile stimulation unit based on multi-source sensing data collected by the ambient light sensor, inertial measurement unit, camera, and microphone.

[0009] Preferably, the EEG acquisition module includes: Multiple brain electrodes are configured in accordance with the international 10-20 system at at least two of the O1, O2, Oz, and POz locations in the occipital lobe region of the user's head to collect EEG signals of steady-state visual evoked potentials and motor imagery. The reference electrode is positioned behind the user's ear at the mastoid process, serving as a signal reference point; The grounding electrode is positioned at the Fpz position on the user's forehead for grounding and reducing common-mode noise; The signal acquisition circuit board is used to sequentially perform preamplification, analog-to-digital conversion, data buffering and packet processing on the EEG signals acquired by multiple brain electrodes, and transmit the processed EEG signals wirelessly to an external computing module; the signal acquisition circuit board includes: a preamplifier, an analog-to-digital converter, a microcontroller, a communication module and a power management system.

[0010] Preferably, the ambient light sensor is used to detect the ambient light intensity in real time; the illuminance detection range of the ambient light sensor is 0.1~100000 lux; The inertial measurement unit is used to track the user's head movements and postures; the inertial measurement unit includes a three-axis accelerometer and a three-axis gyroscope; The camera is used to capture environmental images for background color and brightness distribution analysis; The microphone is used to measure the intensity and spectral characteristics of ambient noise.

[0011] Preferably, the visual stimulation unit is used to generate flashing visual stimuli at a specific frequency, and the plurality of stimulation targets flash at different frequencies in the range of 8 to 15 Hz. The auditory stimulation unit is used to output rhythmic sound stimulation or binaural beat stimulation; the frequency of the rhythmic sound stimulation or binaural beat stimulation is 20~50Hz; The tactile stimulation unit includes a linear actuator or a vibration motor, which is disposed at the temple of the head-mounted display device to generate regular vibration stimulation with a frequency of 50~250Hz.

[0012] Preferably, the adaptive control sub-module includes: a visual adaptive control sub-unit, an auditory adaptive control sub-unit, and a tactile adaptive control sub-unit.

[0013] Preferably, the visual adaptive control subunit is used to calculate the optimal brightness and color scheme of the visual stimulus unit based on the illuminance value output by the ambient light sensor and the background color distribution data output by the camera, thereby maximizing the contrast between the stimulus and the background.

[0014] Preferably, the auditory adaptive control subunit is used to adjust the output gain and frequency parameters of the auditory stimulation unit according to the ambient noise spectrum output by the microphone using an automatic gain control algorithm, so as to ensure that the signal-to-noise ratio of the auditory stimulation is not lower than a preset threshold.

[0015] Preferably, the tactile adaptive control subunit is used to adjust the vibration intensity of the tactile stimulation unit according to the head movement amplitude data output by the inertial measurement unit, and to increase the vibration intensity when the head movement amplitude exceeds a preset threshold.

[0016] Preferably, the adaptive control submodule stores an adaptive control algorithm; The adaptive control algorithm runs in the local processing unit of the augmented reality module. The processing of multi-source sensing data and the adjustment of parameters of the visual stimulation unit, auditory stimulation unit and tactile stimulation unit by the adaptive control algorithm are completed locally in the augmented reality module without relying on external computing modules, and are used to achieve millisecond-level response. The local processing unit is a system-on-a-chip, microcontroller, field-programmable gate array, or dedicated integrated circuit.

[0017] Preferably, the external computing module includes: The BCI algorithm unit is used to perform preprocessing, feature extraction, and classification of EEG signals. The sensor fusion algorithm unit is used to perform feature-level fusion or decision-level fusion of EEG signals, inertial data from the inertial measurement unit, illuminance data from the ambient light sensor, image data from the camera, and audio data from the microphone to generate control commands.

[0018] In summary, the multi-sensor fusion-based environmental adaptive AR-BCI multimodal stimulation system provided by this invention offers the following advantages compared to traditional technologies: Through a collaborative architecture of an EEG acquisition module, an augmented reality module, and an external computing module, an integrated AR-BCI system is constructed, combining multi-source environmental perception, multimodal adaptive stimulation, and local real-time control. By acquiring and fusing real-time data from multiple sensors such as ambient light sensors, cameras, and microphones, the system can automatically adjust the stimulation parameters of each modality based on conditions such as lighting, background color, and environmental noise. It maintains the stability of EEG evoked potentials in environments ranging from extremely dark to bright light, effectively overcoming [the limitations of traditional technologies]. Overcoming the technical limitations of traditional fixed-parameter systems failing outdoors, this system significantly improves environmental adaptability. The synergistic effect of multimodal stimuli can automatically compensate for or switch modes when a single stimulus mode is disturbed by the environment, significantly improving the overall robustness of the system. The entire adaptive control process is completed independently on the local end of the augmented reality module, without relying on external computing modules, achieving millisecond-level response and significantly reducing latency compared to traditional BCI systems that rely on external computing. Fully automated parameter optimization allows the system to intelligently adapt to the optimal stimulus scheme according to environmental changes without any manual intervention from the user, greatly improving ease of use in mobile and outdoor scenarios.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the architecture of the multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system of the present invention. Detailed Implementation

[0021] The technical method of the present invention will be further described below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0022] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0023] Techniques, systems, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the instruction manual.

[0024] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0025] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0026] This invention provides a multi-sensor fusion-based environment-adaptive AR-BCI multimodal stimulation system, such as... Figure 1 As shown, it includes: an EEG acquisition module, an augmented reality module, and an external computing module. The specific workflow is as follows: the environmental sensing sub-module of the augmented reality module collects data such as lighting, background, noise, and head movements in real time. The local processing unit adaptively adjusts the parameters of visual, auditory, and tactile stimuli based on this data. The EEG acquisition module simultaneously collects the user's EEG signals under multimodal stimulation. After processing by the external computing module, control commands are generated and fed back to the augmented reality module to execute the corresponding operations.

[0027] The EEG acquisition module is mounted on a wearable device, such as a helmet, hat, or headband, to collect the user's EEG signals and transmit them wirelessly to an external computing module. Specifically, the EEG acquisition module includes: Multiple brain electrodes, configured according to the international 10-20 system, are placed at least two of the O1, O2, Oz, and POz locations in the occipital lobe region of the user's head to collect EEG signals for steady-state visual evoked potentials and motor imagery.

[0028] The reference electrode is positioned behind the user's ear at the mastoid process as a signal reference point.

[0029] The grounding electrode, positioned at the Fpz position on the user's forehead, is used for grounding and reducing common-mode noise. Both the reference electrode and the grounding electrode can be dry or wet electrodes, and the material is unrestricted.

[0030] The signal acquisition circuit board is used to sequentially perform preamplification, analog-to-digital conversion, data buffering, and packet processing on EEG signals acquired from multiple brain electrodes, and then wirelessly transmit the processed EEG signals to an external computing module. The signal acquisition circuit board includes: a preamplifier, an analog-to-digital converter, a microcontroller, a communication module, and a power management system. The preamplifier has a gain range of 60dB~100dB and is used to amplify weak EEG signals (amplitude approximately 1μV~100μV) to a processable range; it can be integrated with the analog-to-digital converter. The analog-to-digital converter has 16-bit or 24-bit resolution and a sampling rate of at least 250Hz, and is used to convert the amplified analog signal into a digital signal. The microcontroller controls the operation of the analog-to-digital converter, data buffering and packetization, and performs basic preprocessing (such as filtering and noise reduction). The communication module uses Bluetooth or Wi-Fi protocols to wirelessly transmit EEG signal data to the external computing module. The power management system provides low-noise power and manages battery charging and discharging.

[0031] Augmented reality (AR) modules are head-mounted display devices (such as AR glasses) that interact directly with the environment to provide multimodal stimuli and collect environmental sensing data. An AR module includes an environmental sensing sub-module, a multimodal stimulation sub-module, and a local processing unit.

[0032] The environmental sensing submodule includes an ambient light sensor, an inertial measurement unit (IMU), a camera, and a microphone. The ambient light sensor detects the ambient light intensity in real time and outputs the ambient illuminance value, with a detection range of 0.1 to 100,000 lux. The IMU includes a three-axis accelerometer and a three-axis gyroscope to track the user's head movements and posture, outputting acceleration and angular velocity data. The camera acquires environmental images for background color and brightness distribution analysis to optimize stimulus color and contrast. The microphone measures ambient noise intensity and spectral characteristics for automatic gain and frequency adjustment of auditory stimuli.

[0033] The multimodal stimulus sub-modules include: visual stimulus unit, auditory stimulus unit, and tactile stimulus unit.

[0034] The visual stimulation unit generates flashing visual stimuli at specific frequencies, with multiple stimulation targets flashing at different frequencies within the range of 8 to 15 Hz. Specifically, the visual stimulation unit is a flashing pattern displayed on the AR display, used to induce steady-state visual evoked potentials.

[0035] The auditory stimulation unit is used to output rhythmic sound stimulation or binaural beat stimulation at a frequency of 20-50Hz to induce auditory steady-state response, i.e. brain waves.

[0036] The tactile stimulation unit is a linear actuator or vibration motor, which is located at the temple of the head-mounted display device to generate regular vibration stimulation with a frequency of 50~250Hz to induce steady-state somatosensory evoked potentials.

[0037] The local processing unit is a system-on-a-chip, microcontroller, field-programmable gate array, or dedicated integrated circuit, including an adaptive control submodule. The adaptive control submodule automatically adjusts the brightness and color parameters of the visual stimulation unit, the gain and frequency parameters of the auditory stimulation unit, and the intensity and rhythm parameters of the tactile stimulation unit based on multi-source sensing data collected by the ambient light sensor, inertial measurement unit, camera, and microphone.

[0038] The adaptive control sub-module includes: The visual adaptive control subunit is used to calculate the optimal brightness and color scheme of the visual stimulation unit based on the illuminance value output by the ambient light sensor and the background color distribution data output by the camera, using a contrast optimization algorithm to maximize the contrast between the stimulus and the background and ensure stable stimulus frequency.

[0039] The auditory adaptive control subunit is used to adjust the output gain and frequency parameters of the auditory stimulation unit according to the ambient noise spectrum output by the microphone, using an automatic gain control algorithm to ensure that the signal-to-noise ratio of the auditory stimulation is not lower than a preset threshold.

[0040] The tactile adaptive control subunit is used to adjust the vibration intensity of the tactile stimulation unit based on the head movement amplitude data output by the inertial measurement unit. When the head movement amplitude exceeds a preset threshold, the vibration intensity is increased.

[0041] Furthermore, the adaptive control sub-module stores the adaptive control algorithm. The adaptive control algorithm runs in the local processing unit of the augmented reality module. The processing of multi-source sensing data and the parameter adjustment of the visual stimulation unit, auditory stimulation unit and tactile stimulation unit are completed locally in the augmented reality module. It can achieve millisecond-level response without relying on external computing modules.

[0042] Specifically, the adaptive control process of the present invention is as follows: Visual adaptive control: The visual adaptive control subunit acquires the illuminance value output by the ambient light sensor and the background color distribution data output by the camera, and uses a contrast optimization algorithm to calculate the optimal brightness and color scheme of the visual stimulus unit to maximize the contrast between the stimulus and the background.

[0043] Furthermore, when the illuminance value is greater than 1000 lux (outdoor sunlight scene), the stimulation brightness is adjusted to the maximum output value and a high-contrast color scheme is adopted (such as white background with black text or black background with yellow text); when the illuminance value is between 10 lux and 1000 lux (indoor normal lighting scene), the preset standard brightness and color scheme are adopted; when the illuminance value is less than 10 lux (dark light scene), the stimulation brightness is reduced to avoid glare and a soft color scheme is adopted.

[0044] Auditory adaptive control: The auditory adaptive control subunit adjusts the output gain and frequency parameters of the auditory stimulation unit according to the ambient noise spectrum output by the microphone using an automatic gain control algorithm to ensure that the signal-to-noise ratio of the auditory stimulation is not lower than a preset threshold (such as 6dB).

[0045] Furthermore, when the ambient noise exceeds 65dB, the output intensity of the auditory stimulus is increased and the frequency is adjusted appropriately to improve perception; when the ambient noise is below 40dB(A), the intensity of the auditory stimulus is reduced to avoid discomfort.

[0046] Tactile adaptive control: The tactile adaptive control subunit calculates the head movement amplitude based on the acceleration data output by the inertial measurement unit. When the head movement amplitude exceeds a preset threshold (such as 0.5g), the vibration intensity of the tactile stimulation unit is increased to enhance the perception. When the user is stationary, the tactile stimulation intensity is reduced to save power consumption.

[0047] The external computing module has a bidirectional communication connection with the augmented reality module and also communicates with the EEG acquisition module to receive EEG signals. It is an independent computing device (such as a smartphone, tablet, or dedicated computing box) used to execute EEG signal processing and sensor fusion algorithms. The external computing module includes: The BCI algorithm unit performs preprocessing, feature extraction, and classification of EEG signals. For steady-state visual evoked potentials, canonical correlation analysis is used to extract frequency features; for motor imagery, common spatial patterns are used to extract spatial features. The classifier can employ a support vector machine or a convolutional neural network to output the user's intent recognition result.

[0048] The sensor fusion algorithm unit is used to perform feature-level fusion or decision-level fusion of EEG signals, inertial data from the inertial measurement unit, illuminance data from the ambient light sensor, image data from the camera, and audio data from the microphone to generate control commands.

[0049] The external computing module transmits the generated control commands to the augmented reality module. The augmented reality module then executes the corresponding augmented reality operations based on the control commands, such as selecting virtual buttons, controlling the movement of the virtual cursor, or triggering changes in the AR scene.

[0050] Typical application scenarios of this invention are as follows: Medical Rehabilitation: This invention's system, when placed on a patient's head, induces motor cortex activity through a combination of motor imagery and visual stimulation, assisting stroke or nerve injury patients in limb rehabilitation training. The system provides immediate visual, vibrational, or auditory feedback, enhancing patient motivation and neural plasticity. Simultaneously, by utilizing the stability and amplitude changes of steady-state visual evoked potential responses, it can assist in assessing the integrity of the patient's sensory pathways, and can be applied to brain nerve function examinations.

[0051] Assisted Communication: For patients with severe movement disorders, a virtual keyboard or communication interface can be controlled using steady-state visual evoked potentials or motor imagery signals. Patients can input text or select icons by gazing at a specific flashing target or imagining specific limb movements. The system can also be used to control smart devices such as home appliances and wheelchairs.

[0052] Smart Wearables and Human-Computer Interaction: In an AR environment, users can operate applications by looking at specific virtual buttons (visual stimulation) or by imagining their hands (kinematic visualization), achieving a hands-free, direct human-computer interaction. Embedding this system into smart glasses, helmets, or safety helmets makes it suitable for industrial operations, remote maintenance, and military applications.

[0053] Immersive Interaction and Entertainment: Combining visual, auditory, and tactile stimulation with kinesthetic imagination, a multi-channel input control method is established for VR / AR / MR game operation, enhancing immersion and real-time responsiveness. Specific AR effects or scene changes are triggered by brainwaves, applicable to interactive art exhibitions, immersive theaters, museum guided tours, and other fields.

[0054] The proposed multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system overcomes the technical shortcomings of traditional AR-BCI systems, such as poor environmental adaptability, single stimulation modality, high response latency, and insufficient operational convenience, through a collaborative architecture of EEG acquisition module, augmented reality module, and external computing module. Its main contributions are: by integrating multiple sensors such as ambient light sensor, inertial measurement unit, camera, and microphone, it achieves comprehensive and real-time perception of ambient light intensity, background color distribution, ambient noise spectrum, and user head movement amplitude; by deploying adaptive control algorithms in the local processing unit of the augmented reality module, it enables millisecond-level real-time adaptive adjustment of visual stimulus brightness and color parameters, auditory stimulus gain and frequency parameters, and tactile stimulus intensity and rhythm parameters based on multi-source sensing data, achieving low-latency closed-loop control without relying on external computing modules; and through the synergistic and complementary mechanism of multimodal stimulation, it ensures the overall robustness of the system when a single stimulation modality is affected by environmental interference.

[0055] This invention can stably maintain effective acquisition and recognition of EEG evoked potentials within a wide illumination range of 0.1 lux to 100,000 lux, expanding the applicability of the BCI system from laboratory environments to real-world scenarios such as outdoor bright light, indoor dim light, and mobile use. Simultaneously, the fully automated parameter optimization mechanism eliminates the burden of manual adjustments by the user, significantly improving the system's intelligence and ease of use. This has significant technical and economic value for promoting the deep integration of brain-computer interfaces and augmented reality technologies and their industrial applications in fields such as medical rehabilitation, assisted communication, smart wearables, and immersive interaction.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system, characterized in that, The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system includes: The EEG acquisition module, mounted on a wearable device, is used to collect the user's EEG signals. Augmented reality module, a head-mounted display device, is used to provide multimodal stimulation and collect environmental sensing data; An external computing module is bidirectionally connected to the augmented reality module and also communicates with the EEG acquisition module to receive EEG signals, which are used to execute EEG signal processing and sensor fusion algorithms. The augmented reality module includes: an environmental sensing sub-module, a multimodal stimulation sub-module, and a local processing unit; The environmental sensing sub-module includes: an ambient light sensor, an inertial measurement unit, a camera, and a microphone; The multimodal stimulus sub-module includes: a visual stimulation unit, an auditory stimulation unit, and a tactile stimulation unit; The local processing unit includes an adaptive control submodule; the adaptive control submodule automatically adjusts the brightness and color parameters of the visual stimulation unit, the gain and frequency parameters of the auditory stimulation unit, and the intensity and rhythm parameters of the tactile stimulation unit based on multi-source sensing data collected by the ambient light sensor, inertial measurement unit, camera, and microphone.

2. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 1, characterized in that, The EEG acquisition module includes: Multiple brain electrodes are configured in accordance with the international 10-20 system at at least two of the O1, O2, Oz, and POz locations in the occipital lobe region of the user's head to collect EEG signals of steady-state visual evoked potentials and motor imagery. The reference electrode is positioned behind the user's ear at the mastoid process, serving as a signal reference point; The grounding electrode is positioned at the Fpz position on the user's forehead for grounding and reducing common-mode noise; The signal acquisition circuit board is used to sequentially perform preamplification, analog-to-digital conversion, data buffering and packet processing on the EEG signals acquired by multiple brain electrodes, and transmit the processed EEG signals wirelessly to an external computing module; the signal acquisition circuit board includes: a preamplifier, an analog-to-digital converter, a microcontroller, a communication module and a power management system.

3. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 1, characterized in that, The ambient light sensor is used to detect the ambient light intensity in real time; the illuminance detection range of the ambient light sensor is 0.1~100000 lux. The inertial measurement unit is used to track the user's head movements and postures; The inertial measurement unit includes: a three-axis accelerometer and a three-axis gyroscope; The camera is used to capture environmental images for background color and brightness distribution analysis; The microphone is used to measure the intensity and spectral characteristics of ambient noise.

4. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 1, characterized in that, The visual stimulation unit is used to generate flashing visual stimuli at a specific frequency, with multiple stimulation targets flashing at different frequencies in the range of 8 to 15 Hz. The auditory stimulation unit is used to output rhythmic sound stimulation or binaural beat stimulation; the frequency of the rhythmic sound stimulation or binaural beat stimulation is 20~50Hz; The tactile stimulation unit includes a linear actuator or a vibration motor, which is disposed at the temple of the head-mounted display device to generate regular vibration stimulation with a frequency of 50~250Hz.

5. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 1, characterized in that, The adaptive control sub-module includes: a visual adaptive control sub-unit, an auditory adaptive control sub-unit, and a tactile adaptive control sub-unit.

6. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 5, characterized in that, The visual adaptive control subunit is used to calculate the optimal brightness and color scheme of the visual stimulus unit based on the illuminance value output by the ambient light sensor and the background color distribution data output by the camera, thereby maximizing the contrast between the stimulus and the background.

7. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 5, characterized in that, The auditory adaptive control subunit is used to adjust the output gain and frequency parameters of the auditory stimulation unit according to the ambient noise spectrum output by the microphone, using an automatic gain control algorithm, to ensure that the signal-to-noise ratio of the auditory stimulation is not lower than a preset threshold.

8. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 5, characterized in that, The tactile adaptive control subunit is used to adjust the vibration intensity of the tactile stimulation unit according to the head movement amplitude data output by the inertial measurement unit, and to increase the vibration intensity when the head movement amplitude exceeds a preset threshold.

9. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 1, characterized in that, The adaptive control sub-module stores an adaptive control algorithm. The adaptive control algorithm runs in the local processing unit of the augmented reality module. The processing of multi-source sensing data and the adjustment of parameters of the visual stimulation unit, auditory stimulation unit and tactile stimulation unit by the adaptive control algorithm are completed locally in the augmented reality module without relying on external computing modules, and are used to achieve millisecond-level response. The local processing unit is a system-on-a-chip, microcontroller, field-programmable gate array, or dedicated integrated circuit.

10. The multi-sensor fusion environment-adaptive AR-BCI multimodal stimulation system according to claim 1, characterized in that, The external computing module includes: The BCI algorithm unit is used to perform preprocessing, feature extraction, and classification of EEG signals. The sensor fusion algorithm unit is used to perform feature-level fusion or decision-level fusion of EEG signals, inertial data from the inertial measurement unit, illuminance data from the ambient light sensor, image data from the camera, and audio data from the microphone to generate control commands.

Images