A multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection

Abstract

The present invention discloses a multi-modal immersive synchronous acquisition system based on eye tracking and brain function activity detection, including a VR display device, wherein the VR display device is built with a display module, an eye movement acquisition module, a cerebral blood oxygen acquisition module and a communication module, wherein the eye movement acquisition module is used to acquire eye images of a subject, and the cerebral blood oxygen acquisition module is used to acquire cerebral blood oxygen signals of the subject's head, and the VR display device and the eye movement acquisition module and the cerebral blood oxygen acquisition module respectively realize data transmission and clock synchronization with an external terminal device through a communication module. The present invention adopts a VR display device with a virtual immersive experience in the entire acquisition system, and the entire test and data acquisition process can completely avoid external interference with the subject. At the same time, the present invention takes into account the synchronous acquisition of eye movement and cerebral blood oxygen signals, and directly transmits data with an external terminal device through a communication module, and adopts a clock synchronization method to solve the synchronization problems of real-time characteristics and multi-modal signals.

Description

Technical Field

[0001] The present invention relates to the field of cognitive neuroscience technology, and in particular to a multimodal immersive synchronous acquisition device based on eye tracking-brain function activity detection. Background Art

[0002] Electroencephalogram (EEG) is the spontaneous, rhythmic electrical activity of brain cell groups recorded by electrodes. It contains a large amount of physiological and pathological information and is one of the methods for examining neurological function. EEG reflects the electrical activity and various functional states of brain tissue. Its basic characteristics include amplitude, period, phase, etc. EEG signals at specific locations can also reflect cognitive abilities. Eye tracking technology is a technology that measures the movement of the pupil of the human eye through an eye movement measurement device, and then estimates the direction and position of the person's line of sight. Eye tracking technology can be used in a variety of applications, such as in cars to detect driver fatigue; in the cognitive field, it can analyze the user's psychological activities by tracking the user's line of sight.

[0003] In the prior art, only a single technical means is used to obtain parameters related to brain function. For example, when a user browses a page to be evaluated during a test, an eye movement data acquisition device is used to collect the user's gaze position on the page to be evaluated and the gaze duration used when gazing at each gaze position. The user's gaze position can be recorded in the form of gaze coordinates or gaze contours. When browsing the page to be evaluated, there will be external interference, which affects the accuracy of brain function detection.

[0004] In order to conduct comprehensive multi-parameter detection of human cognitive nerves, there is an urgent need for a method that can combine modern EEG technology and eye tracking technology to achieve multi-type parameter collection and multi-modal detection of human brain function and cognitive function, providing more comprehensive parameters for brain function analysis. Summary of the invention

[0005] The present invention aims to solve the technical problem that interference exists when collecting data related to cognitive neural parameters, which affects the accuracy of brain function analysis and detection. The present invention provides a multi-modal immersive synchronous acquisition system based on eye tracking-brain function activity detection.

[0006] The present invention adopts the following technical solution:

[0007] A multimodal immersive synchronous acquisition system based on eye tracking-brain function activity detection, characterized in that the system includes a VR display device worn on the head, the VR display device has a built-in display module, an eye movement acquisition module, a cerebral blood oxygen acquisition module and a communication module, the eye movement acquisition module is used to acquire eye images of the subject, the cerebral blood oxygen acquisition module is used to acquire cerebral blood oxygen signals of the subject's head, the VR display device and the eye movement acquisition module, the cerebral blood oxygen acquisition module respectively realize data transmission and clock synchronization with an external terminal device through the communication module.

[0008] The eye movement acquisition module includes a plurality of near-infrared lamps and a near-infrared camera. The plurality of near-infrared lamps surround the edge area of ​​the display module to form a near-infrared light array, and the near-infrared camera is located at the fundus of the VR display device.

[0009] The cerebral blood oxygen collection module includes a probe light source module and a probe photosensing point arranged at the forehead position of the VR display device; the probe light source module is used to irradiate infrared rays to the subject's skin, and the probe photosensing point receives the reflected light of the probe light source module irradiating human tissue at different angles, and converts the optical signal into an electrical signal, and transmits the electrical signal to an external terminal device for processing through the communication module.

[0010] An electroencephalogram sensor for collecting electroencephalogram signals in the forehead area is also provided above the display module of the VR display device. The electroencephalogram sensor transmits the collected electroencephalogram signals to an external terminal device for processing through the communication module.

[0011] The probe light source emits red light and infrared light, and its light emission band is 600-900nm.

[0012] The system also includes an EEG acquisition device, which is connected to the VR display device and is used to collect EEG signals from the subject's head. The EEG acquisition device realizes data transmission with an external terminal device through the communication module.

[0013] The EEG acquisition device is a cap-type structure, which includes multiple electrodes and a signal processor connected to the multiple electrodes. The electrodes collect the EEG information of the subject, and the signal processor performs hardware filtering, signal enhancement and analog-to-digital conversion on the EEG information. The signal processor transmits data with an external terminal device through the communication module.

[0014] The electrodes are AgCl electrodes, which are distributed in specific areas of the human head and are used to collect brain electrical signals of the subject.

[0015] The communication module is a universal data interface for data transmission and clock synchronization with the wired USB port data.

[0016] The communication module is a Bluetooth communication module, which can realize wireless data transmission and clock synchronization with an external terminal device.

[0017] The technical solution of the present invention has the following advantages:

[0018] A. The present invention adopts a VR display device with a virtual immersive experience in the entire acquisition system. The entire test and data acquisition process can completely avoid external interference with the subjects. At the same time, the present invention takes into account the synchronous acquisition of eye movement and cerebral blood oxygen signals, directly transmits data with an external terminal device through a communication module, and adopts a clock synchronization method to solve the synchronization problems of real-time characteristics and multi-modal signals.

[0019] B. The present invention collects data on EEG signals, eye movement signals, and cerebral blood oxygen signals. The external device can provide more comprehensive parameters for brain function analysis through the acquired multiple brain function signals, thereby making a more accurate assessment of human cognition.

[0020] C. The present invention integrates an eye movement acquisition module, a cerebral blood oxygen acquisition module, and an electroencephalogram acquisition device with a VR display device, and synchronously transmits various types of acquisition signals to an external terminal device for data analysis through wireless or wired data transmission, making the entire detection more convenient and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the specific embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a structural schematic diagram of the synchronous acquisition system provided by the present invention;

[0023] Figure 2 This is a schematic diagram of the structure of the EEG acquisition device;

[0024] Figure 3 It is the physical topology diagram of the system provided by the present invention.

[0025] The following are marked in the figure:

[0026] 1-VR display device; 11-display module, 12-eye movement acquisition module, 121-near infrared light, 122-near infrared camera, 13-brain blood oxygen acquisition module, 131-probe light source module, 132-probe photosensitizing point;

[0027] 2-EEG acquisition device, 21-electrode, 22-signal processor;

[0028] 3- EEG sensor. DETAILED DESCRIPTION

[0029] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] like Figure 3 As shown, the present invention provides a multimodal immersive synchronous acquisition system based on eye tracking-brain function activity detection, including a VR display device 1, the VR display device 1 is built with a display module 11, an eye movement acquisition module 12, a cerebral blood oxygen acquisition module 13 and a communication module, the eye movement acquisition module 12 is used to acquire the subject's eye image, the cerebral blood oxygen acquisition module 13 is used to acquire the cerebral blood oxygen signal of the subject's head, and the VR display device 1 and the eye movement acquisition module 12, the cerebral blood oxygen acquisition module 13 respectively realize data transmission and clock synchronization with an external terminal device through a communication module. The present invention adopts a VR display device with a virtual immersive experience in the entire acquisition system, the subject wears a virtual reality presentation device on the eyes, and the subject watches stimulating materials such as videos or pictures through a built-in color display module, and the entire test and data acquisition process can completely avoid external interference with the subject; the present invention also takes into account the synchronous acquisition of eye movement and cerebral blood oxygen signals, the communication module directly transmits data with an external terminal device (such as a computer), and solves the synchronization problem of real-time characteristics and multimodal signals by adopting a clock synchronization method, so that various types of data can be transmitted synchronously.

[0031] The eye movement acquisition module 12 here preferably includes a plurality of near-infrared lamps 121 and a near-infrared camera 122. The plurality of near-infrared lamps 121 surround the edge area of ​​the display module 11 to form a near-infrared light array. The near-infrared camera 122 is located at the fundus of the VR display device 1, and transmits the collected original eye image to the computer for processing through the communication module. Because the eye area is relatively dark when wearing a VR display device, additional illumination of the eye area is required to collect pupil information. The characteristic of near-infrared light is that it is invisible light and will not affect the human eye. At the same time, the screen viewed by the user is very close to the eye area, and the brightness of the screen will interfere with the collection of eye images. The near-infrared camera 22 is used to filter out all visible light bands and only collect information illuminated by the near-infrared lamp.

[0032] The communication module here can be a universal data interface, compatible with different data formats, and can realize wired USB port data transmission and clock synchronization. Of course, it can also be a Bluetooth communication module, which can realize wireless data transmission and clock synchronization. The communication module is compatible with common communication protocols, and the synchronization function can be performed by the built-in clock or controlled by an external terminal device.

[0033] like Figure 1 As shown, the cerebral blood oxygen acquisition module 13 includes a probe light source module 131 and a probe photosensitive point 132 arranged at the forehead position of the VR display device 1; the probe light source module 131 is used to irradiate infrared and near-infrared rays to the forehead skin position of the subject, wherein the luminous wavelength band is 600-900nm, and the probe photosensitive point 132 receives the reflected light irradiated on the subject's skin, and converts the optical signal into an electrical signal, and transmits the obtained electrical signal to the external terminal device for processing through the communication module. Of course, as Figure 1 As shown, an EEG sensor 3 for collecting EEG signals in the forehead area may also be provided above the display module 11 of the VR display device. The EEG sensor 3 transmits the collected EEG signals to an external terminal device for processing via the communication module.

[0034] In addition, it is further preferred that the system is further provided with an EEG acquisition device 2, which is connected to the VR display device 1 to form a whole, which is convenient for carrying and testing, and is used to detect the EEG signal of the subject's head. The EEG acquisition device 2 realizes data transmission with the external terminal device through the communication module. Figure 2 As shown, the EEG acquisition device 2 here is preferably a cap-type structure, which is worn on the head of the subject, and specifically includes multiple electrodes 21 and a signal processor 22 connected to the multiple electrodes 21. The multiple electrodes 21 are distributed at specific positions on the subject's head. AgCl electrodes are preferably used to collect EEG information from specific areas of the subject's head, and the signal processor performs hardware filtering, signal enhancement and analog-to-digital conversion on the EEG information. The signal processor transmits data with an external terminal device through a communication module, processes the data through the external terminal device, and evaluates the cognitive ability of the subject.

[0035] The present invention uses an integrated VR display device with EEG detection, eye movement tracking and cerebral blood oxygen detection functions to synchronously acquire data on EEG signals, eye movement signals and cerebral blood oxygen signals on an external terminal device. The external device can provide more comprehensive parameters for brain function analysis through the acquired multiple brain function signals, thereby making a more accurate assessment of human cognition.

[0036] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. However, the obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A multi-modal immersive synchronous acquisition system based on eye tracking and brain function activity detection, characterized in that: The system comprises a VR display device (1) worn on the head, wherein the VR display device (1) has a built-in display module (11), an eye movement acquisition module (12), a cerebral blood oxygen acquisition module (13) and a communication module, wherein the eye movement acquisition module (12) is used to acquire eye images of a subject, and the cerebral blood oxygen acquisition module (13) is used to acquire cerebral blood oxygen signals from the subject's head, and the VR display device (1) and the eye movement acquisition module (12) and the cerebral blood oxygen acquisition module (13) respectively realize data transmission and clock synchronization with an external terminal device through the communication module.

2. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 1 is characterized in that: The eye movement acquisition module (12) comprises a plurality of near-infrared lamps (121) and a near-infrared camera (122); the plurality of near-infrared lamps (121) surround an edge region of the display module (11) to form a near-infrared light array; and the near-infrared camera (122) is located at the fundus of the VR display device (1).

3. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 1 is characterized in that: The cerebral blood oxygen collection module (13) comprises a probe light source module (131) and a probe light-sensitive point (132) arranged at the forehead of the VR display device (1); the probe light source module (131) is used to irradiate infrared rays to the skin of the subject, and the probe light-sensitive point (132) receives the reflected light of the human tissue irradiated by the probe light source module (131) at different angles, converts the optical signal into an electrical signal, and transmits the electrical signal to an external terminal device for processing via the communication module.

4. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 3 is characterized in that: An electroencephalogram sensor (3) for collecting electroencephalogram signals in the forehead region is also provided above the display module (11) of the VR display device. The electroencephalogram sensor (3) transmits the collected electroencephalogram signals to an external terminal device for processing via the communication module.

5. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 3 is characterized in that: The probe light source module (131) emits red light and infrared light, and its light emission wavelength range is 600-900nm.

6. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to any one of claims 1 to 5, characterized in that: The system further comprises an electroencephalogram (EEG) acquisition device (2), which is connected to the VR display device (1) and is used to acquire EEG signals from the subject's head. The EEG acquisition device (2) realizes data transmission with an external terminal device via the communication module.

7. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 6 is characterized in that: The electroencephalogram (EEG) acquisition device (2) is a cap-type structure, comprising a plurality of electrodes (21) and a signal processor (22) connected to the plurality of electrodes (21); the electrodes (21) acquire the EEG information of the subject, and the signal processor (22) performs hardware filtering, signal enhancement and analog-to-digital conversion on the EEG information; the signal processor (22) transmits data with an external terminal device via the communication module.

8. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 7, characterized in that: The electrode (21) is an AgCl electrode, which is distributed in a specific area of ​​a person's head and is used to collect brain electrical signals of a subject.

9. The multimodal immersive synchronous acquisition system based on eye tracking and brain function activity detection according to claim 1, characterized in that: The communication module is a universal data interface for data transmission and clock synchronization with the wired USB port data.

10. The multi-modal immersive synchronous acquisition system based on eye tracking-brain function activity detection according to claim 1, characterized in that: The communication module is a Bluetooth communication module, which can realize wireless data transmission and clock synchronization with an external terminal device.

Images