An eye movement detection and analysis system based on multi-channel array electrooculogram electrodes

Abstract

The present invention belongs to the technical field of human motion monitoring, and is specifically an eye movement detection and analysis system based on a multi-channel array electrooculogram electrode. The present invention comprises an acquisition module, a lower computer and a host computer; data is transmitted between the two by wired or wireless means; the acquisition module and the lower computer include a front-end acquisition device and a signal processing hardware circuit; the host computer is a client computer loaded with a signal analysis and information display program; the client receives the information collected by the lower computer and displays the analysis results in a user graphical interface. The acquisition device is provided with a multi-channel array electrooculogram electrode, which is used to collect eye movement information and related physiological information, and the multi-source and multi-dimensional physiological signals collected are processed through signal processing analysis methods and intelligent algorithms, so as to effectively detect eye movement conditions with micro-expressions. The acquisition module and the lower computer are arranged in a wearable headband structure; the system of the present invention obtains richer information for analyzing eye movement events, and the user experience is more comfortable.

Description

Technical Field

[0001] The invention belongs to the technical field of human motion monitoring, and in particular relates to an eye movement monitoring and analysis system. Background Art

[0002] Vision is one of the important sensory channels of animals. As a visual receptor, eye movement controls human's reception of visual information. Physiologically, the movement of each eye is controlled by three groups of antagonistic muscles, which are controlled by different nuclei in the brainstem. The different tension and relaxation combinations produced control 10 different basic movements of our eyes. Since eye movement is controlled by brain nerves, the way the eye moves can also reflect some aspects of neurological health to a certain extent, such as involuntary nystagmus caused by abnormalities in the nerve conduction mechanism. In addition, daily eye movements also reflect our cognitive status, such as attention, areas of interest, or physiological conditions, such as fatigue.

[0003] Eye movements can be roughly divided into five types. Fixation, where the line of sight briefly stays on a certain point; saccade, where the line of sight quickly jumps from one point to another; pursuit, where the line of sight follows a slowly moving point; nystagmus, where the line of sight oscillates back and forth within a small range; and vergence, where the line of sight moves in different directions. Most human eye movements can be decomposed into a combination of these five types of eye movements. Understanding the movement of the eyeballs when these eye movements and their combinations occur can provide a data basis for understanding and studying human cognitive patterns. Research such as exploring human attention through eye movements and inferring a person's area of ​​interest through the stay of the line of sight are often explored topics in cognitive science.

[0004] Therefore, information related to eye movements can be used in many fields. For example, using human eye movements as a means of interaction can enrich the means of human-computer interaction and improve human efficiency. Another example is to obtain and analyze eye movement patterns to understand people's information processing patterns, areas of interest, etc., to provide objective data for user surveys. Or from a physiological perspective, understand the characteristics of people's eye movements in different situations or the different physiological characteristics of eye movements, and provide theoretical basis and research data for health assessment, bioinformatics collection, etc.

[0005] In short, the acquisition of eye movement data is of great significance from the perspective of practical application, theoretical research, physiological exploration and engineering implementation.

[0006] The research on methods of obtaining eye movement information has a history of more than 50 years. The most representative methods are iris coil method, iris reflection method, video method and electrooculography.

[0007] From the signal source of the detection method, it can be roughly divided into two technical routes: collecting peripheral signals to reflect eye movements and directly collecting physiological electrical signals generated by the human body during eye movements to infer eye movements. The essential difference between the two technical routes is that the raw data collected by the latter is the electrical signal generated by the human body, so it has certain physiological significance, while the raw data collected by the former does not come from the human body itself.

[0008] The more representative technologies that collect peripheral signals to respond to eye movements include the iris coil method, the iris reflection method, and the video method. These methods mainly use the device to generate markers on the eyeball, and then detect the changes in the markers when eye movements occur to indirectly infer the eye movements. For example, the iris coil method requires the use of contact lenses with coils to be worn on the cornea of ​​the subject, and the video method requires the use of an optical camera to capture the optical image of the subject's pupil. The advantages of these methods are high accuracy and sensitivity, but due to many disadvantages, such as high configuration difficulty, poor wearing comfort, data processing requires a large amount of resources, and poor anti-interference ability, they cannot meet the requirements of daily eye movement detection for portability, comfort, ease of use, and long use time. Therefore, products developed based on the above technology are only suitable for laboratory environments with controlled conditions, and still cannot meet the needs of obtaining eye movement data in daily life.

[0009] Electro-oculogram (EO) is a method for detecting the changes in the electric field formed around the eyeball by the physiological current between the retina and the cornea when the eyeball moves. It is currently a widely recognized eye movement detection technology based on physiological electrical signals. Compared with other eye movement detection technologies that use peripheral signals, in addition to the fact that the data obtained by EO has more physiological significance due to its data form, the advantages of EO obtaining EO information over other eye movement detection technologies are as follows: 1) From the perspective of acquisition, since its signal source is the electrical signal generated by the human body, the main source of interference comes from other physiological signals generated by the subject, such as electromyographic signals, and motion artifacts. Compared with the iris coil that uses magnetic field information and the video method that uses video information as the source, it is less affected by other factors in the environment and has better stability. 2) From the perspective of signal transmission and processing, since the electrooculogram signal is a physiological electrical signal, the data volume is extremely small compared to the image signals of the video method and the iris reflection method, so the resources required to process the electrooculogram signal are also much smaller than the video method, so the acquisition time can be longer than the video method. 3) From the perspective of the configuration of the acquisition environment, the electrooculogram requires multiple electrodes to be placed on the epidermis around the eye socket to obtain the electrooculogram signal. Compared with other methods, the configuration is simpler and easier to operate, the signal is relatively more stable, and the restrictions on the subject are relatively smaller, so the user experience is relatively better. 4) There is no restriction on whether the eyeball is visible, and the eyeball movement of the subject can be detected when the eyes are closed.

[0010] The existing detection methods based on electrooculogram signals have various electrode arrangements. The most widely accepted ones are standard leads such as Fig.14 In addition, several consumer-grade EO-based consumer products and research have also proposed different EO electrode arrangement schemes, such as the EO electrode arrangement scheme proposed by Japan's JINs company and IMEC in the Netherlands that is easy to mount on eyeglass frames.

[0011] However, the existing electrooculography detection method still adopts the classic electrooculography data collection lead method proposed in the 1960s, or simply adjusts the number of electrodes or the position of electrodes based on it. In essence, it has not changed. It still uses a single electrode to obtain the potential change caused by the electric field change in a certain part around the electrode when the eyeball moves, and then collects and analyzes the potential changes of different parts collected by several electrodes distributed around the eye sockets to infer the movement of the eyeball, and finally infers the movement of the line of sight.

[0012] The problems with this approach are:

[0013] 1. Two or more electrodes are required. It is impossible to accurately detect multi-directional eye movement with only one electrode. The main reasons for this problem are two aspects: 1) The traditional electrode design makes it necessary to compare the signals obtained by two electrodes to extract clean electrooculogram signals, so as to remove the noise in the signal and obtain effective electrooculogram signals. 2) The electrooculogram signal obtained by a traditional electrooculogram electrode is only the information about the change in the potential size in the electric field formed by the eyeball when the eye moves. This information cannot reflect spatial information, so it is necessary to jointly analyze the electrooculogram signals obtained by electrodes at multiple points to restore the transformation of the electric field generated by the eyeball in space, so as to finally obtain the movement of the eyeball. Therefore, the existing electrooculogram technology requires a large number of electrodes distributed at different positions in the eye sockets of the subject to provide sufficient information.

[0014] The use of multiple electrodes will bring two problems: First, the existing electrooculographic system electrode configuration is mostly composed of several electrodes spread around the orbit. The process of configuring multiple electrodes before each test is cumbersome and not friendly to the testee. During the collection process, problems such as wire entanglement will also occur, affecting the accuracy. Second, the placement of electrodes will introduce certain human errors, resulting in different placements of electrodes at the same position. The placement of multiple electrodes will increase this error to a certain extent, resulting in large differences in the human errors introduced by different collection tasks, which will ultimately affect the accuracy.

[0015] 2. It is impossible to obtain the spatial information of the electrooculogram signals of each channel. The essence of electrooculogram is a field signal. The position of each acquired signal in the electric field is an important information, which can help signal analysis and improve accuracy. The electrodes of existing electrooculogram acquisition systems are mostly unit-separated, that is, each electrode is a unit and performs acquisition work independently, and the electrodes are set relatively independently. Because there is no relative spatial information between the electrodes, the final signal acquired is the change in the potential amplitude acquired by each electrode over time. The spatial relationship between the signal sequences cannot be obtained.

[0016] 3. It is not possible to separate eye movement signals from other noise signals. The main noise sources that affect electrooculogram signals are: EEG signals, electromyographic signals, artifacts and other noises. The existing processing method is to use filters to remove signals other than electrooculogram signals from the original signal in the preprocessing stage before or after acquisition. However, the design concept of such a processing method is based on the premise that other signals in the eye movement process are useless signals. In fact, when an eye movement event occurs, it is often inevitably accompanied by other valuable signals that can be used to describe the eye movement event, such as electromyographic signals and electroencephalographic signals caused by facial expressions. These multi-modal physiological signals are helpful in analyzing the eye movement related situations of the subject in different scenarios. In the existing electrooculogram acquisition system, since the electrode positions are independent of each other and there is no spatial correlation, each electrode obtains different noise signals except the electrooculogram signal, so the signals obtained by each electrode cannot be compared, and it is impossible to accurately analyze the various signal components in the aliasing of all noise signals. Summary of the invention

[0017] The purpose of the present invention is to propose an eye movement detection system with a multi-channel array of electrooculogram electrodes as the core, which is easy to use, highly robust and highly integrated, so as to overcome the problems caused by the limitations of existing solutions.

[0018] The eye movement detection and analysis system of the multi-channel array electrooculogram electrode provided by the present invention collects the eye movement information and related physiological information (including but not limited to electromyography information, electroencephalogram information, and motion artifact information) of the subject by placing one or more multi-channel array electrooculogram electrodes on the epidermis around the eye sockets of the subject (such as at the outer corners of the eyes), and processes the multi-source and multi-dimensional physiological signals collected by the multi-channel array electrooculogram electrodes through the signal processing and analysis method and intelligent algorithm proposed by the system, so as to effectively detect eye movements with micro-expressions.

[0019] The eye movement detection and analysis system based on multi-channel array electrooculogram electrodes proposed in the present invention includes two major parts, an acquisition module and a lower computer and a host computer. Data is transmitted between the two parts through wired and wireless transmission equipment; the acquisition module and the lower computer part include two parts: the front-end acquisition device and the signal processing hardware circuit; the host computer part is a user client computer pre-loaded with two programs: signal analysis and information display and communication. The client collects and analyzes the information received from the lower computer and displays the results in a graphical user interface. The overall system framework is as follows: Figure 1 As shown. Among them:

[0020] The front-end acquisition device has the main function of collecting physiological electrical signals. It includes: a multi-channel array electrooculogram electrode, a reference electrode and a ground electrode. The ground electrode is arranged at the mastoid of the left or right ear of the subject, and the reference electrode is arranged at the center of the eyebrows; the multi-channel array electrooculogram electrode is arranged on the epidermis around the user's eye socket, such as the skin outside the outer corner of the eye.

[0021] The multi-channel array electrooculogram electrode is an electrooculogram collection array comprising a plurality of contacts, and the number of contacts integrated on a single electrode is not less than 9. When in use, for an effective detection, only one multi-channel array electrooculogram electrode is needed to collect the obtained data, and the use of multiple multi-channel array electrooculogram electrodes can improve the detection accuracy.

[0022] The multiple contacts on the multi-channel array electrooculogram electrode are integrated on a soft but non-ductile substrate to form a multi-channel array electrooculogram electrode. The available solutions include but are not limited to integrating the contacts on a double-layer flexible PCB board, using a partial via design for the contacts, and connecting each contact to the socket at the tail end of the collection array electrode through a copper cable with a gold-plated surface. The multi-channel array electrooculogram electrode can be fixed to the skin in various ways that can make the contacts closely contact the epidermis around the eye socket, including but not limited to the fixing method using double-sided tape combined with conductive paste.

[0023] In the present invention, the multi-channel array electrooculogram electrode can be placed at any position around the eye socket, such as the outer skin of the outer corner of the eye. Figure 2 The display shows a placement method of selecting a single electrode to be placed outside the right outer corner of the eye of the subject, wherein the multi-channel array electrooculogram electrode is as follows Figure 2 As shown in figure e.

[0024] The signal processing hardware circuit mainly includes a power management module, a physiological electrical signal acquisition module and a main control module. The working principle of the signal processing hardware circuit is as follows Figure 4 As shown. Among them:

[0025] The power management module receives the power supply and converts the power supply voltage into a voltage that can meet the voltage of the main control module and the physiological electrical signal module through a series of operational amplifier chip combinations.

[0026] The solution in this embodiment is: mainly powered by a 5V power supply through the USB interface, using the TPS73225 chip to stably generate a +2.5V voltage, and using the TPS60403 and TPS72325 chips to stably generate a -2.5V voltage. In addition, the LP2985 chip is also used to stably generate a 3.3V voltage to meet the voltage requirements of the main control module and the physiological electrical signal module.

[0027] Other equally available options include replacing the TPS73225 chip with similar uA431 chips, TL431 chips, LM136-2.5 and other LDO voltage regulator chips.

[0028] The physiological electrical signal acquisition module performs relative differential processing on the electrical signal obtained from the front-end acquisition device to obtain the potential signal of each channel. The obtained signal will be further processed by the main control module.

[0029] The solution in this embodiment is to use the analog front-end ADS1299 chip, in which all analog front-ends share a common ground, to collect the potential difference of all channels to the common ground point. The working principle of the physiological signal acquisition module is as follows Figure 3 shown.

[0030] Another equally available alternative is to replace the ADS1299 chip with a similar acquisition chip such as the RHD2000 series.

[0031] The main control module is responsible for controlling signal acquisition, preliminary information processing, and integrating and sending the packaged signals to the upper computer.

[0032] The scheme in this embodiment: using STM32F103C8T6 chip, it has the advantages of low cost, low power consumption and small size. The main control board module performs SPI communication configuration on multiple ADS1299 chips in the physiological electrical signal acquisition module in a daisy chain manner, and sets sampling related parameters such as sampling synchronization clock, sampling frequency, analog front-end magnification, data accuracy, etc. By reading the DRDY pin of ADS1299, the main control module can detect and send a read command to read the acquisition voltage value of each ADS1299 chip, and upload it to the host computer. In the analog signal part, the main control module is connected to the multi-channel array eye electrode through the FPC interface, wherein the analog small signal routing is as short as possible to ensure that the acquisition signal does not couple too much noise. The collected signal is first encapsulated into a frame using the Uart protocol, and the parity code is used to detect whether the sent data packet is abnormal.

[0033] After collecting all the original electrooculogram signals at each moment and completing the digital conversion of analog signals, the designed hardware system transmits the data to the host computer using wireless methods (such as Bluetooth, Wifi, etc.) or wired methods (such as USB, etc.). After receiving and decoding the data packet, the host computer can store the data in the agreed format.

[0034] Other alternatives available are: the processing chip can be replaced by the corresponding main control module, such as: single-chip microcomputer, FPGA, Raspberry Pi and other highly integrated main control modules; the communication protocol can use I²C, UART, CAN and other synchronous serial data transmission standards similar to SPI; in addition to the daisy chain connection, other topological connection methods can be used, such as linear, ring, and star topologies. In addition to the internal clock of the chip, the synchronous clock can also be achieved by using an external crystal oscillator; the protocol used in the signal packaging method can also use a similar data transmission protocol, such as a wireless Wifi transmission protocol.

[0035] The host computer can be a machine similar to a personal computer, with computing, storage, processing and other functions, and a human-computer interaction interface. It includes personal computers, tablet computers, mobile phones, etc. The host computer part includes signal analysis and user graphic display interface. Among them:

[0036] The signal analysis includes a signal preprocessing module, a signal analysis module and an intelligent algorithm module.

[0037] The signal preprocessing module performs preprocessing work including bandpass filtering, power frequency removal, artifact elimination, bad track detection and damaged data removal on the original signal obtained from the main control module, and then transmits the processed signal to the signal analysis module.

[0038] The signal analysis module first performs independent component analysis on the preprocessed signal to parse out the electrooculogram signal and other physiological signals related to eye movement events (such as electromyography signals, etc.), and then sends the parsed signals from multiple sources to the intelligent algorithm for further eye movement event analysis and detection.

[0039] The intelligent algorithm module extracts the time domain information, frequency domain information and spatial information of the signals from various different sources (electrooculogram signals, electromyography signals, etc.) and sends them to the detection model generated in the intelligent algorithm module to finally obtain the detection results of eye movement events.

[0040] The user graphical display interface is used for information display interaction, including organizing the analysis and test results into information that is easy for users to understand and presenting it to the person being tested in a graphical form. The data that can be presented include: the trajectory of line of sight movement, classification of eye movement conditions, micro-expressions during eye movement, etc.

[0041] The innovative features of the system of the present invention are mainly as follows:

[0042] Part I. In terms of electrode design, the present invention proposes a multi-channel array electrooculogram electrode design that is easy to use and highly integrated. This solution only requires the placement of one electrode on the epidermis around the eye socket of the subject, such as the outer side of the outer canthus, for collection. Compared with the existing traditional electrooculogram collection system that requires a number of electrodes distributed around several eyes, the solution of the present invention improves ease of use and comfort. This provides a high user experience and a high user acceptance basis for long-term eye movement detection in daily scenarios based on electrooculogram data sources.

[0043] In the second part, in terms of the source signal type selection, the signal collected by the electrode proposed in the first part is a separable multi-source signal. Through a highly interpretable signal processing method, the physiological electrical signals of various different sources collected by the scheme of the present invention when an eye movement event occurs can be parsed one by one, such as the electrooculogram signal and the electromyogram signal of the electrode coverage area. This makes the information that can be used to analyze eye movement events richer than the existing electrooculogram eye movement event detection technology.

[0044] In the third part, in terms of the information dimension of the electrooculogram signal, since the electrode design of the present invention adopts the form of a multi-channel array electrode, the electrooculogram signal of each electric shock can be obtained while retaining the spatial information between each signal. Compared with the traditional electrooculogram eye movement event detection technology, the information dimension is richer. Through the intelligent algorithm, eye movement events can be accurately detected with only one electrode.

[0045] Combining the second and third parts, the eye movement signals collected by the multi-channel array electrooculogram electrodes can be processed by the signal processing and intelligent algorithm to obtain the gaze position of the person being tested when he or she turns his or her eyeballs, as well as the facial state during eye movement.

[0046] In view of the drawbacks of the existing electrooculogram system, the present invention proposes a scheme of using only one multi-channel array electrooculogram electrode to be placed around the eye socket to collect physiological signals related to eye movement events. This scheme uses a simple placement method and has a relatively small impact on the facial movements of the subject. The physiological electrical signals of the subject's eye movements are collected using high-resolution array electrodes, and the spatial information between each piece of collected information is retained at the same time, making the data dimensions that can be used to establish an eye movement detection model richer. The data collected by this collection scheme includes a variety of physiological electrical signals including electrooculogram, which makes it possible to analyze different physiological electrical signals related to a variety of eye movement events through signal processing methods such as blind source separation, so as to restore the subject's eye movements from different angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a diagram showing the structural framework of the eye movement detection and analysis system of the present invention.

[0048] Figure 2 Schematic diagram of electrode placement using a single multi-channel array electrooculogram electrode placed outside the right outer canthus.

[0049] Figure 3 This is the schematic diagram of the physiological signal acquisition module.

[0050] Figure 4 This is the schematic diagram of the signal processing hardware circuit sub-part.

[0051] Figure 5 It is the overall system diagram of the embodiment.

[0052] Figure 6 The subjects are given a multi-channel array electrooculogram data acquisition headband.

[0053] Figure 7-1 This is the hardware appearance diagram.

[0054] Figure 7-2 This is the hardware exploded view (1).

[0055] Figure 7-3 This is the hardware exploded view (2).

[0056] Figure 8 Schematic diagram of data preprocessing on the PC host computer for electrooculogram detection headband signals.

[0057] Fig. 9 This is the schematic diagram of the signal analysis and intelligent algorithm of the electrooculogram detection head with a personal computer host computer.

[0058] Fig.10 Comparison of the decomposition results of rightward saccades under different expression states.

[0059] Fig.11 Comparison of the spatial information of each channel of the multi-channel array electrooculogram electrode with the actual eye movement.

[0060] Fig.12 Results of eye movement recognition in 8 directions.

[0061] Fig.13 A use case diagram for the graphical user interface.

[0062] Fig.14 This is a typical electrooculogram electrode arrangement scheme.

[0063] The numbers in the figure are: a is the operating status indicator light, b is the reference electrode, c is the electrode insulating silicone pad, d is the physiological electrical signal acquisition module, e is the multi-channel array electrooculogram electrode, f is the power management module, g is the headband hardware circuit placement frame, h is the ground electrode, i is the adjustable head frame support, j is the circuit protection shell, k is the main control module, l is the battery, m is the headband control button, and n is the USB interface. DETAILED DESCRIPTION

[0064] The eye movement detection and analysis system of the multi-channel array electrooculogram electrode provided by the present invention, as an embodiment, is designed to include a complete hardware and software system including a multi-channel array electrooculogram data acquisition headband and a personal computer host terminal. The acquisition module and the lower computer are designed as a wearable headband structure, which integrates the functions of eye movement event-related physiological electrical signal acquisition, amplification, and control. The host computer adopts a personal computer terminal, and the overall system diagram is as follows: Figure 5 shown.

[0065] The wearable electrooculogram data acquisition module is designed as a headband. The base material of the headband is a tough plastic material. The headband integrates a multi-channel array electrooculogram electrode sheet, a reference electrode, a ground electrode, a signal processing hardware circuit and other auxiliary circuits. The signal processing hardware circuit includes a power management module, a main control module and a physiological electrical signal acquisition module. The multi-channel array electrooculogram electrode, the reference electrode, the ground electrode and the signal processing hardware circuit are all connected through FPC wires.

[0066] Before use, the subject should first wear the headband on his head so that it just covers the brow bone and the multi-channel array electrooculogram electrode can cover the outer corner of the right eye. Then adjust the three adjustable headrests to fit the size of the subject's head. It can be used only after it is stable and comfortable to wear. Figure 6 shown.

[0067] After the original electrooculogram signal is acquired by the multi-channel array electrooculogram electrode sheet, it is transmitted to the signal processing hardware circuit through the flexible FPC wire. The signal processing hardware circuit performs a series of operations such as synchronization, amplification, and digital-to-analog conversion on the original data collected by each channel data, and then uploads it to the PC host computer through USB high-speed wired transmission. The collected original data can also be wirelessly transmitted to the PC host computer through the Bluetooth module for analysis.

[0068] 1. Design of wearable headband structure

[0069] In this embodiment, the acquisition module and the various parts of the lower computer are designed and arranged in a wearable headband structure. The overall appearance of the wearable headband structure is as follows: Figure 7-1 As shown, Figure 7-2 The headband hardware exploded diagram is shown in FIG. The headband support structure comprises two parts: a headband hardware circuit placement frame g and an adjustable head frame head support i. Among them:

[0070] The headband hardware circuit placement frame g is divided into three parts: the left placement frame g-1, the right placement frame g-2 and the front forehead guard g-3. The material of the headband hardware circuit placement frame g is selected from non-toxic and harmless materials with good elasticity and impact resistance. The circuit unit and the sensor unit are carried inside and outside the headband hardware circuit placement frame. The circuit installed on the headband hardware circuit placement frame g includes: an acquisition unit composed of a multi-channel array electrooculogram electrode e and a physiological electrical signal acquisition module d is installed on the right placement frame g-2, and a main control module k is installed, and a transmission, control and feedback unit composed of a headband control button m, an operating status indicator light a and a USB interface n. A battery l is installed on the left placement frame g-1, and a power management module f and a ground electrode h constitute a supporting circuit. A reference electrode b is installed on the front forehead guard g-3. An insulating silicon base c is installed near all electrodes that contact the skin. All electronic components are wrapped in a circuit protection shell j. The specific positions of each part are as follows Figure 7-2 shown.

[0071] There are three adjustable head frame head supports i, which are distributed above the forehead, above the back of the head and at the back of the head. The adjustable head frame head support i is an adjustable design, consisting of an adjustable metal frame i-1 and a head support i-2. The length of the adjustable head frame head support i can be adjusted by adjusting the length of the head support i-2 metal frame in the head support to adapt to the head size of different subjects. The headband composed of the headband hardware circuit placement frame g and the adjustable head frame head support i can provide good inward pressure, so that the sensor units on the inside of the headband can be better pressed on the skin surface of the subject. The reference electrode b located in the middle of the front forehead g-3 and the multi-channel array electrooculogram electrode e at the outer corner of the right eye are mounted on the insulating silicon base c, and the electrode contact position is slightly higher than the surface of the silicone base. The three insulating silicone bases c ensure that the electrode contacts can have good contact with the skin, and the insulating and skin-friendly soft material also provides a comfortable wearing experience and prevents the circuit from contacting the skin.

[0072] The multi-channel array electrooculogram probe is a whole PCB. Its feature is that it integrates 64 2mm circular contact probes, with a spacing of 3mm between each probe. The circular contact probe is made of copper and the surface is gold-plated. The overall hardware design of the headband ensures the requirements of easy use, reliable fixation, comfortable wearing, and long-term use.

[0073] (II) Circuit design

[0074] Considering the wearing comfort and convenience of the entire embodiment, the overall circuit design adopts a distributed modular design, which includes three parts: power management module, physiological electrical signal acquisition module and main control module. The circuit modules are connected through FPC.

[0075] (1) Power management module

[0076] This module uses a variety of low-dropout linear regulator chips to convert the 5V power supply from USB to meet the power supply requirements of other modules in the hardware system. Its function is to convert the 5V input voltage into a stable +2.5V voltage, -2.5V voltage and +3.3V voltage to ensure the normal operation of other modules.

[0077] (2) Physiological electrical signal acquisition module

[0078] This module is connected to the multi-channel array electrooculogram electrode probe through the FPC interface. Each signal amplification circuit is designed with multiple parallel 8-channel bioelectric amplification and quantization analog front-end ADS1299, which amplifies the original multi-channel electrooculogram signal and performs analog-to-digital conversion to ensure that the signal used for subsequent analysis has a high signal-to-noise ratio. The amplified signal is transmitted to the main control module through the FPC wiring.

[0079] (3) Main control module

[0080] All components in the main control module are integrated on a whole PCB. The main chip STM32F103C8T6 chip is responsible for controlling sampling synchronization clock, sampling frequency, analog front-end magnification, data accuracy and other sampling related parameters, storing the collected data, and transmitting the data to the personal computer host through the Bluetooth module according to the transmission protocol. High-performance microprocessor and Bluetooth module interface in the main control module. The high-performance microprocessor uses a parallel multi-SPI interface to control multiple analog front ends at the same time, and provides a unified sampling clock output to the analog front end. While collecting all the original electrooculogram signals at each moment, the data is sent to the personal computer host. The personal computer host decodes the received data packets and stores them in the personal computer in the agreed data format.

[0081] 3. Software and intelligent algorithms on the PC host

[0082] The data processed by the main control module is imported into the PC wirelessly or wiredly using Bluetooth or USB, so that the host computer analysis software can pre-process the original signal and then use intelligent algorithms to analyze the obtained data in combination with the task. The obtained analysis results will be presented in the user graphical interface.

[0083] In the present invention, the working process of the system is as follows:

[0084] When the person being tested uses the device for the first time, a system calibration is required. After the person being tested wears the device, he or she needs to complete the corresponding actions according to the prompts. The collected data is transmitted to the host computer for storage and then used to classify eye movement events based on electrooculogram signals. After the model training and personalized calibration are completed, the user can use the device to detect and analyze eye movement events.

[0085] The person being tested can browse and manage the records and reports by using the client software installed on the PC.

[0086] (1) Data preprocessing

[0087] The goal of data preprocessing is to filter out useless information and ensure the quality of data sent to the intelligent algorithm training to improve the accuracy of algorithm predictions, while saving the computing power of the personal computer host and the storage resources of the storage medium.

[0088] The pre-processing module first sends the raw data to a bandpass filter to obtain an electrical signal of 0.1-900Hz, and then uses a 50Hz notch filter to remove the power frequency interference corresponding to 50Hz and its multiples, thereby obtaining a relatively pure electrooculogram signal. Artifact removal is to process the data using wavelet transform motion artifact algorithm, bad track detection, damaged data removal and other operations. The basic processing flow is as follows: Figure 8 shown.

[0089] (2) Signal analysis and intelligent algorithms

[0090] The signal analysis module used in this embodiment focuses on collecting and decomposing the electrical signals related to eye movement events according to the usage scenario of this embodiment, so as to obtain the electrooculogram components and other expression-related components. Among them, the electrooculogram components are multi-channel electrooculogram signals, and other expression-related components are the aliasing of electromyographic signals and other signals. The multi-channel electrooculogram signals and other expression-related components are sent to the intelligent algorithm module, and the classification analysis of eye movement conditions and the classification analysis of expressions during eye movement can be obtained. The entire intelligent algorithm module includes two main parts, namely the electrooculogram signal processing submodule and the expression-related signal processing submodule. The electrooculogram signal processing part includes downsampling, feature discriminator, and voting decision, and can obtain the eye movement direction classification corresponding to the signal by processing the multi-channel electrooculogram signals. The expression-related signal processing part includes feature extraction and integrated learning. This part can obtain the expression and emotion classification corresponding to the signal by analyzing the electromyographic signals and other signals in the expression-related components. The intelligent algorithm process of this embodiment is as follows Fig. 9 shown.

[0091] (2.1) Signal analysis

[0092] In the signal analysis module, the main method is ICA (Independent Component Analysis). ICA is used to extract original independent signals from mixed data and is a commonly used signal separation method. Among many ICA algorithms, the fixed point algorithm (FastICA) is often used because of its fast convergence speed and good separation effect. The FastICA algorithm has forms based on kurtosis, maximum likelihood, and maximum negative entropy. The FastICA based on maximum negative entropy is used in the present invention. The algorithm uses maximum negative entropy as a search direction and can extract independent sources sequentially.

[0093] In the present invention, FastICA based on the maximum negative entropy is used to obtain electrooculographic components, expression and action-related electromyographic components. Since the electrodes collect the aliasing of signals from multiple different sources, we decentralize the data obtained for a certain period of time, make the mean of the data 0, and then whiten the data, and then optimize and solve according to the maximum negative entropy principle to obtain independent components in the data. Then, the kurtosis tool can be used to distinguish the electrooculographic components and the electromyographic components to obtain a more accurate signal feature expression, so as to prepare data for subsequent different tasks.

[0094] (2.2) Intelligent algorithm-expression-related signal processing

[0095] Expression-related signal processing is completed by the expression-related signal processing submodule. The signal analysis module extracts the RMS, Mean and SSC features of the n channels after low-pass filtering of the analyzed electromyographic signal components. Each movement is converted into a column of 3*n dimensional feature vectors. After all the action features of multiple types of expressions are extracted, the model is trained through an integrated learning classifier using ten-fold cross validation to obtain the final classification accuracy. The three types of features used: RMS can reflect the power of the signal, Mean can characterize the average amplitude of the signal, and SSC helps to reflect the muscle force and provide physiological information to distinguish the three types of expression movements. The integrated learning classifier used mainly combines multiple base learners to obtain better generalization performance than a single learner.

[0096] (2.3) Intelligent algorithm-electrooculogram signal processing

[0097] The electrooculogram signal processing is completed by the electrooculogram signal processing submodule. Based on the results of previous verification experiments, downsampling the data in the task of distinguishing eye movement angles does not affect the final discrimination accuracy and can greatly reduce the complexity of the algorithm. Therefore, the electrooculogram component signal found is downsampled four times.

[0098] Through previous verification experiments and literature research, it was found that the eye movement angle has a strong linear characteristic within the range of 30 degrees. Therefore, the present invention sends the downsampled data to the linear discriminator for training to obtain a discriminant model. In the decision-making stage, the channel-based linear discriminant results are used for voting. For example, linear discrimination is performed on the n data obtained by a multi-channel array electrooculogram electrode with a specification of n (n>=9) channels, and the category with the most channel data discrimination results is selected as the final discrimination result. Based on this discrimination method, the accuracy of data discrimination can be improved. In addition, it has strong robustness against situations such as poor contact of some electrodes and interference of channel signal data during use of the device.

[0099] 4. Experimental setup and results

[0100] To test the feasibility of this embodiment, we used the system solution described in this embodiment to collect the subjects' eye movement data in eight directions (horizontal to the right, horizontal to the left, vertically upward, vertically downward, 45 degrees to the upper right, 45 degrees to the upper left, 45 degrees to the lower right, and 45 degrees to the lower left) in three facial expression states (no expression, staring, and gritting teeth), and used the signal processing method described in the present invention combined with an intelligent algorithm to analyze the eye movement events from multiple dimensions such as eye movement classification and expression classification.

[0101] (1) Results of the eye movement event component decomposition experiment

[0102] pass Fig.10 It can be seen that the electrical signals generated by the eye saccade movement of the same eye angle under different expression states collected by the electrodes designed by the present invention are different. The left figure shows the original data collected by all channels when the subjects perform the task of controlling the eyeball to saccade to the right in two different expression states of clenching teeth and staring. The right figure shows the respective eye movement components and expression-related components decomposed from the two expression state information after the independent component decomposition of the eye movement event. It can be seen that since the eye movement conditions in the two states are similar, the decomposed electrooculographic components are similar, and because the expression states of the two states are different, the decomposed expression-related components are different.

[0103] (2) Comparison of eye movement images and corresponding channel spatial feature heat maps under different eye movement conditions

[0104] Fig.11 The lower middle figure shows that after the data obtained from all 9 channels are arranged according to the physical space position, the mean of each channel is calculated according to the 20ms window, and the heat map is drawn using the mean of each channel. Fig.11 The upper middle picture shows the eye movements of the experimental subjects. Fig.11The upper and lower figures in the middle are in a one-to-one correspondence. It can be seen that there is a corresponding relationship between the changes in the spatial characteristics of the data collected by each channel of the collection method proposed by the present invention and the eye movement of the subject.

[0105] (3) Display of eye movement classification results

[0106] Finally, the accuracy of eye movement recognition at 8 angles reached 96.8%. Fig.12 As shown. After data preprocessing, the original signal collected by the present invention can obtain a high accuracy rate in models such as linear discrimination, integrated subspace discrimination and neural network. In addition, the classification task of eye saccades in three expression states (expressionless state saccades, saccades in squinting expression state and saccades in clenching teeth expression state) in this experiment can achieve an accuracy rate of 95.1%.

[0107] User Graphical Interface: Provides users with a graphical interface to browse reports, browse records, and manage this information. It is responsible for displaying results, generating reports, and managing reports.

[0108] Fig.13 This is a use case diagram of the user interface of this embodiment.

[0109] The above-described devices and operations will be familiar to and understood by those of ordinary skill in the art.

[0110] The above description of specific examples is given by way of illustration and should not be considered as limiting. Further, many changes and modifications may be made within the scope of the embodiments of the present invention without departing from the spirit of the present invention, and the present invention includes such changes and modifications.

[0111] Compared with the prior art, the advantages of the present invention are:

[0112] 1. The present invention proposes an electrode design scheme for collecting physiological signals related to eye movement events of a subject using a multi-channel array of electrooculogram electrodes, which retains the spatial information of each electrooculogram signal and provides a more multi-dimensional signal basis for eye movement detection using a single electrode compared to the prior art;

[0113] 2. The present invention proposes an eye movement detection and analysis system solution that can detect the eye movement of the subject by using only one electrode placed around the eye socket of the subject to collect the electrooculogram. This makes the detection setting simpler and introduces less human operation errors;

[0114] 3. The present invention proposes a signal preprocessing and signal analysis scheme that separates the electrooculogram signal, other eye movement related physiological electrical signals (such as electromyography signals, electroencephalogram signals, etc.) and noise signals obtained through a multi-channel array electrooculogram electrode. The acquisition method, signal processing, and intelligent algorithm provided by the present invention expand the collected signals from the detection of simple eye movement to the detection of related conditions during eye movement. Therefore, compared with the existing eye movement method that can only detect eye movement, the method provided by the present invention has a wider range of application scenarios.

Claims

1. An eye movement detection and analysis system based on a multi-channel array electrooculogram electrode, characterized in that: It consists of two parts: the acquisition module and the lower computer and the upper computer. The two parts transmit data through wired and wireless transmission equipment. Among them, the acquisition module and the lower computer part include the front-end acquisition equipment and signal processing hardware circuit; the upper computer part is the user client computer pre-loaded with two programs: signal analysis and information display and communication; the client collects and analyzes the information received from the lower computer and displays the results in the graphical user interface; among them: The front-end acquisition device is used to collect physiological electrical signals; it includes: a multi-channel array electrooculogram electrode, a reference electrode and a ground electrode; the ground electrode is arranged at the mastoid of the left or right ear of the subject, and the reference electrode is arranged at the center of the eyebrows; the multi-channel array electrooculogram electrode is arranged at the epidermis around the user's eye socket; the multi-channel array electrooculogram electrode is an electrooculogram acquisition array containing a plurality of contacts, and the number of contacts integrated on a single electrode is not less than 9; The signal processing hardware circuit is used to process the physiological electrical signals collected by the front-end acquisition device; it includes: a power management module, a physiological electrical signal acquisition module and a main control module; wherein: The power management module receives the power supply and converts the power supply voltage to a voltage that can meet the needs of the main control module and the physiological electrical signal module through a series of operational amplifier chip combinations; The power management module uses the 5V power supply of the USB interface for power supply, uses the TPS73225 chip to stably generate a +2.5V voltage, and uses the TPS60403 and TPS72325 chips to stably generate a -2.5V voltage; in addition, the LP2985 chip is also used to stably generate a 3.3V voltage to meet the voltage requirements of the main control module and the physiological electrical signal module; The physiological electrical signal acquisition module performs relative differential processing on the electrical signal obtained from the front-end acquisition device to obtain the potential signal of this channel; the obtained signal is provided to the main control module for further processing; The physiological electrical signal acquisition module adopts an analog front-end ADS1299 chip, in which all analog front-ends share a common ground, and is used to collect the potential difference of all channels to the common ground point; The main control module is responsible for controlling signal acquisition, preliminary information processing, and integrating and sending the packaged signals to the upper computer; The main control module adopts the STM32F103C8T6 chip; the main control module performs SPI communication configuration on multiple ADS1299 chips in the physiological electrical signal acquisition module in a daisy chain manner, and sets sampling related parameters such as sampling synchronization clock, sampling frequency, analog front-end magnification, data accuracy, etc.; by reading the DRDY pin of ADS1299, the main control module detects and sends a read command to read the acquisition voltage value of each ADS1299 chip, and uploads it to the host computer; the main control module is connected to the multi-channel array eye electrode through the FPC interface in the analog signal part; the collected signal is encapsulated into a frame using the Uart protocol, and the parity check code is used to detect whether the sent data packet is abnormal; The host computer is a machine with computing, storage, processing and other functions and a human-computer interaction interface; The host computer part includes signal analysis and user graphical display interface; wherein: The signal analysis includes a signal preprocessing module, a signal analysis module and an intelligent algorithm module; wherein: The signal preprocessing module performs bandpass filtering, power frequency removal, artifact elimination, bad track detection and damaged data removal preprocessing on the original signal obtained by the main control module, and then transmits the processed signal to the signal analysis module; The signal analysis module first performs independent component analysis on the preprocessed signal to parse out the electrooculogram signal and other physiological signals related to eye movement events, and then sends the parsed signals from multiple sources to the intelligent algorithm module for further eye movement event analysis and detection; The intelligent algorithm module extracts the time domain information, frequency domain information and spatial information of the signals from the various sources analyzed, and sends them to the detection model generated based on the artificial intelligence algorithm, and finally obtains the detection results of the eye movement events; The user graphical display interface is used for information display interaction, including organizing the analysis and detection results into information that is easy for users to understand, and presenting it to the person being tested in a graphical form; the data that can be presented include: the trajectory of line of sight movement, the classification of eye movement conditions, and micro-expressions during eye movement.

2. The eye movement detection and analysis system according to claim 1, characterized in that: Several contacts on the multi-channel array electrooculogram electrode are integrated on a soft but non-ductile substrate to form a multi-channel array electrooculogram electrode; the contacts use a partial via design, and each contact is connected to the socket interface at the tail end of the collection array electrode through a copper cable with a gold-plated surface; the multi-channel array electrooculogram electrode is fixed to the skin using a double-sided tape combined with a conductive paste.

3. The eye movement detection and analysis system according to claim 1, characterized in that: In the signal analysis module, FastICA based on maximum negative entropy is used. The algorithm uses maximum negative entropy as a search direction to sequentially extract independent sources; including using FastICA to obtain electrooculogram components and myoelectric components related to expressions and movements; because the electrodes collect a mixture of signals from multiple different sources, the data obtained for a certain period of time is decentralized to make the mean of the data 0, and then the data is whitened, and then optimized and solved according to the maximum negative entropy principle to obtain independent components in the data; then the kurtosis tool is used to distinguish the electrooculogram components and the myoelectric components to obtain a more accurate signal feature expression, so as to prepare data for subsequent different tasks.

4. The eye movement detection and analysis system according to claim 3, characterized in that: The intelligent algorithm module includes an electrooculogram signal processing submodule and an expression-related signal processing submodule; The electrooculogram signal processing submodule is used to process the electrooculogram signal, including downsampling, feature discriminator, and voting decision. This submodule processes the multi-channel electrooculogram signal to obtain the eye movement direction classification corresponding to the signal. The expression-related signal processing submodule is used to process expression-related signals, including feature extraction and integrated learning. This module analyzes and processes the electromyographic signals and other signals in the expression-related components to obtain the expression and emotion classification corresponding to the signal.

5. The eye movement detection and analysis system according to claim 4, characterized in that: The expression-related signal processing submodule processes expression-related signals, including extracting three types of features, namely, RMS, Mean and SSC, of ​​n channels after low-pass filtering of the electromyographic signal components from the signal analysis module, and converting each action into a column of 3*n dimensional feature vectors; after all action features of multiple types of expressions are extracted, a model is trained through an integrated learning classifier using ten-fold cross validation to obtain a final classification; the three types of features used: RMS reflects the power of the signal, Mean describes the average amplitude of the signal, and SSC reflects the force of the muscle, providing physiological information for distinguishing the three types of expression actions.

6. The eye movement detection and analysis system according to claim 4, characterized in that: The electrooculogram signal processing submodule processes the electrooculogram signal and performs four-fold downsampling on the electrooculogram component signal found; the downsampled data is sent to a linear discriminator for training to obtain a discriminant model; In the decision-making stage, the channel-based linear discrimination results are used for voting, and the n data obtained by the multi-channel array electrooculogram electrodes of n channels are linearly discriminated. The category with the most channel data discrimination results is selected as the final discrimination result.

7. The eye movement detection and analysis system according to any one of claims 1 to 6, characterized in that: The acquisition module and various parts of the lower computer are arranged in a wearable headband structure, which includes two parts: a headband hardware circuit placement frame (g) and an adjustable head frame head support (i); wherein: The headband hardware circuit placement frame (g) is divided into three parts: a left placement frame (g-1), a right placement frame (g-2) and a front forehead guard (g-3); the right placement frame (g-2) is used to install a collection unit composed of a multi-channel array electrooculogram electrode (e) and a physiological electrical signal collection module (d), as well as a main control module (k), a transmission, control and feedback unit composed of a headband control button (m), an operating status indicator light (a) and a USB interface (n); the left placement frame (g-1) is used to install a supporting circuit composed of a battery (l), a power management module (f) and a ground electrode (h); the front forehead guard (g-3) is used to install a reference electrode (b); all electrodes are equipped with insulators at positions near the skin. Silicon base (c); all electronic components are wrapped in a circuit protection shell (j); there are three adjustable head frame head supports (i), distributed above the forehead, above the back of the head and at the back of the head; the adjustable head frame head support (i) is an adjustable design, consisting of an adjustable metal frame (i-1) and a head support (i-2); by adjusting the length of the metal frame of the head support (i-2) in the head support, the length of the adjustable head frame head support (i) can be adjusted to adapt to the head size of different subjects; the headband composed of the headband hardware circuit placement frame (g) and the adjustable headframe head support (i) can provide better inward pressure, so that the sensor units on the inner side of the headband are better pressed onto the skin surface of the subject; front forehead protector (g-3) The reference electrode (b) located in the middle and the multi-channel array electrooculogram electrode (e) located at the outer corner of the right eye are mounted on an insulating silicon base (c), and the electrode contacts are slightly higher than the surface of the silicone base; three insulating silicone bases (c) ensure that the electrode contacts have good contact with the skin.

Images