A Method and System for Threat Target Recognition in Driving Environment Based on EEG and Eye-Motion Coordination

Abstract

This invention discloses a method and system for identifying threat targets in a driving environment based on the coordination of electroencephalography (EEG) and eye movement. The method includes the following steps: S1, acquiring real-time video of the road conditions in the foreground and performing multi-target detection and tracking; S2, displaying flashing arrows of different stimuli on the target and following its movement, collecting eye movement data and EEG data; S3, processing the eye movement data to obtain eye movement results, and processing the EEG data to obtain EEG results; S4, combining the eye movement results and EEG results to output the target tracking result. This invention has advantages such as reducing the false positive rate, shortening the target recognition time, and improving the recognition accuracy.

Description

Technical Field

[0001] This invention mainly relates to the field of target recognition technology, specifically to a method and system for identifying threat targets in a driving environment based on EEG and eye-tracking coordination. Background Technology

[0002] In recent years, the rapidly developing brain-computer interface (BCI) has become a research hotspot in the field of artificial intelligence. BCI aims to establish a direct pathway between the human brain and external devices. As a novel interaction mode, BCI has been widely applied in fields such as medical assistance, vehicle driving, and robot control.

[0003] Brain-controlled vehicles are complex applications where a direct control pathway exists between the human brain and the vehicle. Currently, the main brain-computer interface paradigms used in brain-controlled vehicles include P300, motor imagery, and steady-state visual evoked potentials (SSVEP). P300 is typically induced by visual stimuli, has poor real-time performance, and can only be used to control static targets such as switches and windshield wipers. Motor imagery also has poor real-time performance and limited degrees of freedom (generally less than four), making it unable to complete overall driving tasks. SSVEP has a high information transfer rate (ITR) and good real-time performance. When subjected to a visual stimulus of a fixed frequency, the cerebral cortex generates SSVEP characteristic components at the fundamental or harmonic frequencies of the target stimulus. The target stimulus can be identified by detecting the dominant frequency of the SSVEP. Based on its strong applicability, ease of access, and high accuracy, brain-computer interfaces using SSVEP are beneficial for selecting threatening targets during autonomous driving. However, prolonged flashing stimulation can easily cause driver fatigue.

[0004] With the maturation of eye-tracking technology and the increasing demands for comfortable interaction, eye-tracking-based interaction methods are receiving growing attention. Unlike EEG, eye-tracking interaction is more natural and can further reduce user fatigue. Furthermore, eye-tracking interaction has a low learning curve, and most users can operate it without specialized training. However, eye-tracking still has some drawbacks. In natural states, humans often exhibit eye movements that are not guided by conscious attention. If the system fails to distinguish these eye movements, it may misinterpret human intentions, leading to false triggering, known as the "Middlespot contact" problem. Additionally, eye-tracking technology is not entirely reliable; random and unstable factors can cause system errors. Some eye-tracking-based interactions have already been applied to text spelling and robot control.

[0005] A hybrid brain-computer interface (BCI) system typically consists of at least two modalities and outperforms a single-modality BCI. Some studies have utilized hybrid systems combining EEG and EEG to recognize characters. Furthermore, the combination of eye tracking and BCI is increasingly being applied to controlling games, robotic arms, and drones.

[0006] Currently, the significant improvement in computer information fusion capabilities is continuously promoting the development of autonomous driving. Autonomous driving is gradually moving from specific scenarios (such as highways and test parks) to complex urban traffic. Urban traffic conditions are relatively complex, with a large number of dynamic pedestrian targets and their trajectories constantly changing. In such complex road conditions, environmental perception methods based on computer vision technology cannot quickly and accurately predict threatening targets. By integrating the driver's intentions into the vehicle's environmental perception through brain-computer interfaces, driving comfort and safety can be improved.

[0007] Humans often exhibit eye movements naturally that are not triggered by conscious attention. If the system cannot distinguish between conscious and unconscious eye movements, it may misinterpret the true intention and issue incorrect commands, potentially leading to traffic accidents, especially in driving environments. Therefore, the system should be capable of detecting unconscious eye movements to reduce false positives and enhance its safety and reliability.

[0008] In designing online asynchronous brain-computer interface (BCI) systems, the key to achieving asynchronous control lies in distinguishing between idle and active states. In an asynchronous BCI system, the subject sends commands to the external device based on their own choice. The system enters an active state upon receiving the subject's control intention, and remains idle when no control intention is detected. The system needs to continuously monitor the subject's brain activity to differentiate between active and idle states. Summary of the Invention

[0009] The technical problem to be solved by this invention is: in view of the technical problems existing in the prior art, this invention provides a method and system for identifying threatening targets in the driving environment based on EEG and eye movement coordination, which reduces the false positive rate, shortens the target recognition time, and improves the recognition accuracy.

[0010] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0011] A method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination includes the following steps:

[0012] S1. Real-time acquisition of road condition video of the driving scene and multi-target detection and tracking;

[0013] S2. Present flashing arrows of different stimuli on the target and follow their movement to collect eye movement data and EEG data;

[0014] S3. Process the eye-tracking data to obtain eye-tracking results, and process the EEG data to obtain EEG results;

[0015] S4. Based on the combined eye-tracking and EEG results, output the target tracking result.

[0016] Preferably, in step S3, the speed of eye movement is analyzed by an I-VT filter to classify eye movements and output eye movement results; wherein the classification of eye movements includes fixation and saccades.

[0017] Preferably, in step S3, the specific process of classifying eye movements is as follows:

[0018] Eye movement velocity is obtained by the ratio of the distance between two sampling points to the corresponding sampling time;

[0019] When the eye movement speed is higher than the set eye movement threshold, the sample related to that speed is determined to be a saccade sample, and when it is lower than the threshold, it is considered to be part of the fixation sample.

[0020] Preferably, in step S3, the EEG data is feature extracted and classified using a canonical correlation analysis algorithm, multi-channel signal data is fused, and the target is identified by calculating the maximum correlation coefficient between the multi-channel EEG signal and the stimulation frequency.

[0021] Preferably, the process for obtaining the maximum correlation coefficient is as follows:

[0022] Periodic stimulation is represented as a square wave periodic signal, which is decomposed into a Fourier harmonic series:

[0023]

[0024] Where N is the harmonic number, L is the number of sampling points of the original signal, t is the current time point, and S is the sampling rate of the EEG; by calculating the linear combination of each variable of the two data sets (X,Y) (x=X T W x y = Y T W y The maximum correlation coefficient ρ is used to reflect the correlation between the two sets of signals; the formula for calculating ρ is as follows:

[0025]

[0026] Preferably, in step S4, when The target is tracked and output in real time; where {r1,r2…} are eye-tracking results and {s1} are EEG results.

[0027] Preferably, in step S2, flashing stimuli of different frequencies, colors, and directions are superimposed on different targets.

[0028] Preferably, the different frequencies are 6.10Hz, 8.18Hz, 15.87Hz, 12.85Hz, 10.50Hz, 8.97Hz, 13.78Hz, 9.98Hz, 11.23Hz, 7.08Hz, 14.99Hz and 11.88Hz.

[0029] Preferably, after step S4, the performance of target recognition is evaluated by using accuracy and signal transmission rate.

[0030] The present invention also discloses a vehicle environment threat target recognition system based on EEG and eye movement coordination, including a memory and a processor. The memory stores a computer program, which executes the steps of the method described above when run by the processor.

[0031] Compared with the prior art, the advantages of the present invention are as follows:

[0032] This invention integrates steady-state visual evoked potentials (SSVEP) and eye-tracking information to identify potentially threatening pedestrian targets in a driving environment. It uses a deep learning-based target detection and tracking method to obtain the coordinates and IDs of pedestrian targets, providing initial information for a hybrid brain-computer interface (BCI). Arrows flashing in different directions are randomly superimposed on the pedestrian targets. While subjects scan for threatening targets based on the arrows, stimuli of corresponding frequencies evoke SSVEP. Using a traffic scene as the background and dynamic pedestrians as targets, the invention combines computer vision with hybrid BCI. Subjects are required to judge and select pedestrians who pose a threat to driving safety based on their subjective experience, thus achieving brain-computer fusion recognition of threatening pedestrian targets based on BCI.

[0033] The identification method of the present invention effectively avoids erroneous results caused by lack of concentration by incorporating eye tracking, thereby reducing the false positive rate. In addition, the combination of eye tracking and EEG can not only be used to distinguish between working and idle states, but also shorten the identification time of threat targets and improve the accuracy. Attached Figure Description

[0034] Figure 1 This is a flowchart of an embodiment of the target recognition method of the present invention.

[0035] Figure 2 This is a schematic diagram of the target superimposed flashing arrow in this invention.

[0036] Figure 3 This is a schematic diagram of the electrode distribution during EEG detection in this invention (electrodes are selected in dark colors). Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1 As shown, this embodiment of the invention provides a method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination, including the following steps:

[0039] S1. Real-time acquisition of road condition video of the driving foreground using ZED2 binocular camera and multi-target detection and tracking. The acquired image frames and the coordinates and categories of the targets are transmitted to a remote computer via local area network.

[0040] S2. After the remote computer receives the data, flashing arrows for different stimuli appear on the target and follow its movement; the arrows are randomly distributed; the subject calibrates the eye tracker to scan in the direction of the arrows of the threatening target, and the eye tracker and EEG acquisition instrument start collecting the corresponding eye movement data and EEG data simultaneously; the sampling frequency of the eye tracker is 60Hz.

[0041] S3. Eye-tracking data is processed to obtain eye-tracking results, and EEG data is processed to obtain EEG results. In the eye-tracking data processing, an I-VT filter is introduced to process the visual trajectory. Specifically, when the confidence level of the trajectory change over 60 consecutive sampling points (sp) exceeds a certain value (e.g., 70%), the eye-tracking results {r1, r2…} (corresponding to…) are output. Figure 1 Decision-making in the process (I);

[0042] In processing EEG data, the canonical correlation analysis (CCA) algorithm is used to extract features from EEG data over a certain period (e.g., 1000ms) and output the maximum correlation coefficient ρ; when ρ exceeds a set threshold, the EEG selection result {s1} is output (corresponding to...). Figure 1 Decision-making in (II);

[0043] S4. Combining the eye-tracking and EEG results, output the tracking target result; specifically, when there is an overlap between the eye-tracking output result and the EEG output result, output the tracking target (corresponding to...). Figure 1 Decision III in the process, namely The system outputs the tracking target (i.e., the target threatening the driving environment) in real time; otherwise, no result is output and it is considered to be in an idle state; the window slides forward for a certain period of time (e.g., 200ms) to obtain the eye movement data and EEG data of the next second for processing until the target result is output.

[0044] This invention integrates steady-state visual evoked potentials (SSVEP) and eye-tracking information to identify potentially threatening pedestrian targets in a driving environment. It uses a deep learning-based target detection and tracking method to obtain the coordinates and IDs of pedestrian targets, providing initial information for a hybrid brain-computer interface (BCI). Arrows flashing in different directions are randomly superimposed on the pedestrian targets. While subjects scan for threatening targets based on the arrows, stimuli of corresponding frequencies evoke SSVEP. Using a traffic scene as the background and dynamic pedestrians as targets, the invention combines computer vision with hybrid BCI. Subjects are required to judge and select pedestrians who pose a threat to driving safety based on their subjective experience, thus achieving brain-computer fusion recognition of threatening pedestrian targets based on BCI.

[0045] The identification method of the present invention effectively avoids erroneous results caused by lack of concentration by incorporating eye tracking, thereby reducing the false positive rate. In addition, the combination of eye tracking and EEG can not only be used to distinguish between working and idle states, but also shorten the identification time of threat targets and improve the accuracy.

[0046] like Figure 2 As shown, in step S2, the perceived target location and ID are obtained based on target detection and tracking. Flashing stimuli of different frequencies and directions are superimposed on each pedestrian target, and the subject selects the target by gazing at it. The length of the alternating black and white flashing arrow is 60 pixels. Since the number of pedestrians in the video is not fixed, a frequency list is set. Related experimental studies have shown that the frequency range of 8–15 Hz can induce a relatively strong SSVEP response. Furthermore, each frequency must not overlap at the fundamental frequency and its harmonics, and the interval between frequencies should be set as large as possible to ensure signal distinguishability. Considering these factors, the frequency list is set to 6.10 Hz, 8.18 Hz, 15.87 Hz, 12.85 Hz, 10.50 Hz, 8.97 Hz, 13.78 Hz, 9.98 Hz, 11.23 Hz, 7.08 Hz, 14.99 Hz, and 11.88 Hz. The frequencies of the superimposed stimuli are selected sequentially from the frequency list according to the encoding order of each pedestrian target ID. During the experiment, the participants identified the threatening target and scanned the target according to the direction of the stimulus arrow until the target was selected, at which point the superimposed flashing stopped and turned yellow.

[0047] In one specific embodiment, the eye-tracking data acquisition process in step S2 is as follows: eye-tracking data is acquired using Tobii's low-cost eye tracker (Tobii Pro Nano) at a frequency of 60 Hz and an operating distance of 75 cm. Before each experiment, the subject needs to calibrate the eye tracker. The LCD screen used (LEGION Y27gq-25, resolution set to 1920×1080 pixels) has a refresh rate of 240Hz.

[0048] The EEG data acquisition process in step S2 is as follows: The experiment uses a 64-channel extended international 10 / 20 system to record EEG signals, and selects 9 active electrodes for SSVEP detection, such as... Figure 3As shown, electrodes were placed at Pz, PO7, PO3, POz, PO4, PO8, O1, Oz, and O2, respectively. The reference electrode was placed behind the right ear, and the ground electrode was placed on the forehead. Before acquisition, the impedance of each electrode was kept below 10kΩ. The EEG signal was amplified using a BrainAmp DC Amplifier (Brain Products GmbH, Germany). The sampling frequency was 200Hz, and the signal was filtered through a 4-35Hz bandpass filter and a 50Hz notch filter. The BCI2000 was used as the control platform for signal acquisition, the Python extension package PyGame was used to present the stimulation interface, and MATLAB was used for real-time signal processing. The display interface and the control platform were connected via TCP / IP protocol.

[0049] In one specific embodiment, in step S3, the specific process of EEG data processing is as follows: Feature extraction and classification of the preprocessed EEG signals are performed using the canonical correlation analysis (CCA) algorithm. This method integrates multi-channel signal data and identifies targets by calculating the correlation coefficient between the multi-channel EEG signals and the stimulation frequency, where the target is the option corresponding to the maximum SSVEP response score. Periodic stimulation is represented as a square wave periodic signal, which can be decomposed into Fourier harmonic series:

[0050]

[0051] Where N is the harmonic number, L is the number of sampling points of the original signal, t is the current time point, and S is the sampling rate of the EEG. CCA is a multivariate statistical analysis method that calculates the linear combination of variables (x = X) of two data sets (X, Y). T W x y = Y T W y The maximum correlation coefficient ρ is used to reflect the correlation between the two sets of signals.

[0052] The formula for calculating ρ is as follows:

[0053]

[0054] The specific process of eye-tracking data processing is as follows: The velocity threshold identification (I-VT) filter is a popular velocity-based eye-tracking method. The I-VT filter classifies eye movements by analyzing the speed of eye movement in a specific direction. The eye velocity is obtained by the ratio of the distance between two sampling points on the eye tracker to the corresponding sampling time. Velocity is usually expressed in visual degrees per second (° / s). A velocity threshold is set; when the velocity is higher than the threshold, samples related to that velocity are classified as saccades; when it is lower than the threshold, it is considered part of a fixation. For example:

[0055]

[0056] Where v x V represents the velocity in the x-direction. y x represents the velocity in the y-direction, x1 and y1 are the position coordinates of the eyeball at time t1, and x2 and y2 are the position coordinates of the eyeball at time t2.

[0057] In one specific embodiment, accuracy and signal transmission rate are used to evaluate the performance of the SSVEP-BCI selection; wherein the signal transmission rate (ITR) is calculated using the following formula (unit: bits per minute):

[0058]

[0059] T = t s +t b

[0060] Where N represents the total number of targets, P is the target selection accuracy, and T represents the time to complete the sequential target selection, including the stimulus flashing time t for target selection. s and flashing interval t b .

[0061] This invention also discloses a vehicle environment threat target recognition system based on EEG and eye-tracking coordination, including a memory and a processor. The memory stores a computer program, which, when run by the processor, executes the steps of the method described above. The target recognition system of this invention corresponds to the target recognition method described above and also possesses the advantages described therein.

[0062] As shown in this disclosure and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "comprising" or "including" mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, without excluding other elements or objects. The terms "connected" or "linked" are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect.

[0063] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination, characterized in that, Including the following steps: S1. Real-time acquisition of road condition video of the driving scene and multi-target detection and tracking; S2. Present flashing arrows of different stimuli on the target and follow their movement to collect eye movement data and EEG data; S3. Process the eye-tracking data to obtain eye-tracking results, and process the EEG data to obtain EEG results; S4. Based on the combined eye-tracking and EEG results, output the target tracking result; In step S3, the speed of eye movement is analyzed using an I-VT filter to classify eye movements and output eye movement results; the classification of eye movements includes fixation and saccades. In step S3, the specific process of classifying eye movements is as follows: Eye movement velocity is obtained by the ratio of the distance between two sampling points to the corresponding sampling time; When the eye movement speed is higher than the set eye movement threshold, the sample related to that speed is judged as a saccade sample, and when it is lower than the threshold, it is regarded as part of the fixation sample. In step S3, the canonical correlation analysis algorithm is used to extract and classify the features of the EEG data, fuse the multi-channel signal data, and identify the target by calculating the maximum correlation coefficient between the multi-channel EEG signal and the stimulation frequency.

2. The method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination according to claim 1, characterized in that, The process of obtaining the maximum correlation coefficient is as follows: Periodic stimulation is represented as a square wave periodic signal, which is decomposed into a Fourier harmonic series: in N It is the harmonic number. L It is the number of sampling points of the original signal. t It is the current time point. S It is the sampling rate of the EEG; by calculating the linear combination of variables in two data sets (X, Y) x = X T W x , y = Y T W y Maximum correlation coefficient ρ To reflect the correlation between the two sets of signals; ρ The calculation formula is as follows: 。 3. The method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination according to claim 1 or 2, characterized in that, In step S4, the tracking target is output when {r1, r2…}∩{s1}≠ø; where {r1, r2…} is the eye-tracking result and {s1} is the EEG result.

4. The method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination according to claim 1 or 2, characterized in that, In step S2, flashing stimuli of different frequencies and directions are superimposed on different targets.

5. The method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination according to claim 4, characterized in that, The different frequencies are 6.10Hz, 8.18Hz, 15.87Hz, 12.85Hz, 10.50Hz, 8.97Hz, 13.78Hz, 9.98Hz, 11.23Hz, 7.08Hz, 14.99Hz and 11.88Hz.

6. The method for identifying threat targets in a driving environment based on EEG and eye-tracking coordination according to claim 1 or 2, characterized in that, After step S4, the performance of target recognition is evaluated by using accuracy and signal transmission rate.

7. A vehicle environment threat target recognition system based on EEG and eye-tracking coordination, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The computer program, when run by a processor, performs the steps of the method as described in any one of claims 1 to 6.

Images