A data fusion method and system based on neural pathways
By collecting and fusing EEG and eye-tracking data and using cross-correlation analysis, the problem of insufficient eye-tracking ability assessment in existing technologies has been solved, enabling comprehensive assessment of dynamic visual acuity and early screening of neurological diseases.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI AN JIAOTONG UNIV
- Filing Date
- 2026-04-16
- Publication Date
- 2026-05-26
AI Technical Summary
Existing eye-tracking technologies focus only on eye movement control capabilities, ignoring the impact of visual input on eye movement control. This results in an inability to fully reflect complex control structures and capabilities when assessing dynamic visual acuity, and a lack of fusion of visual evoked potential information.
By collecting EEG and eye-tracking data, performing timestamp alignment and preprocessing, extracting EEG and eye-tracking features, using cross-correlation analysis for feature fusion and synergy assessment, and combining the neural structure of the eye-tracking control pathway, a brain state calculation method with time characteristics is extracted.
It provides a more comprehensive dynamic visual acuity assessment, filling the gap of insufficient eye movement information and enhancing the objective basis for the diagnosis and monitoring of neurological diseases, especially for the early screening of neurodegenerative diseases.
Smart Images
Figure CN122075016A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data fusion method and system based on neural pathways. Background Technology
[0002] Smooth eye-tracking ability is an important tool for assessing dynamic visual acuity and cognitive impairment in neurodegenerative diseases. Trajectory tracking tasks and smooth tracking tasks have become typical research paradigms for screening abnormal groups. As a neuropsychological tool combined with eye-tracking technology, it can effectively detect neuropsychological disorders. The study commonly uses indicators such as velocity gain (the ratio of eye movement velocity to target movement velocity), tracking duration, and the number of tracking interruptions for quantitative analysis.
[0003] Velocity gain (the ratio of eye movement speed to target movement speed), tracking duration, number of tracking interruptions, and saccade latency—these quantitative indicators provide objective evidence for the tracking ability of dynamic visual acuity, and for the diagnosis and monitoring of neurological diseases. Eye-tracking technology is playing an increasingly important role in the field of neurology by providing quantifiable biomarkers.
[0004] However, smooth eye-tracking refers to the eye movements that follow a slowly moving target to bring it into the fovea position in order to ensure precise perception of the moving target. This eye movement behavior requires the brain to generate "predictive signals" from the visual information obtained during the tracking process.
[0005] Besides eye-tracking metrics, only the fusion of visual information can better reflect complex control structures and abilities when assessing dynamic visual acuity. While functional magnetic resonance imaging (fMRI) or transcranial magnetic stimulation (TMS) provides high spatial location information, visual evoked potentials (VEPs) offer superior temporal resolution, making them more suitable for analyzing eye-tracking tasks in assessing dynamic visual acuity. Related research indicates that the primary region for VEP generation is the V1 region in the occipital lobe of the brain, and existing TMS evidence suggests that the middle temporal visual area (MT) in the needle lobe is also involved in movement-related tasks. Therefore, relying solely on eye-tracking information is insufficient to fully reflect tracking ability; thus, visual evoked potential information needs to be incorporated into the metrics. Summary of the Invention
[0006] This application provides a data fusion method and system based on neural pathways, which utilizes cross-correlation analysis to fuse EEG features and eye movement features, and has interpretability at the level of eye movement control pathways.
[0007] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a data fusion method based on neural pathways, comprising the following steps: First, collecting electroencephalogram (EEG) data and eye-tracking data, aligning them using timestamps, and preprocessing them separately; then, extracting EEG features based on the preprocessed EEG data; extracting eye-tracking features based on the preprocessed eye-tracking data; next, fusion of features based on the EEG features and eye-tracking features by aligning the data using downsampling; and finally, evaluating the synergy of the fused features.
[0008] In some exemplary embodiments, collecting EEG data and eye-tracking data includes: placing EEG cap electrodes at selected channel locations, providing the subject with a visual paradigm consistent with motor characteristics, and collecting eye-tracking data and EEG data according to the BCI procedure.
[0009] In some exemplary embodiments, preprocessing of EEG data includes performing preprocessing operations such as detrending, power frequency notch filtering, bandpass filtering, and standardization on the acquired EEG data in sequence.
[0010] In some exemplary embodiments, the eye-tracking data is preprocessed, including: sequentially performing data alignment, blink recognition, and data completion operations on the acquired eye-tracking data.
[0011] In some exemplary embodiments, based on preprocessed EEG data, EEG features are extracted, including: Confirm the window length and movement step size; (1) in, L Indicates the window length in seconds; The sampling frequency of EEG data; Indicates the step size; Within each time window HA Parameter calculation, based on L and S The windowed EEG signals are represented as follows: (2) in , N yes EEG Data length; The following formula is used to calculate the first... k The window : (3) Then slide the window and calculate the first... k +1 window Values, and so on, are obtained. STHAThe formula is as follows: (4) in, This represents the average value for each time window.
[0012] In some exemplary embodiments, eye movement features are extracted based on preprocessed eye movement data, including: calculating the Euclidean distance from the eye to the target based on the preprocessed eye movement data to obtain eye movement features.
[0013] In some exemplary embodiments, the formula for calculating the Euclidean distance from the eye to the target is: (5) in, and The coordinates of the eye. and The coordinates of the stimulus target.
[0014] In some exemplary embodiments, feature fusion is performed on data aligned by downsampling based on EEG and eye-tracking features, as shown in the following formula: (6) Among them, discrete signals and The results are obtained by formulas (4) and (5) respectively; n discrete signal STHA Length; Time lag represents the time difference between events or variables, and is used to measure the nonlinear dependence between variables in EEG signal features and multiscale mutation detection.
[0015] In some exemplary embodiments, cross correlation is used to evaluate the synergy of the fused features; cross correlation is used to maximize the observation of the temporal difference in the correlation between two discrete or continuous sequence signals, and the temporal difference is regarded as binocular coordination in the time dimension, as defined below: (7) in, and They are obtained respectively through formula (6); and The time at which the two discrete sequences are most correlated is... The moment .
[0016] Secondly, embodiments of this application also provide a data fusion system based on neural pathways. This system is used to implement the data fusion method based on neural pathways as described in the above embodiments. The system includes: a data acquisition module, a preprocessing module, a feature extraction module, a feature fusion module, and a synergy evaluation module connected in sequence. The data acquisition module is used to acquire EEG data and eye-tracking data. The preprocessing module is used to align the acquired EEG data and eye-tracking data using timestamps and preprocess them separately. The feature extraction module is used to extract EEG features from the preprocessed EEG data and to extract eye-tracking features from the preprocessed eye-tracking data. The feature fusion module is used to perform feature fusion by aligning the data using downsampling based on the EEG features and eye-tracking features. The synergy evaluation module is used to perform a synergy evaluation on the fused features.
[0017] The technical solution provided in this application has at least the following advantages: This application provides a data fusion method and system based on neural pathways. The method includes the following steps: First, collecting EEG data and eye movement data, aligning them by timestamps, and preprocessing them separately; then, extracting EEG features based on the preprocessed EEG data; extracting eye movement features based on the preprocessed eye movement data; next, merging features based on the EEG features and eye movement features by downsampling the data; finally, evaluating the synergy of the fused features.
[0018] This application, based on the physiological structure of the eye-tracking control pathway, utilizes eye-tracking and EEG data to obtain a reliable method for evaluating smooth eye-tracking through cross-correlation to obtain fused signals. This fills a gap in the analysis of neural structure-related methods for characterizing smooth eye-tracking ability using EEG signal features. The selection of cross-correlation results emphasizes the correlation between eye-tracking and EEG, as well as the time delay at the most correlated point. This difference in indicators has physiological structural interpretability, constituting a novel evaluation index representing brain-eye coordination when assessing dynamic visual acuity. This application comprehensively considers both eye-tracking signals and EEG data as features of eye-tracking ability, and starts from the neural structure of the eye-tracking control pathway. By proposing a brain state calculation method with temporal characteristics, compared to EEG such as fMRI which has high spatial resolution but only low temporal resolution, this application preserves the temporal features of both EEG and eye-tracking to the greatest extent. Based on these two features, this application proposes a sliding window-based method to extract EEG features, and selects cross-correlation results to fuse eye-tracking and EEG features. This fusion method conforms to the neural control characteristics in the neural control pathway and has strong interpretability.
[0019] This application constructs a complete evaluation system by establishing a framework encompassing visual stimulation paradigms, signal acquisition platforms, and analytical methods related to neural structures. The time delay between eye movement and electroencephalogram (EEG) characteristics at the most relevant points in the time domain is used as an indicator of brain-eye synergy. The difference between these indicators is interpretable by physiological structures, constituting a novel evaluation index for brain-eye synergy when assessing dynamic visual acuity. Attached Figure Description
[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0021] Figure 1 The neural control pathway provided in one embodiment of this application.
[0022] Figure 2 This is a flowchart illustrating a data fusion method and system based on neural pathways, provided as an embodiment of this application.
[0023] Figure 3 This is a schematic diagram of the hardware platform setup of an embodiment of this application.
[0024] Figure 4 This is a schematic diagram of the electrode fixing channel required in one embodiment of this application.
[0025] Figure 5 This is an embodiment of the present application used to evaluate the characteristics of brain-eye coordination as a function of movement speed. Detailed Implementation
[0026] As the background technology indicates, current assessments of eye-tracking ability focus solely on eye movement control, neglecting the influence of visual input on this control. In the field of neurology, the potential for eye movement control to reflect neurological lesions has been recognized. However, no research or related patents yet explain the underlying neurological mechanisms.
[0027] In this study, speed gain (the ratio of eye movement speed to target movement speed, smooth pursuit velocity gain, SPVG), tracking duration, and the number of tracking interruptions are commonly used for quantitative analysis.
[0028] Velocity gain, tracking duration, number of tracking interruptions, and saccade latency—these quantitative indicators provide objective evidence for the tracking ability of dynamic visual acuity, and for the diagnosis and monitoring of neurological diseases. Eye-tracking technology is playing an increasingly important role in the field of neurology by providing quantifiable biomarkers.
[0029] However, smooth eye-tracking refers to the eye movements that follow a slowly moving target to bring it into the fovea position in order to ensure precise perception of the moving target. This eye movement behavior requires the brain to generate "predictive signals" from the visual information obtained during the tracking process.
[0030] Besides eye-tracking metrics, only the fusion of visual information can better reflect complex control structures and abilities when assessing dynamic visual acuity. While functional magnetic resonance imaging (fMRI) or transcranial magnetic stimulation (TMS) provides high spatial location information, visual evoked potentials (VEPs) offer superior temporal resolution, making them more suitable for analyzing eye-tracking tasks in assessing dynamic visual acuity. Related research indicates that the primary region for VEP generation is the V1 region in the occipital lobe of the brain, and existing TMS evidence suggests that movement-related needle lobe regions also include MT / V5. Relying solely on eye-tracking information is insufficient to fully reflect tracking ability; therefore, visual evoked potential information needs to be incorporated into the metrics.
[0031] To address the aforementioned technical problems, this application provides a data fusion method and system based on neural pathways. The method includes the following steps: First, collecting electroencephalogram (EEG) data and eye-tracking data, aligning them using timestamps, and preprocessing them separately; then, extracting EEG features based on the preprocessed EEG data; extracting eye-tracking features based on the preprocessed eye-tracking data; next, merging the EEG and eye-tracking features by downsampling the data; finally, evaluating the synergy of the fused features. This application starts from the neural pathways controlling eye movement, extracting the EEG change features induced by visual signals and fusing them with eye-tracking signal features to comprehensively characterize eye-tracking ability. This application provides the possibility for dynamic visual acuity assessment and also offers a technical method for the early screening of neurodegenerative diseases such as Parkinson's disease.
[0032] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.
[0033] See Figure 1 This application provides a data fusion method based on neural pathways, comprising the following steps: Step S1: Collect EEG data and eye movement data, align them by timestamps, and preprocess them separately.
[0034] Step S2: Extract EEG features based on the preprocessed EEG data.
[0035] Step S3: Extract eye movement features based on the preprocessed eye movement data.
[0036] Step S4: Based on EEG and eye movement features, perform feature fusion by aligning the data through downsampling.
[0037] Step S5: Evaluate the synergy of the fused features.
[0038] In some embodiments, step S1, which involves collecting EEG data and eye-tracking data, includes: placing the EEG cap electrodes at selected channel locations, providing the subject with a visual paradigm that conforms to the motion characteristics, and collecting eye-tracking data and EEG data according to the BCI procedure.
[0039] In some embodiments, preprocessing of EEG data includes performing preprocessing operations such as detrending, power frequency notch filtering, bandpass filtering, and standardization on the acquired EEG data in sequence.
[0040] In some embodiments, eye-tracking data preprocessing includes: sequentially performing data alignment, blink recognition, and data completion operations on the collected eye-tracking data.
[0041] In some embodiments, step S2 involves extracting EEG features based on the preprocessed EEG data, including: Confirm the window length and movement step size; (1) in, L Indicates the window length in seconds; F s The sampling frequency of EEG data; S Indicates the step size.
[0042] Within each time window HA Parameter calculation, based on L and S The windowed EEG signals are represented as follows: (2) in , N yes EEG Data length; The following formula is used to calculate the first... k The window : (3) Then slide the window and calculate the first... k +1 window Values, and so on, are obtained. STHA The formula is as follows: (4) in This represents the average value for each time window.
[0043] In some embodiments, eye movement features are extracted based on preprocessed eye movement data, including: calculating the Euclidean distance from the eye to the target based on the preprocessed eye movement data to obtain eye movement features.
[0044] In some embodiments, the formula for calculating the Euclidean distance from the eye to the target is: (5) in, and The coordinates of the eye. and The coordinates of the stimulus target.
[0045] Specifically, step S2 extracts the short-time high-frequency activity (SEE) signal with temporal characteristics. STHA And the calculated Euclidean distance from the eye to the target. All of them are time-domain signals.
[0046] In some embodiments, feature fusion is performed on data aligned by downsampling based on EEG and eye-tracking features, as shown in the following formula: (6) Among them, discrete signals and The results are obtained by formulas (4) and (5) respectively; n discrete signal STHA Length; Time lag represents the time difference between events or variables, and is used to measure the nonlinear dependence between variables in EEG signal features and multiscale mutation detection.
[0047] In some embodiments, cross correlation is used to evaluate the synergy of the fused features; cross correlation is used to maximize the observation of the temporal difference in the correlation between two discrete or continuous sequence signals, and the temporal difference is regarded as binocular coordination in the time dimension, defined as follows: (7) in, and They are obtained respectively through formula (6); and The time at which the two discrete sequences are most correlated is... The moment .
[0048] Utilize the obtained STHA and , STHA and The cross-correlation results yielded the most relevant point. The difference in value, that is It was used to describe the temporal characteristics of brain-eye coordination.
[0049] The data fusion method provided in this application covers all channels in the occipital region of the brain and can record raw data of the coordinate positions of both eyes. Furthermore, this method is used for synchronized eye-tracking and electroencephalogram (EEG) data in the time dimension. This application extracts brain state features with temporal characteristics, rather than simply analyzing from the time or frequency domain. This method is applicable to speed ranges within the smooth eye-tracking range, i.e., 0-30 deg / s.
[0050] Furthermore, this application embodiment also provides a data fusion system based on neural pathways. This system is used to implement the data fusion method based on neural pathways as described in the above embodiments. The system includes: a data acquisition module, a preprocessing module, a feature extraction module, a feature fusion module, and a synergy evaluation module connected sequentially. The data acquisition module is used to acquire EEG data and eye-tracking data; the preprocessing module is used to align the acquired EEG data and eye-tracking data using timestamps and preprocess them separately; the feature extraction module is used to extract EEG features from the preprocessed EEG data and extract eye-tracking features from the preprocessed eye-tracking data; the feature fusion module is used to perform feature fusion by aligning the data using downsampling based on the EEG features and eye-tracking features; and the synergy evaluation module is used to perform a synergy evaluation on the fused features.
[0051] The data fusion method based on neural pathways provided in this application will be described in detail below with reference to specific embodiments.
[0052] First, the hardware components involved in this disclosure will be explained.
[0053] Figure 3 This is a data acquisition platform illustrated according to an exemplary embodiment, such as... Figure 3 As shown, the method includes providing a user with a moving visual stimulus paradigm using a display, the paradigm being able to move according to a specified direction and speed.
[0054] An eye-tracking analyzer is placed below the screen to record the user's eye movement information, including but not limited to eye position coordinates, blinks, and pupil size.
[0055] Users need to wear a non-invasive EEG cap, with electrodes arranged according to the international 10-10 standard, and at least one [electrode type] should be selected. Figure 4 The eight circled electrodes are PO7, PO8, PO3, PO2, PO4, O1, O2, and O2.
[0056] Before the experiment begins, the user needs to use a headrest to fix the height of their head, adjust the headrest height, use the eye tracker's calibration function to center both eyes on the screen, and measure the distance between the screen and the eyes. After inputting the data into the software program, the resolution of the stimulus paradigm can be recalculated based on this distance to meet the usage requirements.
[0057] Figure 2 This is a flowchart illustrating a neural pathway-based data fusion method according to an exemplary embodiment, such as... Figure 2 As shown, the method includes the following steps: 1) Acquire the user's EEG and eye-tracking data. Using the constructed hardware platform, set the sampling frequency of the eye tracker to 120Hz and the sampling frequency of the EEG device to 1200Hz. Design and provide the user with visual stimulation paradigms using Matlab software and the PhsychToolBox toolbox. Data is automatically saved to a designated folder after each program run. During data acquisition, the user's distance from the screen should be automatically synchronized to Matlab and recorded based on the eye tracker's display data.
[0058] 2) Feature extraction was performed on the EEG data to obtain brain state features, and feature extraction was also performed on the eye movement data to obtain eye movement features. First, basic bandpass and narrowband filtering was applied to the EEG data to remove 50Hz power frequency interference. The original data was then detrended using the Matlab function (command: `detrend`), and the mean was removed using the MIN-MAX method. EEG features were then extracted using the methods described in the previous steps. Eye movement features needed to be aligned and cropped with the EEG features based on the system timestamp, and then the Euclidean distance was calculated using the formula mentioned above. Since the sampling rates of eye movement and EEG signals are different, the EEG signal was down-sampled according to the eye movement sampling frequency. At this point, the lengths of the EEG and eye movement features were the same.
[0059] 3) Feature fusion of EEG and eye movement (EMG) characteristics yields target fused features, which characterize brain-eye coordination. First, cross-correlation analysis is used to analyze the correlation between the left eye and STHA (Standardized Time-of-Habitat) EEG data, and between the right eye and STHA EEG data. Since this method can reflect changes in correlation over time, the difference is calculated by selecting the time point where the two correlation results are most correlated. This difference is the physiological indicator of brain-eye coordination.
[0060] Using the above method, this application comprehensively considers the characteristics of eye movement signals and EEG data as eye movement tracking ability. Starting from the neural structure of the eye movement control pathway, this application proposes a brain state calculation method with time characteristics. Compared with the characteristics of fMRI and other EEG, which have high spatial resolution but only low temporal resolution, this application preserves the characteristics of EEG and eye movement in the time dimension to the greatest extent. This application proposes a sliding window-based method for extracting EEG features, and selects cross-correlation results to fuse eye movement and EEG features; this fusion method conforms to... Figure 1 The neural control characteristics in the study are highly interpretable.
[0061] This application constructs a complete evaluation system by establishing a framework encompassing visual stimulation paradigms, signal acquisition platforms, and analytical methods related to neural structures. The time delay between eye movement and electroencephalogram (EEG) characteristics at the most relevant points in the time domain is used as an indicator of brain-eye synergy. The difference between these indicators is interpretable by physiological structures, constituting a novel evaluation index for brain-eye synergy when assessing dynamic visual acuity.
[0062] The technical solutions of this application will be further described and illustrated below with reference to the embodiments and accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.
[0063] Reference Figure 1 , Figure 2 and Figure 3 A brain-eye tracking data fusion method based on eye-tracking neural pathways is proposed. The hardware includes an 8-channel EEG cap, an eye tracker, a display, and a headrest to collect signals. The method includes: first, collecting relevant raw data based on a task; then, selecting representative features from the raw data and confirming EEG and eye-tracking features through speed sensitivity, where EEG features are selected to represent brain states, and a time-domain EEG feature is constructed using a sliding window method for further analysis; next, calculating the correlation between the distance from the eye to the target and the EEG features using a cross-correlation method; finally, calculating the difference at the time corresponding to the most correlated point, and the result represents the temporal feature of brain-eye coordination.
[0064] The aforementioned method for fusion of EEG and eye-tracking data based on eye-tracking control neural pathways includes the following steps: Step 1) Data acquisition and preprocessing; Data acquisition is specifically as follows: For a specific visual-motor task, the EEG cap electrodes are placed in the occipital lobe region, covering the MT / V5 region; SSVEP training data is acquired according to the standard BCI (Brain Computer Interface) procedure; in this embodiment, electrode channels are set up according to a 10 / 20 electrode system, recording signals from 8 channels; 3 samples are acquired for each target, with a sampling frequency of 1200Hz, and the sample duration varies depending on the movement speed, but is at least 5 seconds; An eye tracker is placed below the display to collect eye movement data during visual-motor tasks. The eye tracker used in this embodiment has a sampling frequency of 120Hz. Five-point calibration is required for each subject before each session to ensure the accuracy of the eye movement data. Eye-tracking data preprocessing includes: aligning fixation data and EEG data at the time starting point according to the system timestamp, thereby clearly identifying the start and end points of each trial; preprocessing includes further extracting the x-axis and y-axis coordinates of the left and right eyes respectively; EEG data preprocessing includes: performing conventional preprocessing operations such as detrending, power frequency notch filtering, bandpass filtering, and standardization on the collected EEG data according to the specific task characteristics, and using independent component analysis to remove eye movement interference components and retain useful signal components; in this embodiment, detrending, 50Hz power frequency notch filtering, 2-30Hz bandpass filtering, and Z-Score standardization preprocessing operations are performed sequentially on each channel of each data sample; 2) Feature extraction from EEG data: The original eye position data may be recorded as NaN due to blinking or eye reflection; in such cases, after identifying the blink segment data, these NaN data points are replaced by linear interpolation; then, the eye position data is multiplied by the display size to convert to pixel values, since the stimulus target is also designed to move in pixel units, and then the Euclidean distance between the moving stimulus and the left or right eye can be calculated by formula (5): When tracking moving targets, EEG characteristics that reflect brain state should also be considered. Studies have shown that Hjorth parameters can effectively describe temporal brain activity with low computational cost. In addition, based on previous research, Hjorth activity parameters (HA) are sensitive to speed. Therefore, Short Time Hjorth Activity (STHA) was designed to describe changes in brain activity over time.
[0065] Then, based on EEG and eye-tracking features, feature fusion was performed by downsampling the data using formula (6). Finally, cross-correlation was used to evaluate the synergy of the fused features; using the obtained data... STHA and , STHA and The cross-correlation results yielded the most relevant point. The difference in value, that is It was used to describe the temporal characteristics of brain-eye coordination.
[0066] To verify the feasibility of the method described in this application, this embodiment utilizes signals collected from 20 subjects and implements the above steps to verify its effectiveness. (Refer to...) Figure 3 , Figure 3 These are electrodes arranged in a 10-20 system to collect EEG signals from motor-sensitive regions. The collected signals have been validated and show a significant correlation with speed. (See reference...) Figure 5 After standardizing the eye movement, EEG, and time difference information calculated in the example using the Min-Max method, the results showed that the overall trend of the correlation with speed changes was similar, which is consistent with the physiological structural characteristics based on the eye movement control pathway mentioned above.
[0067] Compared with existing technologies, the data fusion method based on neural pathways provided in this application has the following advantages: compared with traditional methods that focus solely on eye movement features or electroencephalogram (EEG) features, this invention comprehensively considers the characteristics of both features. Moreover, the starting point for feature fusion is based on neural control pathways, rather than simply fusing features directly.
[0068] Based on the above technical solutions, this application provides a data fusion method and system based on neural pathways. The method includes the following steps: First, collecting EEG data and eye movement data, aligning them by timestamps, and preprocessing them separately; then, extracting EEG features based on the preprocessed EEG data; extracting eye movement features based on the preprocessed eye movement data; next, merging features based on the EEG features and eye movement features by downsampling the data; finally, evaluating the synergy of the fused features.
[0069] This application, based on the physiological structure of the eye-tracking control pathway, utilizes eye-tracking and EEG data to obtain a reliable method for evaluating smooth eye-tracking through cross-correlation to obtain fused signals. This fills a gap in the analysis of neural structure-related methods for characterizing smooth eye-tracking ability using EEG signal features. The selection of cross-correlation results emphasizes the correlation between eye-tracking and EEG, as well as the time delay at the most correlated point. This difference in indicators has physiological structural interpretability, constituting a novel evaluation index representing brain-eye coordination when assessing dynamic visual acuity. This application comprehensively considers both eye-tracking signals and EEG data as features of eye-tracking ability, and starts from the neural structure of the eye-tracking control pathway. By proposing a brain state calculation method with temporal characteristics, compared to EEG such as fMRI which has high spatial resolution but only low temporal resolution, this application preserves the temporal features of both EEG and eye-tracking to the greatest extent. Based on these two features, this application proposes a sliding window-based method to extract EEG features, and selects cross-correlation results to fuse eye-tracking and EEG features. This fusion method conforms to the neural control characteristics in the neural control pathway and has strong interpretability.
[0070] This application constructs a complete evaluation system by establishing a framework encompassing visual stimulation paradigms, signal acquisition platforms, and analytical methods related to neural structures. The time delay between eye movement and electroencephalogram (EEG) characteristics at the most relevant points in the time domain is used as an indicator of brain-eye synergy. The difference between these indicators is interpretable by physiological structures, constituting a novel evaluation index for brain-eye synergy when assessing dynamic visual acuity.
[0071] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.
Claims
1. A data fusion method based on neural pathways, characterized in that, Includes the following steps: EEG and eye-tracking data were collected, aligned using timestamps, and preprocessed separately. Based on the preprocessed EEG data, EEG features are extracted; Based on the preprocessed eye-tracking data, eye-tracking features are extracted; Based on EEG and eye-tracking features, feature fusion is performed by aligning data through downsampling. A synergistic evaluation is performed on the fused features.
2. The data fusion method based on neural pathways according to claim 1, characterized in that, Collecting EEG and eye-tracking data includes: placing the EEG cap electrodes at selected channel locations, providing the subject with a visual paradigm consistent with the motor characteristics, and collecting eye-tracking and EEG data according to the BCI procedure.
3. The data fusion method based on neural pathways according to claim 1, characterized in that, Preprocessing of EEG data includes performing preprocessing operations such as detrending, power frequency notch filtering, bandpass filtering, and standardization on the acquired EEG data.
4. The data fusion method based on neural pathways according to claim 1, characterized in that, Preprocessing of eye-tracking data includes: sequentially aligning the collected eye-tracking data, recognizing blinks, and performing data completion operations.
5. The data fusion method based on neural pathways according to claim 1, characterized in that, Based on the preprocessed EEG data, EEG features are extracted, including: Confirm the window length and movement step size; （1） in, L Indicates the window length in seconds; The sampling frequency of EEG data; Indicates the step size; Within each time window HA Parameter calculation, based on L and S The windowed EEG signals are represented as follows: （2） in , N yes EEG Data length; The following formula is used to calculate the first... k The window : （3） Then slide the window and calculate the first... k +1 window Values, and so on, are obtained. STHA The formula is as follows: （4） in, This represents the average value for each time window.
6. The data fusion method based on neural pathways according to claim 5, characterized in that, Based on the preprocessed eye-tracking data, eye-tracking features are extracted, including: calculating the Euclidean distance from the eye to the target based on the preprocessed eye-tracking data to obtain eye-tracking features.
7. The data fusion method based on neural pathways according to claim 6, characterized in that, The formula for calculating the Euclidean distance from the eye to the target is: （5） in, and The coordinates of the eye. and The coordinates of the stimulus target.
8. The data fusion method based on neural pathways according to claim 7, characterized in that, Based on EEG and eye-tracking features, feature fusion is performed by aligning the data through downsampling, as shown in the following formula: （6） Among them, discrete signals and The results are obtained by formulas (4) and (5) respectively; n discrete signal STHA Length; Time lag represents the time difference between events or variables, and is used to measure the nonlinear dependencies between variables in EEG signal features and multiscale mutation detection.
9. The data fusion method based on neural pathways according to claim 8, characterized in that, Cross-correlation is used to perform a synergistic evaluation of the fused features; Cross-correlation is used to maximize the observation of the temporal difference in the correlation between two discrete or continuous signal sequences. This temporal difference is considered as binocular coordination in the time dimension and is defined as follows: （7） in, and They are obtained respectively through formula (6); and The time at which the two discrete sequences are most correlated is... The moment .
10. A neural pathway-based data fusion system, wherein the system is used to implement the neural pathway-based data fusion method as described in any one of claims 1 to 9, characterized in that, The system includes: a data acquisition module, a preprocessing module, a feature extraction module, a feature fusion module, and a collaborative evaluation module, connected in sequence; among them, The data acquisition module is used to collect electroencephalogram (EEG) data and eye movement data; The preprocessing module is used to align the collected EEG data and eye movement data using timestamps and to preprocess them separately. The feature extraction module is used to extract EEG features based on the preprocessed EEG data; and to extract eye movement features based on the preprocessed eye movement data. The feature fusion module is used to perform feature fusion based on EEG features and eye movement features by aligning the data through downsampling. The synergy assessment module is used to perform synergy assessment on the fused features.