Neurocognitive Feature Recognition Method Based on Multimodal Signal Analysis

Abstract

This invention discloses a neurocognitive feature recognition method based on multimodal signal analysis. It simultaneously acquires EEG and eye-tracking signals, extracting EEG time-frequency feature sequences and fixation point spatial trajectory feature sequences respectively. A recurrent coding network is used to encode the two modal feature sequences to obtain latent state vectors. Then, the element-wise product of each time step is calculated and global average pooling is performed to obtain a cross-modal correlation metric. Finally, an activation function maps these to dynamic fusion weights, adaptively weighting and summing the latent state vectors of the two modalities. This method solves the problem of insufficient recognition stability caused by single-modal signals being susceptible to environmental noise, individual physiological differences, and measurement artifacts. It also overcomes the shortcomings of traditional fixed-weight fusion strategies that fail to fully consider the differences in time scale and feature space between different modalities, achieving deep collaborative utilization of multimodal information and improving the accuracy and stability of neurocognitive feature recognition.

Description

Technical Field

[0001] This invention relates to the field of neurocognitive feature recognition technology, and in particular to a neurocognitive feature recognition method based on multimodal signal analysis. Background Technology

[0002] With the deep integration of neuroscience and information technology, neurocognitive state assessment based on physiological signals has been widely applied in fields such as human-computer interaction and medical rehabilitation. Currently, most neurocognitive feature recognition methods still rely primarily on single-modal signals, such as using EEG signals or eye-tracking data alone for cognitive state judgment. However, single signal sources are easily affected by environmental noise, individual physiological differences, and measurement artifacts, making it difficult to meet the high-precision requirements of complex scenarios. Furthermore, some studies have attempted to jointly analyze multiple physiological signals, but these typically employ simple feature splicing or fixed-weight fusion strategies, failing to fully consider the differences in temporal scale and feature space between different modal signals. This results in limited synergistic utilization of multimodal information, hindering further improvements in recognition accuracy. Therefore, there is an urgent need for a neurocognitive feature recognition method that can deeply integrate the advantages of multimodal signals and effectively overcome the limitations of single-modal methods to achieve more accurate and stable cognitive state discrimination. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a neurocognitive feature recognition method based on multimodal signal analysis. The technical solution of this method is as follows: Acquire electroencephalogram (EEG) and eye movement signals simultaneously collected during the performance of cognitive tasks by the subjects; Extract the EEG time-frequency feature sequence from the EEG signal, and extract the fixation point spatial trajectory feature sequence from the eye movement signal; The EEG time-frequency feature sequence and the fixation point spatial trajectory feature sequence are respectively input into the corresponding recurrent coding network to obtain the EEG hidden state vector and the eye movement hidden state vector. The element-wise product of the EEG latent state vector and the eye movement latent state vector at each time step is calculated, and the product results at each time step are subjected to global average pooling to obtain the cross-modal correlation metric. The cross-modal correlation metric is mapped to a dynamic fusion weight through an activation function. The dynamic fusion weight is used as the weighting coefficient of the EEG latent state vector. The difference between 1 and the dynamic fusion weight is used as the weighting coefficient of the eye movement latent state vector. The EEG latent state vector and the eye movement latent state vector are weighted and summed to generate a multimodal fusion feature vector. The multimodal fusion feature vector is input into the classification network to obtain the neurocognitive feature category label of the subject.

[0004] The technical solution of this invention simultaneously acquires EEG signals and eye movement signals, extracts EEG time-frequency feature sequences and fixation point spatial trajectory feature sequences respectively, encodes the two modality feature sequences using a recurrent coding network to obtain latent state vectors, calculates the element-wise product of each time step and performs global average pooling to obtain cross-modal correlation metrics, and then maps them to dynamic fusion weights through activation functions to adaptively weight and sum the latent state vectors of the two modalities. This solves the problem that single-modal signals are easily affected by environmental noise, individual physiological differences and measurement artifacts, resulting in insufficient recognition stability. At the same time, it overcomes the shortcomings of traditional fixed-weight fusion strategies that fail to fully consider the differences of different modalities in time scale and feature space, realizes the deep collaborative utilization of multimodal information, and improves the accuracy and stability of neurocognitive feature recognition.

[0005] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0006] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0007] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of a neurocognitive feature recognition method based on multimodal signal analysis according to the present invention. Detailed Implementation

[0008] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0009] Figure 1 This diagram illustrates a flowchart of an embodiment of a neurocognitive feature recognition method based on multimodal signal analysis provided by the present invention, executed by a control terminal. Figure 1 As shown, this neurocognitive feature recognition method based on multimodal signal analysis includes the following steps: S1. Acquire EEG and eye movement signals simultaneously collected from the subject during the cognitive task.

[0010] The subject refers to a natural person who participates in the neurocognitive feature recognition experiment and receives signal collection. For example, a 25-year-old male volunteer signed an informed consent form before the experiment, was informed of the experimental procedure and precautions, and wore an electrode cap and sat in front of the eye-tracking device as instructed, ready to begin performing a cognitive task. The cognitive task execution process refers to the time interval during which the subject completes a specific cognitive task according to the experimental design requirements, from the issuance of the task start instruction to the receipt of the task end instruction. For example, after receiving the "Start" prompt displayed on the computer screen, the subject responds to continuously presented visual stimuli by pressing buttons until the screen displays the "End" prompt; this approximately 5-minute period constitutes the cognitive task execution process.

[0011] Electroencephalogram (EEG) signals refer to the electrophysiological signals generated by the electrical activity of neurons in the cerebral cortex and their changes over time, collected from the subject's scalp via electrodes. For example, during a subject's key press response, voltage changes at each electrode point are continuously recorded at a sampling rate of 1000 Hz using a 32-lead electrode cap fixed to the scalp; the recorded multi-channel voltage time series constitutes the EEG signal. Eye movement signals refer to the spatial and temporal parameters reflecting the subject's eye movement state, collected by an eye-tracking device. For example, while the subject gazes at visual stimuli on a screen, the eye tracker continuously records the horizontal and vertical coordinates of the center of the subject's pupils in the screen coordinate system at a sampling rate of 1000 Hz; the recorded binocular coordinate time series constitutes the eye movement signal.

[0012] S2. Extract the EEG time-frequency feature sequence based on the EEG signal, and extract the fixation point spatial trajectory feature sequence based on the eye movement signal.

[0013] Among them, the EEG time-frequency feature sequence refers to the data sequence formed by arranging the energy values ​​of each frequency band at each sampling moment extracted after the EEG signal is transformed by time-frequency transformation in chronological order; for example, after the subject's 32-lead EEG signal is decomposed by wavelet packet, the energy values ​​of the θ frequency band, α frequency band and β frequency band at each sampling moment are extracted, and the frequency band energy values ​​at all sampling moments are arranged in chronological order to obtain the EEG time-frequency feature sequence.

[0014] Among them, the gaze point spatial trajectory feature sequence refers to the data sequence formed by arranging the geometric features reflecting the motion law of the gaze point, which are converted from the gaze point position coordinates in the eye movement signal, in chronological order. For example, the gaze point coordinates of the subject at each sampling time are converted into the displacement vector magnitude and direction angle between the gaze point and the previous sampling time, and the displacement vector magnitude and direction angle of all sampling times are arranged in chronological order to obtain the gaze point spatial trajectory feature sequence.

[0015] S3. Input the EEG time-frequency feature sequence and the fixation point spatial trajectory feature sequence into the corresponding cyclic coding network to obtain the EEG hidden state vector and the eye movement hidden state vector.

[0016] Among them, a recurrent coding network refers to a neural network model with a recurrent connection structure, which is used to encode temporal input data and output a hidden state vector of fixed length. For example, a bidirectional gated recurrent unit network is used as a recurrent coding network. This network contains gated recurrent unit layers in two directions, which process the subject's EEG time-frequency feature sequence in forward and reverse order, respectively, and finally output a hidden state vector of fixed dimension.

[0017] The EEG latent state vector refers to the dense vector representation of the temporal dependencies of EEG signals output by the recurrent coding network after encoding the EEG time-frequency feature sequence. For example, after inputting the subject's EEG time-frequency feature sequence into a bidirectional gated recurrent unit network, the network concatenates the forward-order final latent state and the reverse-order final latent state into a dense vector of dimension 256, which is the EEG latent state vector. The eye-tracking latent state vector refers to the dense vector representation of the temporal dependencies of eye-tracking signals output by the recurrent coding network after encoding the eye-tracking spatial trajectory feature sequence. For example, after inputting the subject's fixation point spatial trajectory feature sequence into a bidirectional gated recurrent unit network, the network concatenates the forward-order final latent state and the reverse-order final latent state into a dense vector of dimension 256, which is the eye-tracking latent state vector. The EEG latent state vector is denoted as... The eye-tracking latent state vector is denoted as Both are dimensions. The vector.

[0018] S4. Calculate the element-wise product of the EEG latent state vector and the eye-tracking latent state vector at each time step, and perform global average pooling on the product results at each time step to obtain the cross-modal correlation metric.

[0019] The product result at each time step refers to the vector obtained by multiplying the corresponding elements of the EEG latent state vector and the eye-tracking latent state vector at the same discrete time step. For example, at a certain sampling moment, the first element of the subject's EEG latent state vector is multiplied by the first element of the eye-tracking latent state vector, the second element by the second element, and so on, until a 256-dimensional vector is obtained, which is the product result at that time step. The cross-modal correlation metric is a scalar value reflecting the temporal correlation between the EEG and eye-tracking modalities, obtained by globally averaging and pooling the product results across all time steps. For example, the element-by-element product results at all sampling moments of the subject are averaged along the time dimension to obtain a scalar value, which is used to quantify the overall coordination between the subject's EEG and eye-tracking signals across the entire task timeline.

[0020] Specifically, let the EEG latent state vector at the t-th time step be... The eye-tracking hidden state vector is ,in T is the total number of time steps. The element-wise product at time step t is denoted as... The calculation formula is as follows: ; in, This represents element-wise multiplication. Global average pooling is performed on the product results across all time steps to obtain the cross-modal correlation metric r, calculated using the following formula: ; in, This represents the product result at time step t. The d-th element, Represents the hidden state vector of EEG The d-th element, Represents the eye-tracking hidden state vector The d-th element. Therefore, the cross-modal association metric r is a scalar used to quantify the overall degree of synergy between EEG modality and eye-tracking modality across all task timelines and all feature dimensions.

[0021] S5. The cross-modal correlation metric is mapped to a dynamic fusion weight through an activation function. The dynamic fusion weight is used as the weighting coefficient of the EEG latent state vector. The difference between 1 and the dynamic fusion weight is used as the weighting coefficient of the eye movement latent state vector. The EEG latent state vector and the eye movement latent state vector are weighted and summed to generate a multimodal fusion feature vector.

[0022] The activation function refers to a nonlinear mathematical function that maps real-number inputs to a finite interval, used to convert cross-modal correlation metrics into dynamic fusion weights. For example, a sigmoid function is used as the activation function to map the subject's cross-modal correlation metrics to a value between 0 and 1, which serves as the dynamic fusion weight for subsequent weighted summation. The dynamic fusion weight refers to a real-number coefficient obtained after mapping the cross-modal correlation metrics using the activation function, used to control the respective contributions of the EEG latent state vector and the OMG latent state vector during the fusion process. For example, the subject's cross-modal correlation metrics, after being mapped using the sigmoid function, yield a value of 0.65, which is the dynamic fusion weight.

[0023] The multimodal fusion feature vector refers to the joint feature vector that integrates information from two modalities, generated by weighting and summing the EEG latent state vector and the eye-tracking latent state vector using dynamic fusion weights. For example, 0.65 is multiplied by the subject's EEG latent state vector, 0.35 is multiplied by the eye-tracking latent state vector, and the two products are added together to obtain the fused feature vector with a dimension of 256.

[0024] Specifically, the formula for calculating the dynamic fusion weight w is: ; Where r represents the cross-modal correlation metric. This represents the implicit state vector of the electroencephalogram (EEG). Let w represent the eye-tracking latent state vector, and w represent the dynamic fusion weights. Represents a sigmoid function. , and For learnable scalar parameters, This represents the square of the second norm of a vector. It is a small positive constant; , , and The dynamic fusion weight w is updated synchronously with the network parameters during the classification network training process. The value range of the dynamic fusion weight w is from 0 to 1.

[0025] The dynamic fusion weight w is used as the EEG hidden state vector. The weighting coefficients, in As the eye-tracking latent state vector The weighting coefficients are used to weight and sum the EEG latent state vector and the eye-tracking latent state vector to generate a multimodal fusion feature vector z, the calculation formula of which is: ; Its d-th dimension element is: ; in, , The multimodal fusion feature vector z is the joint feature vector obtained by fusing EEG time-frequency features and eye-tracking spatial trajectory features. Its content is determined by weighted summation of the EEG latent state vector and the eye-tracking latent state vector according to dynamic fusion weights.

[0026] S6. Input the multimodal fusion feature vector into the classification network to obtain the neurocognitive feature category label of the subject.

[0027] Here, the classification network refers to a neural network model that receives multimodal fusion feature vectors as input and outputs category probability distributions. For example, a feedforward neural network with hidden layers is used as the classification network. After receiving the subject's multimodal fusion feature vectors, this network performs hidden layer transformations and output layer mappings to output probability distributions for three categories. The neurocognitive feature category label refers to a discrete symbol output by the classification network used to identify the subject's neurocognitive state type. For example, after processing the subject's multimodal fusion feature vectors, the classification network outputs the "high load" category corresponding to the highest probability; this category is the neurocognitive feature category label.

[0028] Specifically, the formula for calculating the category probability vector p output by the classification network is as follows: ; Where z represents the multimodal fusion feature vector, and p represents the class probability vector output by the classification network. and These are the weight matrix and bias vector of the classification network, respectively. For a learnable projection matrix, For learnable scalar parameters, This represents element-wise product. The category label is the category number corresponding to the maximum probability value in the category probability vector p.

[0029] For example, after receiving a multimodal fusion feature vector z with a dimension of 256, the classification network outputs a probability distribution of three categories after hidden layer transformation and output layer mapping; the category corresponding to the maximum probability is used as the neurocognitive feature category label of the subject, such as "high load", "medium load" or "low load".

[0030] The technical solution of this embodiment synchronously acquires EEG signals and eye movement signals, and extracts EEG time-frequency feature sequences and fixation point spatial trajectory feature sequences respectively. It uses a recurrent coding network to encode the feature sequences of the two modalities to obtain latent state vectors. Then, it calculates the element-wise product of each time step and performs global average pooling to obtain cross-modal correlation metric values. Finally, it maps the latent state vectors of the two modalities to dynamic fusion weights through activation functions and performs adaptive weighted summation. This solves the problem that single-modal signals are easily affected by environmental noise, individual physiological differences and measurement artifacts, resulting in insufficient recognition stability. At the same time, it overcomes the shortcomings of traditional fixed-weight fusion strategies that fail to fully consider the differences of different modalities in time scale and feature space. It realizes the deep collaborative utilization of multimodal information and improves the accuracy and stability of neurocognitive feature recognition.

[0031] In one alternative approach, the steps of acquiring simultaneous EEG and eye-tracking signals from the subject during the performance of a cognitive task include: A visual stimulus sequence is presented to the subject, who then performs a key press response in response to a target stimulus in the visual stimulus sequence. The electroencephalogram (EEG) signals of the subject during the key press response are collected using a multi-lead electrode cap, and the eye movement signals of the subject during the fixation of the visual stimulus sequence are collected simultaneously using an eye tracker. The sampling times of the EEG signals and the eye movement signals are aligned by a synchronous trigger pulse.

[0032] The visual stimulus sequence refers to a set of multiple visual images or symbols presented on a display device in a preset time order. For example, a computer screen might present 100 geometric figures sequentially at a rate of one per second, with each figure presented for 500 ms and a 500 ms interval between adjacent figures; all figures constitute the visual stimulus sequence. Key response refers to the action of a subject pressing a designated key with their finger after observing the target stimulus. For example, a subject might immediately press the space bar on the keyboard with their right index finger each time they see a circular figure, and the time interval from the appearance of the figure to the completion of the key press is recorded as the behavioral response.

[0033] The multi-lead electrode cap refers to a device with multiple electrodes fixed to it and worn on the subject's head, used to simultaneously collect electroencephalogram (EEG) signals from multiple scalp locations. For example, the electrode cap worn by the subject has 32 Ag / AgCl electrodes fixed to it, arranged according to an international standard system, and contacted with the scalp through conductive gel to reduce impedance. The eye tracker refers to a device that uses optical imaging technology to detect and record the position and movement parameters of the subject's eyes in real time. For example, an eye tracker based on infrared pupil-corneal reflection technology is used, installed below a display directly in front of the subject, and collects the coordinates of the gaze points of both eyes at a frequency of 1000 Hz throughout the task.

[0034] Here, sampling time refers to the discrete time points at which the EEG acquisition device and eye tracker perform a single data sampling under the same clock reference. For example, at 1 ms, 2 ms, and 3 ms after the task starts, the EEG device and eye tracker simultaneously trigger data recording, with each millisecond constituting a sampling time. Synchronization trigger pulse refers to the electrical signal generated by the synchronization controller to simultaneously activate the EEG acquisition device and eye tracker for a single sampling. For example, at the start of the task, the synchronization controller continuously generates a high-level pulse at a frequency of 1000 Hz, with each pulse simultaneously sent to the EEG amplifier and eye tracker to ensure that both devices complete sampling at the same time.

[0035] In the above-mentioned optional methods, subjects are further guided to perform key responses through visual stimulus sequences. EEG and eye movement signals are collected synchronously using a multi-lead electrode cap and an eye tracker. The sampling time is aligned with the synchronous trigger pulse, so that the data of the two modalities are acquired under a unified time reference, providing an accurate time synchronization basis for cross-modal correlation analysis.

[0036] In one alternative approach, the visual stimulus sequence is an Oddball paradigm sequence containing a target stimulus and interfering stimuli, the electrode placement positions of the multi-lead electrode cap include the forehead area, central area, and parietal lobe area, and the eye tracker acquires the pupil center position and corneal reflex point.

[0037] In this context, the target stimulus refers to a stimulus in the visual stimulus sequence that requires a specific response from the subject. For example, in a visual stimulus sequence containing circles and squares, the circle is pre-designated as the target stimulus; the subject only needs to press a button when seeing the circle, while the square, as an interfering stimulus, requires no response. The interfering stimulus refers to a stimulus in the visual stimulus sequence that does not require a specific response from the subject. For example, in a visual stimulus sequence containing circles and squares, the square is pre-designated as an interfering stimulus; the subject does not perform any action when seeing the square, only needs to gaze at it. The Oddball paradigm sequence refers to a visual stimulus sequence arranged in a random interleaving pattern of low-probability target stimuli and high-probability interfering stimuli. For example, in 100 stimulus presentations, the circle appears randomly with a 20% probability, and the square appears randomly with an 80% probability; the subject needs to press a button each time the circle appears, but not when the square appears.

[0038] The frontal region refers to the area on the scalp surface corresponding to the anterior part of the frontal lobe of the brain, located between the hairline and the central sulcus. For example, on the subject's scalp, the locations of electrode points Fp1, Fp2, and Fz, according to the international standard system, constitute the frontal region. The central region refers to the area on the scalp surface corresponding to the central sulcus of the brain, located at the junction of the frontal and parietal lobes. For example, on the subject's scalp, the locations of electrode points C3, C4, and Cz, according to the international standard system, constitute the central region. The parietal region refers to the area on the scalp surface corresponding to the parietal lobe of the brain, located between the posterior part of the central sulcus and the occipital lobe. For example, on the subject's scalp, the locations of electrode points P3, P4, and Pz, according to the international standard system, constitute the parietal region.

[0039] The pupil center position refers to the spatial coordinates of the geometric center of the pupil in the coordinate system of the eye tracker. For example, the eye tracker outputs the horizontal coordinates (356 pixels) and vertical coordinates (412 pixels) of the subject's left pupil center in the screen coordinate system in real time; this coordinate point is the pupil center position at that sampling moment. The corneal reflection point refers to the position coordinates of the reflected light spot formed on the subject's corneal surface by the infrared light emitted by the eye tracker. For example, the eye tracker projects an infrared light source onto the subject's corneal surface, forming an infrared reflected light spot. The device detects and outputs the coordinate position of this light spot in the screen coordinate system, which, together with the pupil center position, is used to calculate the gaze direction.

[0040] In the above-mentioned optional methods, the Oddball paradigm sequence is further used to induce EEG components with clear neurocognitive significance. Electrodes are covered in the frontal, central, and parietal lobe areas to capture EEG activities related to cognitive processing. An eye tracker collects the pupil center position and corneal reflex point to obtain high-precision fixation information, thereby improving the reliability of cognitive feature recognition from the source.

[0041] In one alternative approach, the step of extracting the time-frequency feature sequence of the electroencephalogram (EEG) signal includes: The EEG signal is decomposed into multiple sub-bands using wavelet packet decomposition, the energy value of each sub-band is extracted, and the energy values ​​of all sub-bands at each sampling time are combined into a feature vector and arranged in chronological order to form the EEG time-frequency feature sequence.

[0042] Here, a sub-band refers to multiple signal components with different frequency ranges after wavelet packet decomposition of the original EEG signal. For example, after three layers of wavelet packet decomposition, the subject's EEG signal is divided into eight sub-bands with different frequency ranges, each sub-band corresponding to a specific frequency band range. The energy value refers to the cumulative amount of the squared signal amplitude within a sub-band within a given time window, used to quantify the activity level of that band at a specific moment. For example, the amplitude of the EEG signal within the subject's theta band sub-band is squared, summed within the time window of the current sampling moment, and a non-negative value is obtained; this value is the energy value of that sub-band at that sampling moment.

[0043] In the above-mentioned optional methods, wavelet packet decomposition is further used to decompose the EEG signal into multiple sub-bands and extract energy values. The energy values ​​of all sub-bands at each sampling time are combined into a feature vector and arranged in time order. While preserving the time-frequency localization characteristics of the EEG signal, a structured feature sequence is constructed to provide a time-frequency resolution input representation for the recurrent coding network.

[0044] In one alternative approach, the wavelet packet decomposition uses Daubechies family wavelets as the mother wavelet and sets the decomposition level to three levels to extract the energy values ​​of the θ band, α band, and β band in the multiple sub-frequency bands.

[0045] The Daubechies family of wavelets refers to a set of wavelet functions with compact support and orthogonality. For example, the db4 wavelet from the Daubechies family is selected as the mother wavelet. This wavelet has a fourth-order vanishing moment and a finite support length, making it suitable for non-stationary time-frequency analysis of EEG signals. The mother wavelet refers to the basic wavelet waveform used as a basis function in wavelet packet decomposition. For example, the db4 wavelet is used as the mother wavelet, and a set of wavelet basis functions is generated through scaling and translation transformations for multi-scale decomposition of the subject's EEG signal. The number of decomposition levels refers to the number of times the signal is recursively decomposed in the wavelet packet transform. For example, the number of decomposition levels is set to 3. The first level of decomposition of the subject's EEG signal yields low-frequency and high-frequency components. A second level of decomposition is then performed on each component, and so on, ultimately obtaining 8 sub-bands.

[0046] Among the above-mentioned optional methods, the Daubechies family wavelet is further selected as the mother wavelet and a three-level decomposition is set up to extract the energy values ​​of the θ band, α band and β band that are closely related to the cognitive state. While taking into account both computational efficiency and frequency resolution accuracy, the feature expression ability of the frequency bands sensitive to cognitive processing is enhanced.

[0047] In one alternative approach, the step of extracting a spatial trajectory feature sequence of the gaze point based on the eye movement signal includes: Calculate the displacement vector of the gaze point at each sampling time relative to the gaze point at the previous sampling time, extract the magnitude and orientation angle of the displacement vector, concatenate the magnitude and orientation angle to form the feature vector of that sampling time, and arrange them in chronological order to form the spatial trajectory feature sequence of the gaze point.

[0048] The displacement vector refers to the spatial position change vector between two adjacent sampling times. For example, at the 50 ms sampling time, the subject's gaze point coordinates are (400, 300), and at the 51 ms sampling time, the gaze point coordinates change to (402, 298). The vector from (400, 300) to (402, 298) is the displacement vector at that time step.

[0049] In the above-mentioned optional methods, the displacement vectors of the gaze point at each sampling time and the gaze point at the previous time are further calculated, the magnitude and orientation angle are extracted and concatenated into feature vectors, and arranged in chronological order to form a spatial trajectory feature sequence of the gaze point. While preserving the dynamic change law of the gaze point, this provides an input representation with spatial motion information for the recurrent coding network.

[0050] In one alternative approach, it also includes: The displacement vector is subjected to a second-order difference operation to obtain the acceleration vector of the gaze point. The magnitude and direction angle of the acceleration vector are extracted as acceleration features. The magnitude and direction angle of the displacement vector, and the magnitude and direction angle of the acceleration vector are concatenated together to form the feature vector at the sampling time.

[0051] The second-order difference operation refers to the mathematical operation of calculating the change of displacement vector between adjacent time steps to obtain acceleration information of fixation point movement. For example, the difference between the displacement vector between the subject at 52 ms and 51 ms and the displacement vector between 51 ms and 50 ms is calculated, and the difference is divided by the square of the sampling interval to obtain the acceleration vector at that time step.

[0052] The acceleration vector refers to the rate of change of the fixation point's velocity over time, calculated from the displacement vector sequence through second-order difference operations. For example, the acceleration vector of the subject at the 52 ms sampling time is (200, -150), representing the rate of change of the fixation point's velocity in the horizontal and vertical directions at that time. The acceleration feature refers to the set of parameters extracted from the acceleration vector to characterize the dynamics of the fixation point's motion. For example, the magnitude and direction angle of the subject's acceleration vector are extracted; the magnitude reflects the magnitude of the acceleration, and the direction angle reflects the direction of the acceleration, together constituting the acceleration feature for that time step.

[0053] In the above-mentioned optional methods, a second-order difference operation is further performed on the displacement vector to obtain the gaze point acceleration vector. The magnitude and direction angle of the acceleration vector are included in the feature vector and concatenated with the magnitude and direction angle of the displacement vector to form a complete feature vector, which supplements the information on the rate of change of the gaze point motion and improves the depth of characterization of the dynamic pattern of gaze behavior.

[0054] In one alternative approach, the recurrent coding network is a bidirectional gated recurrent unit network. The bidirectional gated recurrent unit network processes the input feature sequence in both forward and reverse chronological order, and concatenates the forward-order final hidden state with the reverse-order final hidden state to obtain the EEG hidden state vector or the eye-tracking hidden state vector.

[0055] Among them, the bidirectional gated recurrent unit network refers to a recurrent neural network structure composed of a forward gated recurrent unit layer and a backward gated recurrent unit layer in parallel, which can simultaneously utilize the forward and backward contextual information of temporal data. For example, the bidirectional gated recurrent unit network is used as a recurrent coding network. The forward layer of the network processes the feature sequence of the subject sequentially from the first sampling time to the 100th sampling time, and the backward layer processes the same feature sequence in reverse order from the 100th sampling time to the first sampling time.

[0056] In the above-mentioned optional approach, a bidirectional gated recurrent unit network is further used to process the input feature sequence in both forward and reverse order of time. The forward-order final hidden state and the reverse-order final hidden state are concatenated into a complete hidden state vector, making full use of the forward and backward context information in the sequence to enhance the modeling ability of EEG and eye movement temporal dependencies.

[0057] In one alternative approach, the formula for calculating the dynamic fusion weights is: ; Where r represents the cross-modal correlation metric. This represents the implicit state vector of the electroencephalogram (EEG). Let w represent the eye-tracking latent state vector, and w represent the dynamic fusion weights. Represents a sigmoid function. , and For learnable scalar parameters, This represents the square of the second norm of a vector. It is a small positive constant. , , and The network parameters are updated synchronously during the training of the classification network.

[0058] It should be noted that the formula for calculating the dynamic fusion weights uses the cross-modal correlation metric as the fundamental variable reflecting the temporal synergy between the EEG and OMG modalities. It also introduces the ratio of the difference between the squared L2 norms of the EEG and OMG latent state vectors to the sum of their squared L2 norms. This ratio quantifies the energy difference between the two modalities in the feature space. The correlation metric and the energy difference term are linearly combined using learnable scaling and bias parameters and then input into a sigmoid function, constraining the output value to the range of 0 to 1. The function of this formula is to dynamically generate fusion weights based on the cross-modal correlation strength and modal energy comparison at the current sampling time. This allows the contributions of the EEG and OMG modalities to adaptively adjust with changes in signal quality and temporal correlation during subsequent weighted summation, thus avoiding the modal information imbalance problem caused by fixed-weight fusion strategies.

[0059] In the above-mentioned optional methods, based on the cross-modal correlation metric, the relative difference between the squared L2 norm of the EEG and eye-tracking latent state vectors is introduced for adaptive adjustment, so that the dynamic fusion weights simultaneously reflect the intermodal correlation and the difference in energy distribution within the modality, thereby improving the adaptability of multimodal fusion to changes in signal quality.

[0060] In one alternative approach, the formula for calculating the category probability vector is: ; Where represents the multimodal fusion feature vector, and p represents the class probability vector output by the classification network. and These are the weight matrix and bias vector of the classification network, respectively. For a learnable projection matrix, For learnable scalar parameters, This represents element-wise multiplication, and the category label is the category number corresponding to the maximum probability value in the category probability vector.

[0061] It should be noted that the formula for calculating the category probability vector involves simultaneously inputting the multimodal fusion feature vector into both a linear transformation path and a nonlinear correction path. In the linear transformation path, a standard fully connected mapping is performed using a weight matrix and a bias vector. In the nonlinear correction path, element-wise squaring of the multimodal fusion feature vector is performed to obtain second-order response information for each dimension. This second-order response information is then projected onto the category space using a learnable projection matrix. The output of the linear path is added to the output of the correction path, scaled by learnable scalar parameters, and this sum serves as the input to the flexible maximum function, ultimately outputting the probability distribution for each category. The function of this formula is to enable the classification network to capture the second-order interaction information between the dimensions of the multimodal fusion feature vector by introducing element-wise squaring terms and a projection matrix. This adds nonlinear correction capability to the standard linear classification, thereby improving the classification network's accuracy in distinguishing different neural cognitive state categories.

[0062] In the above-mentioned optional approach, a quadratic transformation term based on the element-wise squaring of the multimodal fusion feature vector is further introduced into the classification network. This term, together with the linear transformation term, constitutes the category probability output, thereby enhancing the classification network's ability to fit the nonlinear discrimination boundary in the feature space and improving the discrimination accuracy of neural cognitive feature categories.

[0063] In one alternative approach, the step of performing global average pooling on the product results at each time step includes: Let the product result at time step t be denoted as Calculate the L2 square of the product result at the t-th time step. Squaring the second norm By inputting the flexible maximum function, the temporal attention weights at the t-th time step are obtained: ; Where T represents the total number of time steps; the product results of the time steps are weighted and summed using the time attention weights to obtain a weighted product sequence. Global average pooling is performed on the weighted product sequence to obtain the cross-modal correlation metric. ; in, Represents a weighted product sequence The d-th element. Therefore, time steps with higher energy contribute more to the calculation of cross-modal correlation metrics, reduce the interference of noisy time steps on the correlation metrics, and improve the robustness of cross-modal correlation metrics under signal-to-noise ratio fluctuation conditions.

[0064] In the above-mentioned optional approach, the time attention weights of the product results of each time step are further calculated based on their own energy distribution. The time attention weights are then used to perform a weighted summation of the product results of each time step before global average pooling. This allows time steps with higher energy to make a greater contribution to the calculation of cross-modal correlation metrics, reduces the interference of noisy time steps on the correlation metrics, and improves the robustness of cross-modal correlation metrics under signal-to-noise ratio fluctuation conditions.

[0065] It should be noted that the formula for temporal attention weights treats the product of each time step as a vector reflecting the local correlation strength between the EEG modality and the eye-tracking modality at that time step. The squared L2 norm of the product quantifies the energy of the local correlation vector at that time step. The higher the energy, the higher the degree of synergy between the two modalities at that time step, and the greater the potential contribution to global cross-modal correlation. The squared L2 norm of each time step is input into a flexible maximum function to normalize the energy value of each time step globally and map it into a probabilistic form of attention weight. Then, the product of each time step is weighted and summed using this weight, so that the time steps with higher energy dominate the weighted product sequence. Finally, global average pooling is performed on the weighted product sequence to obtain the cross-modal correlation metric.

[0066] The function of the above formula is to introduce a time-dimensional attention mechanism, so that the calculation of cross-modal correlation metrics no longer performs equal-weighted averaging of all time steps, but adaptively allocates the contribution based on the energy of the local correlation vectors of each time step. Time steps with lower energy, i.e., time steps with weaker local correlation between the two modes, are suppressed during pooling, while time steps with higher energy, i.e. time steps with stronger local correlation between the two modes, are enhanced during pooling. This improves the sensitivity of cross-modal correlation metrics to the effective time series interval and reduces the interference of environmental noise or concentrated periods of physiological artifacts on the correlation metrics.

[0067] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A neurocognitive feature recognition method based on multimodal signal analysis, characterized in that, include: Acquire electroencephalogram (EEG) and eye movement signals simultaneously collected during the performance of cognitive tasks by the subjects; Extract the EEG time-frequency feature sequence from the EEG signal, and extract the fixation point spatial trajectory feature sequence from the eye movement signal; The EEG time-frequency feature sequence and the fixation point spatial trajectory feature sequence are respectively input into the corresponding recurrent coding network to obtain the EEG hidden state vector and the eye movement hidden state vector. The element-wise product of the EEG latent state vector and the eye movement latent state vector at each time step is calculated, and the product results at each time step are subjected to global average pooling to obtain the cross-modal correlation metric. The cross-modal correlation metric is mapped to a dynamic fusion weight through an activation function. The dynamic fusion weight is used as the weighting coefficient of the EEG latent state vector. The difference between 1 and the dynamic fusion weight is used as the weighting coefficient of the eye movement latent state vector. The EEG latent state vector and the eye movement latent state vector are weighted and summed to generate a multimodal fusion feature vector. The multimodal fusion feature vector is input into the classification network to obtain the neurocognitive feature category label of the subject.

2. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 1, characterized in that, The steps for acquiring simultaneous EEG and eye-tracking signals from subjects during cognitive tasks include: A visual stimulus sequence is presented to the subject, who then performs a key press response in response to a target stimulus in the visual stimulus sequence. The electroencephalogram (EEG) signals of the subject during the key press response are collected using a multi-lead electrode cap, and the eye movement signals of the subject during the fixation of the visual stimulus sequence are collected simultaneously using an eye tracker. The sampling times of the EEG signals and the eye movement signals are aligned by a synchronous trigger pulse.

3. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 2, characterized in that, The visual stimulus sequence is an Oddball paradigm sequence containing target and interference stimuli. The electrodes of the multi-lead electrode cap are placed in the forehead, central, and parietal lobe areas. The eye tracker collects the pupil center position and corneal reflex point.

4. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 1, characterized in that, The steps for extracting the time-frequency feature sequence of the electroencephalogram (EEG) signal include: The EEG signal is decomposed into multiple sub-bands using wavelet packet decomposition, the energy value of each sub-band is extracted, and the energy values ​​of all sub-bands at each sampling time are combined into a feature vector and arranged in chronological order to form the EEG time-frequency feature sequence.

5. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 4, characterized in that, The wavelet packet decomposition uses Daubechies family wavelets as the mother wavelet and sets the decomposition level to three levels to extract the energy values ​​of the θ band, α band and β band in the multiple sub-frequency bands.

6. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 1, characterized in that, The steps for extracting the spatial trajectory feature sequence of the gaze point based on the eye movement signal include: Calculate the displacement vector of the gaze point at each sampling time relative to the gaze point at the previous sampling time, extract the magnitude and orientation angle of the displacement vector, concatenate the magnitude and orientation angle to form the feature vector of that sampling time, and arrange them in chronological order to form the spatial trajectory feature sequence of the gaze point.

7. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 6, characterized in that, Also includes: The displacement vector is subjected to a second-order difference operation to obtain the acceleration vector of the gaze point. The magnitude and direction angle of the acceleration vector are extracted as acceleration features. The magnitude and direction angle of the displacement vector, and the magnitude and direction angle of the acceleration vector are concatenated together to form the feature vector at the sampling time.

8. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 1, characterized in that, The cyclic coding network is a bidirectional gated cyclic unit network. The bidirectional gated cyclic unit network processes the input feature sequence in both forward and reverse order of time, and concatenates the forward-order final hidden state with the reverse-order final hidden state to obtain the EEG hidden state vector or the eye-tracking hidden state vector.

9. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 8, characterized in that, The formula for calculating the dynamic fusion weight is: ； in, This represents the cross-modal correlation metric. This represents the implicit state vector of the electroencephalogram (EEG). This represents the latent eye-tracking state vector. This represents the dynamic fusion weights. Represents a sigmoid function. , and For learnable scalar parameters, This represents the square of the second norm of a vector. It is a small positive constant. , , and The network parameters are updated synchronously during the training of the classification network.

10. The neurocognitive feature recognition method based on multimodal signal analysis according to claim 9, characterized in that, The formula for calculating the category probability vector is: ； in, This represents the multimodal fusion feature vector. This represents the category probability vector output by the classification network. and These are the weight matrix and bias vector of the classification network, respectively. The projection matrix is ​​learnable. For learnable scalar parameters, This represents element-wise multiplication, and the category label is the category number corresponding to the maximum probability value in the category probability vector.

Images