A real-time cognitive load assessment system of multi-modal psychophysiological signals

Abstract

This invention discloses a real-time cognitive load assessment system based on multimodal psychophysiological signals, relating to the field of cognitive load assessment technology. The system includes: an EEG signal acquisition module for acquiring bilateral EEG signals of the subject within a continuous sliding time window, wherein the bilateral EEG signals include at least a first EEG signal from a first brain region and a second EEG signal from a second brain region; a feature extraction module for analyzing a situational feature sequence characterizing bilateral brain activity based on the first and second EEG signals; and a system that constructs situational and synergistic features of bilateral brain activity. The situational features quantify the growth relationship between the left and right hemispheres in the allocation of cognitive resources by analyzing the fluctuation of the ratio of instantaneous power in the bilateral frontal lobes within a specific frequency band. The synergistic features avoid the problem of incomplete representation of the load state by single-dimensional features by analyzing the synchronization degree of instantaneous phase in the bilateral frontal lobes within a specific frequency band.

Description

Technical Field

[0001] This invention relates to the field of cognitive load assessment technology, specifically to a real-time cognitive load assessment system based on multimodal psychophysiological signals. Background Technology

[0002] Cognitive load refers to the level of load on an individual's working memory system when performing cognitive tasks. It has important application value in fields such as human-computer interaction, education and training, driving safety, and monitoring of special jobs. When cognitive load exceeds an individual's capacity, it will lead to an increase in operational error rate, a longer reaction time, and a decline in decision-making quality. Therefore, real-time monitoring of cognitive load is of great significance for ensuring the safety of human-computer systems and optimizing task design.

[0003] Traditional methods for assessing cognitive load are mainly divided into two categories: subjective scale assessment and task performance assessment. Subjective scale assessment involves having test subjects fill out scales such as the task load index to subjectively report their perceived workload. Although subjective scale assessment is simple to operate, it is lagging, cannot achieve real-time monitoring, and is greatly affected by the subjective factors of the test subjects. Task performance assessment indirectly infers the level of cognitive load by analyzing indicators such as the accuracy rate and reaction time of the test subjects when performing the main task. Although task performance assessment is objective, it requires the test subjects to continuously perform explicit operations, which makes it unsuitable in some scenarios. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a real-time cognitive load assessment system based on multimodal psychophysiological signals.

[0005] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a real-time cognitive load assessment system for multimodal psychophysiological signals, specifically comprising: EEG signal acquisition module: acquires bilateral EEG signals of the subject within a continuous sliding time window, wherein the bilateral EEG signals include at least a first EEG signal from a first brain region and a second EEG signal from a second brain region; Feature extraction module: Based on the first and second EEG signals, analyze the situational feature sequences used to characterize the bilateral activity of the brain; Collaborative feature extraction module: Based on the first and second EEG signals, analyze the collaborative feature sequence used to characterize the degree of coordination of bilateral brain activity; Feature fusion module: Combines the situation feature sequence and the cooperative feature sequence within the same time window to construct a two-dimensional dynamic feature vector, and connects the two-dimensional dynamic feature vectors of multiple consecutive time windows in chronological order to form a motion trajectory in a two-dimensional state space; Load index generation module: Based on the current position of the motion trajectory and the statistical distribution relationship between the position and the historical trajectory point set, determine the current cognitive load index of the tested object.

[0006] Preferably, the EEG signal acquisition module acquires bilateral EEG signals of the test subject within a continuous sliding time window. Based on the first acquisition channel, the module senses the electrophysiological activity of the frontal cortex of the brain in a non-contact capacitive coupling manner through a first flexible thin-film electrode sensor placed at a first preset electrode site in the frontal lobe region of the test subject, and converts the sensed analog electrical signals into a digital first raw EEG sequence. Based on the second acquisition channel, a second flexible thin-film electrode sensor is placed in the frontal lobe region of the test subject and is symmetrical with respect to the midline of the skull with the second preset electrode site. The electrophysiological activity of the frontal cortex of the brain is sensed in a non-contact capacitive coupling manner, and the sensed analog electrical signals are converted into a digital second raw EEG sequence. A bandpass filter is performed on the first original EEG sequence to remove frequency components outside the preset passband range, resulting in a first filtered sequence. A bandpass filter is then performed on the second original EEG sequence to remove frequency components outside the preset passband range, resulting in a second filtered sequence. The first filtered sequence is input to the ultra-low frequency extraction branch. The ultra-low frequency extraction branch has a built-in cutoff frequency, which is lower than the lower limit frequency of the preset passband range of the bandpass filter. The ultra-low frequency extraction branch performs low-pass filtering on the first filtered sequence, filtering out frequency components higher than the cutoff frequency and retaining frequency components lower than the cutoff frequency, and outputting the first ultra-low frequency trend term. The first filtered sequence and the first extremely low-frequency trend term are input into a subtractor. The subtractor performs point-by-point subtraction to remove the first extremely low-frequency trend term from the first filtered sequence, obtaining the first EEG signal. Baseline drift correction is then performed on the second filtered sequence, specifically including: The second filtered sequence is input to the ultra-low frequency extraction branch. The ultra-low frequency extraction branch performs low-pass filtering on the second filtered sequence and outputs the second ultra-low frequency trend term. The second filtered sequence and the second ultra-low frequency trend term are input to the subtractor. The subtractor performs point-by-point subtraction to remove the second ultra-low frequency trend term from the second filtered sequence, thus obtaining the second EEG signal. The first EEG signal and the second EEG signal constitute the bilateral EEG signal.

[0007] Preferably, the feature extraction module acquires bilateral EEG signals of the tested object within a continuous sliding time window, and the bilateral EEG signals include a first EEG signal and a second EEG signal. The first EEG signal is divided into segments according to continuous sliding timestamps to obtain a set of first EEG signal segments. Each EEG signal segment in the set of first EEG signal segments corresponds to a time window. The second EEG signal is divided into segments according to continuous sliding time windows to obtain a set of second EEG signal segments. Each second EEG signal segment in the set of second EEG signal segments corresponds to a time window. For each time window, a short-time Fourier transform is performed on the first EEG signal segment to obtain a first time-frequency distribution matrix. From the first time-frequency distribution matrix, frequency components corresponding to the first preset frequency band are extracted, and the energy amplitudes of all frequency components within the first preset frequency band are summed to obtain the first instantaneous power value of the time window. Arrange the first instantaneous power values ​​of all time windows in chronological order to generate the first instantaneous power sequence; For each time window, a short-time Fourier transform is performed on the second EEG signal segment to obtain a second time-frequency distribution matrix. From the second time-frequency distribution matrix, frequency components corresponding to the first preset frequency band are extracted, and the energy amplitudes of all frequency components within the first preset frequency band are summed to obtain the second instantaneous power value of that time window. The second instantaneous power values ​​of all time windows are arranged in chronological order to generate a second instantaneous power sequence. For each time window, the power value at the first instant is used as the numerator and the power value at the second instant is used as the denominator. A division operation is performed to obtain the power ratio of that time window. The power ratios of all time windows are arranged in chronological order to generate a power ratio sequence. Each power ratio in the power ratio sequence is traversed. For each current power ratio, its previous power ratio and next power ratio are obtained. If the current power ratio is greater than or equal to the previous power ratio and the current power ratio is greater than or equal to the next power ratio, mark the current power ratio as a local maximum and record the position index of the local maximum in the power ratio sequence; If the current power ratio is less than or equal to the previous power ratio and less than or equal to the next power ratio, mark the current power ratio as a local minimum and record the position index of the local minimum in the power ratio sequence. Based on the location indices of local maxima and local minima, all extreme points are traversed alternately in chronological order. The difference in the number of time windows between adjacent local maxima and local minima is calculated. The difference in the number of time windows is multiplied by the step size of the sliding time window to obtain the time interval. The total number of alternations of all local maxima and local minima within the time length of the power ratio sequence is counted. This total number is taken as the alternation frequency. The time interval is reciprocally calculated to obtain the alternation frequency. For each pair of adjacent local maxima and local minima, the absolute value of the difference between the local maxima and local minima is calculated to obtain the amplitude of a single alternation. All single alternation amplitudes are averaged to obtain the alternation amplitude. The alternation frequency and alternation amplitude are combined to generate a situational feature sequence that characterizes the bilateral activity of the brain.

[0008] Preferably, the collaborative feature extraction module, based on the first EEG signal and the second EEG signal, analyzes a collaborative feature sequence used to characterize the degree of coordination of bilateral brain activity, specifically including: Based on the first EEG signal, the first instantaneous phase sequence of the first EEG signal within the second preset frequency band is extracted by Hilbert transform; based on the second EEG signal, the second instantaneous phase sequence of the second EEG signal within the second preset frequency band is extracted by Hilbert transform. Calculate the difference between the first instantaneous phase value in the first instantaneous phase sequence and the second instantaneous phase value in the second instantaneous phase sequence at each sampling point to obtain an instantaneous phase difference sequence. For each time window, statistically analyze the concentration of all instantaneous phase differences in the instantaneous phase difference sequence to obtain the phase lock value corresponding to that time window. The specific steps for obtaining the phase lock value include: The instantaneous phase difference sequence is obtained. The instantaneous phase difference matrix consists of several instantaneous phase difference values ​​arranged in the order of sampling time. For each timestamp, the instantaneous phase difference values ​​within the time range corresponding to the time window are extracted from the instantaneous phase difference sequence to form the in-window phase difference subsequence of that time window. For each instantaneous phase difference, calculate the cosine value of the instantaneous phase difference to obtain the abscissa component corresponding to the instantaneous phase difference; for each instantaneous phase difference, calculate the sine value of the instantaneous phase difference to obtain the ordinate component corresponding to the instantaneous phase difference. Summing the abscissa components corresponding to all instantaneous phase differences within the time window yields the sum of abscissas. Summing the ordinate components corresponding to all instantaneous phase differences within the time window yields the sum of ordinates. Calculating the sum of the squares of the sum of abscissas and the sum of the squares of the sum of ordinates yields the sum of squares. Performing the square root operation on the sum of squares yields the vector and its magnitude. Divide the vector and magnitude by the total number of instantaneous phase differences within the time window to obtain the average vector length. The average vector length is the phase lock value corresponding to the time window. Repeat the above steps until all time windows have been processed to obtain the phase lock value corresponding to each time window. Arrange the phase-locked values ​​corresponding to all time windows in chronological order to form a phase-locked value sequence. Calculate the mean and standard deviation of the phase-locked value sequence. Based on the mean and standard deviation, generate a synergistic feature that characterizes the degree of coordination between the two sides of the brain.

[0009] Preferably, the feature fusion module normalizes the situation features and collaborative features calculated within the same time window to obtain two-dimensional coordinate points corresponding to the time window. The horizontal coordinate of the two-dimensional coordinate points is determined by the normalized situation features, and the vertical coordinate of the two-dimensional coordinate points is determined by the normalized collaborative features. For the current time window, obtain the current two-dimensional coordinate point corresponding to the current time window, and obtain the previous two-dimensional coordinate point corresponding to the previous time window adjacent to the current time window. Use the previous two-dimensional coordinate point and the current two-dimensional coordinate point as the two endpoints of the line segment and connect them to form the current line segment. For each time window, the steps of obtaining the current two-dimensional coordinate point, obtaining the previous two-dimensional coordinate point, and connecting them to form the current line segment are repeated to obtain a number of line segments equal to the number of time windows minus the number of line segments. All line segments are connected end to end according to the order of the time windows, that is, the end point of the previous line segment is used as the starting point of the next line segment, and the cumulative process is used to construct a motion trajectory that evolves continuously in the two-dimensional state space.

[0010] Preferably, the load index generation module constructs a historical trajectory point set based on the two-dimensional coordinate points corresponding to a preset number of time windows before the current time, calculates the average abscissa and ordinate of all historical trajectory points in the historical trajectory point set, and obtains the center point coordinates. Calculate the Euclidean distance from each historical trajectory point in the historical trajectory point set to the center point coordinates, and average all Euclidean distances to obtain the average distance. Calculate the Euclidean distance from the current position point on the trajectory to the center point coordinates to obtain the first Euclidean distance. Divide the first Euclidean distance by the average distance to obtain the first basic index. For each pair of adjacent time windows in the historical trajectory point set, calculate the Euclidean distance between the two historical trajectory points and divide it by the sliding step size of the time window to obtain the motion speed corresponding to the adjacent time windows. The historical average speed is obtained by averaging the motion speeds corresponding to all adjacent time windows. The position point on the motion trajectory at the current moment and the position point corresponding to the previous time window are obtained. The Euclidean distance between the two points is calculated and divided by the sliding step size to obtain the instantaneous motion speed. The instantaneous motion speed is divided by the historical average speed to obtain the second ratio. When the first Euclidean distance is greater than a first preset multiple of the average distance, a first enhancement coefficient is applied to the first base index to obtain a corrected first base index. When the instantaneous velocity is greater than a second preset multiple of the historical average velocity, a second enhancement coefficient is applied to the first base index to obtain a corrected first base index. Multiply the first base index or the modified first base index by the first weighting coefficient to obtain the first weighted component. Multiply the second ratio by the second weighting coefficient to obtain the second weighted component. The sum of the first weighting coefficient and the second weighting coefficient is 1. Add the first weighted component and the second weighted component to obtain the current cognitive load index.

[0011] (III) Beneficial Effects This invention provides It has the following beneficial effects: This invention constructs situational and synergistic features of bilateral brain activity. The situational features quantify the growth relationship between the left and right hemispheres in the allocation of cognitive resources by analyzing the fluctuation of the ratio of instantaneous power of the bilateral frontal lobes in a specific frequency band. The synergistic features quantify the synergistic consistency between the left and right hemispheres in the functional integration process by analyzing the degree of synchronization of instantaneous phase of the bilateral frontal lobes in a specific frequency band. This avoids the problem of incomplete representation of the load state by single-dimensional features. The cognitive load index is generated based on the statistical distribution relationship between the current trajectory point and the historical trajectory point set. By constructing a historical trajectory point set with a preset number of time windows before the current moment, the coordinates of its center point and the average distance are calculated to establish a real-time reference benchmark for the individual. By calculating the first Euclidean distance between the current point and the center point and its ratio with the average distance, the deviation of the current state from the historical normal fluctuation range is quantified. By calculating the ratio of instantaneous velocity to historical average velocity, the drastic degree of change in the current state is quantified, thus achieving a truly personalized adaptive assessment. Attached Figure Description

[0012] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please see Figure 1 This invention provides a real-time cognitive load assessment system for multimodal psychophysiological signals, comprising: EEG signal acquisition module: acquires bilateral EEG signals of the subject within a continuous sliding time window, wherein the bilateral EEG signals include at least a first EEG signal from a first brain region and a second EEG signal from a second brain region; In this embodiment of the invention, the EEG signal acquisition module needs to be specifically described. The EEG signal acquisition module acquires bilateral EEG signals of the test subject within a continuous sliding time window. Based on the first acquisition channel, the module senses the electrophysiological activity of the frontal cortex of the brain in a non-contact capacitive coupling manner through a first flexible thin-film electrode sensor placed at a first preset electrode site in the frontal lobe region of the test subject, and converts the sensed analog electrical signals into a digital first raw EEG sequence. Based on the second acquisition channel, a second flexible thin-film electrode sensor is placed in the frontal lobe region of the test subject and is symmetrical with respect to the midline of the skull with the second preset electrode site. The electrophysiological activity of the frontal cortex of the brain is sensed in a non-contact capacitive coupling manner, and the sensed analog electrical signals are converted into a digital second raw EEG sequence. A bandpass filter is performed on the first original EEG sequence to remove frequency components outside the preset passband range, resulting in a first filtered sequence. A bandpass filter is then performed on the second original EEG sequence to remove frequency components outside the preset passband range, resulting in a second filtered sequence. The first filtered sequence is input to the ultra-low frequency extraction branch. The ultra-low frequency extraction branch has a built-in cutoff frequency, which is lower than the lower limit frequency of the preset passband range of the bandpass filter. The ultra-low frequency extraction branch performs low-pass filtering on the first filtered sequence, filtering out frequency components higher than the cutoff frequency and retaining frequency components lower than the cutoff frequency, and outputting the first ultra-low frequency trend term. The first filtered sequence and the first extremely low-frequency trend term are input into a subtractor. The subtractor performs point-by-point subtraction to remove the first extremely low-frequency trend term from the first filtered sequence, obtaining the first EEG signal. Baseline drift correction is then performed on the second filtered sequence, specifically including: The second filtered sequence is input to the ultra-low frequency extraction branch. The ultra-low frequency extraction branch performs low-pass filtering on the second filtered sequence and outputs the second ultra-low frequency trend term. The second filtered sequence and the second ultra-low frequency trend term are input to the subtractor. The subtractor performs point-by-point subtraction to remove the second ultra-low frequency trend term from the second filtered sequence, thus obtaining the second EEG signal. The first EEG signal and the second EEG signal constitute the bilateral EEG signal.

[0015] It should be noted that the frontal lobe region was chosen as the sampling site because this brain region is highly correlated with higher-order cognitive processing of working memory, attention allocation, and executive function. Changes in its electrical activity can sensitively reflect fluctuations in cognitive load. The first raw EEG sequence is an unprocessed analog-to-digital converted signal recorded in time series form, containing effective physiological information of the target frequency band as well as noise components such as power frequency interference, electromyography artifacts, and electrooculography artifacts. By setting a second acquisition channel at the corresponding position on the opposite side of the skull, a spatially symmetrical acquisition architecture of the left and right hemispheres is constructed to capture the electrophysiological representation of functional lateralization and synergistic effects between the brain hemispheres during cognitive activities. When the cognitive load changes, the activation levels of the left and right frontal lobes show a specific pattern of mutual increase or decrease or synchronous enhancement. Symmetrical acquisition enables the subsequent extraction of dynamic interaction features between the hemispheres caused by cognitive activities by comparing the signal differences or correlations of the two channels. This effectively counteracts the common-mode interference of systemic physiological noise (such as heart rate fluctuations) on single-channel signals and improves the signal-to-noise ratio and specificity of the features. After acquiring the first and second raw EEG sequences, bandpass filtering and baseline drift correction were performed on both to obtain the first and second EEG signals. Through cascaded signal conditioning operations, interfering components were removed, preserving core physiological information relevant to cognitive load. First, bandpass filtering employed a preset passband frequency range (e.g., 0.5Hz to 45Hz). This preset passband frequency range was based on EEG physiology to preserve major rhythmic frequency bands such as Delta, Theta, Alpha, and Beta, while effectively suppressing 50 / 60Hz power line interference and high-frequency electromyographic noise. Second, baseline drift correction, using high-pass filtering or a polynomial fitting algorithm, eliminated extremely low-frequency trend terms caused by electrode polarization, changes in skin impedance, or respiration, stabilizing the signal amplitude near the zero baseline.

[0016] Feature extraction module: Based on the first and second EEG signals, analyze the situational feature sequences used to characterize the bilateral activity of the brain; In this embodiment of the invention, the feature extraction module needs to be specifically described. The feature extraction module acquires the bilateral EEG signals of the tested object within a continuous sliding time window. The bilateral EEG signals include a first EEG signal and a second EEG signal. The first EEG signal is divided into segments according to continuous sliding timestamps to obtain a set of first EEG signal segments. Each EEG signal segment in the set of first EEG signal segments corresponds to a time window. The second EEG signal is divided into segments according to continuous sliding time windows to obtain a set of second EEG signal segments. Each second EEG signal segment in the set of second EEG signal segments corresponds to a time window. For each time window, a short-time Fourier transform is performed on the first EEG signal segment to obtain a first time-frequency distribution matrix. From the first time-frequency distribution matrix, frequency components corresponding to the first preset frequency band are extracted, and the energy amplitudes of all frequency components within the first preset frequency band are summed to obtain the first instantaneous power value of the time window. Arrange the first instantaneous power values ​​of all time windows in chronological order to generate the first instantaneous power sequence; For each time window, a short-time Fourier transform is performed on the second EEG signal segment to obtain a second time-frequency distribution matrix. From the second time-frequency distribution matrix, frequency components corresponding to the first preset frequency band are extracted, and the energy amplitudes of all frequency components within the first preset frequency band are summed to obtain the second instantaneous power value of that time window. The second instantaneous power values ​​of all time windows are arranged in chronological order to generate a second instantaneous power sequence. For each time window, the power value at the first instant is used as the numerator and the power value at the second instant is used as the denominator. A division operation is performed to obtain the power ratio of that time window. The power ratios of all time windows are arranged in chronological order to generate a power ratio sequence. Each power ratio in the power ratio sequence is traversed. For each current power ratio, its previous power ratio and next power ratio are obtained. If the current power ratio is greater than or equal to the previous power ratio and the current power ratio is greater than or equal to the next power ratio, mark the current power ratio as a local maximum and record the position index of the local maximum in the power ratio sequence; If the current power ratio is less than or equal to the previous power ratio and less than or equal to the next power ratio, mark the current power ratio as a local minimum and record the position index of the local minimum in the power ratio sequence. Based on the location indices of local maxima and local minima, all extreme points are traversed alternately in chronological order. The difference in the number of time windows between adjacent local maxima and local minima is calculated. The difference in the number of time windows is multiplied by the step size of the sliding time window to obtain the time interval. The total number of alternations of all local maxima and local minima within the time length of the power ratio sequence is counted. This total number is taken as the alternation frequency. The time interval is reciprocally calculated to obtain the alternation frequency. For each pair of adjacent local maxima and local minima, the absolute value of the difference between the local maxima and local minima is calculated to obtain the amplitude of a single alternation. All single alternation amplitudes are averaged to obtain the alternation amplitude. The alternation frequency and alternation amplitude are combined to generate a situational feature sequence that characterizes the bilateral activity of the brain.

[0017] It should be noted that the process of performing a short-time Fourier transform based on the first EEG signal segment to obtain the first time-frequency distribution matrix, and performing a short-time Fourier transform based on the second EEG signal segment to obtain the second time-frequency distribution matrix, specifically includes the following steps: For each time window, a first EEG signal segment is acquired. The first EEG signal segment consists of multiple first discrete sampling points arranged in the order of sampling time. Each first discrete sampling point corresponds to the voltage amplitude at a sampling time. Windowing was applied to the first EEG signal segment, specifically including: Obtain a preset window function, which is a symmetric function with a preset length less than the length of the time window. Align the starting point of the window function with the starting point of the first EEG signal segment. Multiply each first discrete sampling point within the coverage area of ​​the window function by the window function value at the corresponding position to obtain the first windowed segment. Slide the window function backward along the time axis by a preset step length. Repeat the above steps until the end point of the window function exceeds the end point of the first EEG signal segment to obtain multiple first windowed segments. Perform a discrete Fourier transform on each segment after the first windowing, specifically including: The first windowed segment is converted from a time domain signal to a frequency domain signal to obtain the first spectrum vector corresponding to the first windowed segment. The first spectrum vector consists of multiple first frequency components arranged from low to high frequency. Each first frequency component corresponds to a complex value, which includes a real part and an imaginary part. Based on the first spectral vectors corresponding to all segments after the first windowing, a first time-frequency distribution matrix is ​​constructed, specifically including: The first spectrum vector corresponding to each first windowed segment is used as a column of the first time-frequency distribution matrix. All columns are arranged from left to right according to the time order corresponding to the first windowed segment to form the first time-frequency distribution matrix. The row index of the first time-frequency distribution matrix corresponds to the frequency value, and the column index of the first time-frequency distribution matrix corresponds to different times within the time window. For each time window, a second EEG signal segment is obtained. The second EEG signal segment consists of multiple second discrete sampling points arranged in the order of sampling time. Each second discrete sampling point corresponds to the voltage amplitude at a sampling time. Windowing was applied to the second EEG signal segment, specifically including: Obtain a preset window function, align the starting point of the window function with the starting point of the second EEG signal segment, multiply each second discrete sampling point within the coverage area of ​​the window function by the window function value at the corresponding position to obtain the second windowed segment, slide the window function backward along the time axis by a preset step length, repeat the above steps until the end point of the window function exceeds the end point of the second EEG signal segment, and obtain multiple second windowed segments. Perform a discrete Fourier transform on each segment after the second windowing, specifically including: The second windowed segment is converted from a time-domain signal to a frequency-domain signal to obtain the second spectrum vector corresponding to the second windowed segment. The second spectrum vector consists of multiple second frequency components arranged from low to high frequency, and each second frequency component corresponds to a complex value. Based on the second spectral vectors corresponding to all segments after the second windowing, a second time-frequency distribution matrix is ​​constructed, specifically including: The second spectrum vector corresponding to each second windowed segment is used as a column of the second time-frequency distribution matrix. All columns are arranged from left to right according to the time order corresponding to the second windowed segment to form the second time-frequency distribution matrix. The row index of the second time-frequency distribution matrix corresponds to the frequency value, and the column index of the second time-frequency distribution matrix corresponds to different times within the time window. Based on the first time-frequency distribution matrix and the second time-frequency distribution matrix, the subsequent instantaneous power value extraction step is performed.

[0018] When the frontal lobe region of the brain performs cognitive tasks, the oscillation energy of a specific frequency band is directly related to the synchronous firing intensity of local neuronal groups. This energy value can quantify the activation level of the brain region. The first preset frequency band is a frequency range pre-set based on physiological knowledge related to cognitive load, such as 4-8Hz or 8-13Hz, to focus on neural oscillation activities closely related to working memory and attention allocation. For the first EEG signal, the energy amplitude of the preset frequency band in each time window is calculated using time-frequency analysis techniques (such as short-time Fourier transform or wavelet transform), and the energy amplitudes of all time windows are arranged in chronological order to form the first instantaneous power sequence. Simultaneously, for the second EEG signal, the same time-frequency analysis parameters and frequency band range are used to calculate the energy amplitude of the preset frequency band in each time window and arrange them in chronological order to form the second instantaneous power sequence. After obtaining the first and second instantaneous power sequences, a time-window-by-time ratio calculation is performed on them to generate a power ratio sequence. This converts two absolute power values ​​into relative ratios, eliminating the interference of baseline differences between individuals and fluctuations in systemic physiological state on absolute power values. This highlights the relative activation differences of the bilateral frontal lobes in different time windows. For each time window, the power value in the first instantaneous power sequence is used as the numerator, and the power value of the corresponding time window in the second instantaneous power sequence is used as the denominator. The quotient of the two is calculated to obtain the power ratio for that time window. The calculation results of all time windows are connected in chronological order to form a power ratio sequence. The numerical trend of the power ratio sequence directly reflects the activation level of the left and right frontal lobes in the same cognitive frequency band: when the ratio is greater than 1, it indicates that the first brain region is dominant; when the ratio is less than 1, it indicates that the second brain region is dominant; when the ratio fluctuates around 1, it indicates that the activation levels of both sides are comparable. After generating the power ratio sequence, extreme point detection is performed on the power ratio sequence to identify the implicit oscillation patterns. Through the extreme point detection algorithm, all data points of the power ratio sequence are traversed to identify all local maxima (peaks) and local minima (troughs), and the specific time when each extreme point appears on the time axis is recorded. Based on this, the time interval between adjacent extreme points is counted, that is, the time difference from one local maximum to the adjacent local minimum, or the time difference from one local minimum to the adjacent local maximum. At the same time, the number of times extreme points alternate within a unit time window is counted, that is, the frequency of peak and trough conversion. The time interval reflects the rate of bilateral dominance switching, and the number of alternations reflects the activity level of switching. Based on the time interval and the number of alternations, a situational feature sequence representing the bilateral activity status of the brain is generated. The situational feature sequence includes the alternation frequency and the alternation amplitude. The alternation frequency is calculated by dividing the number of alternations by the corresponding time span, and is used to quantify the intensity of the competition for dominance between the two brain regions per unit time. The higher the alternation frequency, the more intense the competition. The alternation amplitude is calculated by the difference between the local maxima and the adjacent local minima, and is used to quantify the significance of the difference in bilateral activation levels. The larger the alternation amplitude, the stronger the degree of unilateral dominance. The combination of alternation frequency and alternation amplitude is the situational feature sequence.

[0019] Collaborative feature extraction module: Based on the first and second EEG signals, analyze the collaborative feature sequence used to characterize the degree of coordination of bilateral brain activity; In this embodiment of the invention, the collaborative feature extraction module needs to be specifically described. This module, based on the first and second EEG signals, analyzes collaborative feature sequences used to characterize the degree of coordination between bilateral brain activities, specifically including: Based on the first EEG signal, the first instantaneous phase sequence of the first EEG signal within the second preset frequency band is extracted by Hilbert transform; based on the second EEG signal, the second instantaneous phase sequence of the second EEG signal within the second preset frequency band is extracted by Hilbert transform. Calculate the difference between the first instantaneous phase value in the first instantaneous phase sequence and the second instantaneous phase value in the second instantaneous phase sequence at each sampling point to obtain an instantaneous phase difference sequence. For each time window, statistically analyze the concentration of all instantaneous phase differences in the instantaneous phase difference sequence to obtain the phase lock value corresponding to that time window. The specific steps for obtaining the phase lock value include: The instantaneous phase difference sequence is obtained. The instantaneous phase difference matrix consists of several instantaneous phase difference values ​​arranged in the order of sampling time. For each timestamp, the instantaneous phase difference values ​​within the time range corresponding to the time window are extracted from the instantaneous phase difference sequence to form the in-window phase difference subsequence of that time window. For each instantaneous phase difference, calculate the cosine value of the instantaneous phase difference to obtain the abscissa component corresponding to the instantaneous phase difference; for each instantaneous phase difference, calculate the sine value of the instantaneous phase difference to obtain the ordinate component corresponding to the instantaneous phase difference. Summing the abscissa components corresponding to all instantaneous phase differences within the time window yields the sum of abscissas. Summing the ordinate components corresponding to all instantaneous phase differences within the time window yields the sum of ordinates. Calculating the sum of the squares of the sum of abscissas and the sum of the squares of the sum of ordinates yields the sum of squares. Performing the square root operation on the sum of squares yields the vector and its magnitude. Divide the vector and magnitude by the total number of instantaneous phase differences within the time window to obtain the average vector length. The average vector length is the phase lock value corresponding to the time window. Repeat the above steps until all time windows have been processed to obtain the phase lock value corresponding to each time window. Arrange the phase-locked values ​​corresponding to all time windows in chronological order to form a phase-locked value sequence. Calculate the mean and standard deviation of the phase-locked value sequence. Based on the mean and standard deviation, generate a synergistic feature that characterizes the degree of coordination between the two sides of the brain.

[0020] It should be noted that, based on the first EEG signal, a first instantaneous phase sequence of the first EEG signal within a second preset frequency band is extracted using Hilbert transform; based on the second EEG signal, a second instantaneous phase sequence of the second EEG signal within the second preset frequency band is extracted using Hilbert transform. The specific steps include: A bandpass filter is applied to the first EEG signal to retain the frequency components within the second preset frequency band and filter out the frequency components outside the second preset frequency band, thereby obtaining a first frequency band-defined signal. The first frequency band-defined signal is used as the real part signal, and a Hilbert transform is performed on the first frequency band-defined signal to obtain a first orthogonal signal. The first orthogonal signal and the first frequency band-defined signal constitute a pair of orthogonal components. The first frequency band-defined signal is used as the real part and the first orthogonal signal is used as the imaginary part to combine and construct a first analytical signal. For each sampling point in the first analytical signal, the ratio of the imaginary part to the real part corresponding to the sampling point is calculated. The arctangent operation is performed on the ratio to obtain the instantaneous phase value of the sampling point. The range of the instantaneous phase value is mapped to the range between negative π and π or between 0 and 2π. The instantaneous phase values ​​of all sampling points are arranged in the order of sampling time to generate the first instantaneous phase sequence. Bandpass filtering is performed on the second EEG signal to retain the frequency components within the second preset frequency band and filter out the frequency components outside the second preset frequency band to obtain the second frequency band-limited signal. The second frequency band-limited signal is used as the real part signal, and Hilbert transform is performed on the second frequency band-limited signal to obtain the second orthogonal signal. The second frequency band-limited signal is used as the real part and the second orthogonal signal is used as the imaginary part to combine and construct the second analytical signal. For each sampling point in the second analytical signal, the ratio of the imaginary part to the real part corresponding to the sampling point is calculated. An arctangent operation is performed on the ratio to obtain the instantaneous phase value of the sampling point. The instantaneous phase values ​​of all sampling points are arranged in order of sampling time to generate the second instantaneous phase sequence.

[0021] The Hilbert transform is an all-pass filtering transform that converts real-valued signals into analytic signals. It can accurately extract the instantaneous phase value at each sampling moment from the signal. The instantaneous phase value reflects the specific position of the neural oscillation in the cycle at each sampling moment. Before performing the transform, the first and second EEG signals are first bandpass filtered to retain only the frequency components within the second preset frequency band. The second preset frequency band is a frequency range pre-set based on physiological knowledge related to functional connectivity. After filtering, the Hilbert transform is performed on each EEG signal to obtain an analytic signal in complex form. Then, the amplitude of the analytic signal is taken to obtain the instantaneous phase value at each sampling moment. The instantaneous phase values ​​at all sampling moments are arranged in chronological order to form the first instantaneous phase sequence and the second instantaneous phase sequence, respectively. After obtaining the first and second instantaneous phase sequences, a sampling-point-by-sampling phase difference operation is performed on both to generate an instantaneous phase difference sequence. For each sampling point, the phase value in the first instantaneous phase sequence is used as the minuend, and the phase value of the corresponding sampling point in the second instantaneous phase sequence is used as the subtrahend. The subtraction operation is then performed to obtain the instantaneous phase difference value for that sampling point. The instantaneous phase difference value reflects whether the neural oscillations of the first and second brain regions are in phase (difference close to 0), out of phase (difference close to π or -π), or any other arbitrary phase relationship at that instant. The calculation results of all sampling points are connected in chronological order to form the instantaneous phase difference sequence. The numerical fluctuation of the instantaneous phase difference sequence directly reflects the phase coordination state of the two brain regions at each moment: when the two brain regions are working together, the phase difference will stabilize around a certain constant value; when the two brain regions are working independently, the phase difference will exhibit a random distribution. After generating the instantaneous phase difference sequence, for each continuous sliding time window, the instantaneous phase difference sequence within the time window is statistically analyzed to calculate its concentration, thus obtaining the phase lock value of the time window. For each time window, all instantaneous phase difference values ​​within the coverage area of ​​that time window are obtained to form a phase difference sample set. Since the phase values ​​have cyclic periodicity (i.e., 0 and 2π are equivalent), traditional linear statistical methods are not applicable to phase data. Therefore, cyclic statistics are used to measure its concentration. For example, each instantaneous phase difference value is regarded as a point on the unit circle, and the vector sum of all points is calculated. The magnitude of the vector sum is the phase lock value. The magnitude of the vector sum ranges from 0 to 1. When all instantaneous phase difference values ​​are exactly the same, the magnitude of the vector sum reaches the maximum value of 1, indicating that the two-sided phase is completely locked. When the instantaneous phase difference values ​​are uniformly distributed on the circle, the magnitude of the vector sum approaches 0, indicating that the two-sided phase is completely unrelated. After obtaining the phase-locked value for each time window, the phase-locked values ​​of all time windows are arranged in chronological order to form a phase-locked value sequence. Each element in the phase-locked value sequence corresponds to a specific time window. The value of the element represents the phase synchronization intensity of the bilateral brain regions within the time window. The fluctuation of the element value over time reflects the stability and trend of the synergistic relationship. When the cognitive load changes, the functional integration needs of the bilateral frontal lobes change accordingly. This change will be reflected in the increase (synergistic enhancement) or decrease (synergistic weakening) of the phase-locked value sequence. Based on the mean and standard deviation of the phase-locked value sequence, a second coordination feature is generated to characterize the degree of coordination between the two sides of the brain. The mean is obtained by arithmetically averaging all elements in the phase-locked value sequence and is used to characterize the average intensity of phase synchronization between the two brain regions over the entire time range. The higher the mean, the stronger the average level of coordination between the two sides. The standard deviation is calculated by measuring the deviation of all elements in the phase-locked value sequence from the mean and is used to characterize the fluctuation range of the phase synchronization intensity. The larger the standard deviation, the worse the stability of the coordination state, that is, the more drastic the changes in the bilateral coordination relationship between different time windows. Combining the mean and standard deviation constitutes the coordination feature. The coordination feature quantitatively describes the dynamic degree of coordination between the two sides of the brain in the form of numerical pairs within a second preset frequency band.

[0022] Feature fusion module: Combines the situation feature sequence and the cooperative feature sequence within the same time window to construct a two-dimensional dynamic feature vector, and connects the two-dimensional dynamic feature vectors of multiple consecutive time windows in chronological order to form a motion trajectory in a two-dimensional state space; In this embodiment of the invention, the feature fusion module needs to be specifically described. The feature fusion module normalizes the situation features and cooperative features calculated within the same time window to obtain two-dimensional coordinate points corresponding to the time window. The horizontal coordinate of the two-dimensional coordinate points is determined by the normalized situation features, and the vertical coordinate of the two-dimensional coordinate points is determined by the normalized cooperative features. For the current time window, obtain the current two-dimensional coordinate point corresponding to the current time window, and obtain the previous two-dimensional coordinate point corresponding to the previous time window adjacent to the current time window. Use the previous two-dimensional coordinate point and the current two-dimensional coordinate point as the two endpoints of the line segment and connect them to form the current line segment. For each time window, the steps of obtaining the current two-dimensional coordinate point, obtaining the previous two-dimensional coordinate point, and connecting them to form the current line segment are repeated to obtain a number of line segments equal to the number of time windows minus the number of line segments. All line segments are connected end to end according to the order of the time windows, that is, the end point of the previous line segment is used as the starting point of the next line segment, and the cumulative process is used to construct a motion trajectory that evolves continuously in the two-dimensional state space.

[0023] It should be noted that the situational features are composed of alternation frequency and alternation amplitude, and their numerical range depends on the fluctuation characteristics of the power ratio. The coordination features are composed of the mean and standard deviation of the phase-locked value sequence, and their numerical range is limited to 0 to 1. If no normalization is performed and the original values ​​are used directly to construct two-dimensional coordinate points, the features with larger dimensions will dominate in the subsequent trajectory analysis. Linear transformations are performed on the situational features and coordination features respectively to map their values ​​to a preset unified numerical range. For example, the minimum value of each feature is mapped to 0 and the maximum value is mapped to 1. After normalization, both the situational features and coordination features are converted into dimensionless standardized values. These two standardized values ​​are used as the horizontal and vertical coordinate values, respectively, which constitute the two-dimensional coordinate points corresponding to the time window. The position of the two-dimensional coordinate points in the state space comprehensively represents the situational and coordination degree of bilateral brain activity within the time window. After obtaining the two-dimensional coordinates of each time window, the two-dimensional coordinates of the current time window are connected with the two-dimensional coordinates of the previous time window to form a line segment. The sequential order of the continuously sliding time windows is used as the time axis index. Starting from the first time window, the first two-dimensional coordinates of the time window are obtained. When entering the second time window, the second two-dimensional coordinates of the time window are obtained. In the two-dimensional state space, a straight line segment is drawn with the first two-dimensional coordinates as the starting point and the second two-dimensional coordinates as the ending point. The direction of the straight line segment is from the starting point to the ending point. Its length represents the Euclidean distance between the state points of the two time windows in space, and its direction represents the dominant trend of state change. When the situation characteristics or cooperative characteristics change significantly, the two-dimensional coordinates of adjacent time windows will be displaced in space, thus forming a line segment with a specific direction and length. This line segment is the basic unit that constitutes the complete motion trajectory. Each line segment carries the state evolution information from one time window to the next. After connecting the two-dimensional coordinate points of each pair of adjacent time windows into line segments, the line segments of all time windows are accumulated in chronological order to construct a motion trajectory that evolves continuously in the two-dimensional state space. Connecting the beginning and end of all adjacent state change line segments within the entire observation period forms a complete path that runs through the entire time. This path is the motion trajectory. Starting from the first line segment between the first and second time windows, the end of the first line segment is taken as the starting point of the second line segment. The second line segment is connected after the first line segment. Then, the end of the second line segment is taken as the starting point of the third line segment. The third line segment is connected after the second line segment, and so on, until the line segments corresponding to all adjacent time windows are connected in sequence. The final result is a broken line composed of multiple line segments connected end to end. This broken line extends meanderingly in the two-dimensional state space. Its overall shape, local curvature, crossing area, and direction of extension over time together constitute a complete geometric description of the dynamic evolution of cognitive load. Each point of the motion trajectory corresponds to the situation and collaborative state of a specific time window, while the overall trend of the trajectory reflects the continuous change pattern of cognitive load in the time dimension. Once the motion trajectory is constructed, it becomes the direct data basis for subsequent cognitive load determination. The motion trajectory contains dynamic information: the location distribution of trajectory points reflects the combination pattern of competition and cooperation under different time windows; the local density of the trajectory reflects the persistence of state dwell; the curvature change of the trajectory reflects the intensity of state transition; and the overall extension direction of the trajectory reflects the overall trend of cognitive load evolution.

[0024] Load index generation module: Based on the current position of the motion trajectory and the statistical distribution relationship between the position and the historical trajectory point set, determine the current cognitive load index of the tested object.

[0025] In this embodiment of the invention, the load index generation module needs to be specifically described. The load index generation module constructs a set of historical trajectory points based on the two-dimensional coordinate points corresponding to a preset number of time windows before the current time, calculates the average horizontal coordinate and the average vertical coordinate of all historical trajectory points in the set, and obtains the coordinates of the center point. Calculate the Euclidean distance from each historical trajectory point in the historical trajectory point set to the center point coordinates, and average all Euclidean distances to obtain the average distance. Calculate the Euclidean distance from the current position point on the trajectory to the center point coordinates to obtain the first Euclidean distance. Divide the first Euclidean distance by the average distance to obtain the first basic index. For each pair of adjacent time windows in the historical trajectory point set, calculate the Euclidean distance between the two historical trajectory points and divide it by the sliding step size of the time window to obtain the motion speed corresponding to the adjacent time windows. The historical average speed is obtained by averaging the motion speeds corresponding to all adjacent time windows. The position point on the motion trajectory at the current moment and the position point corresponding to the previous time window are obtained. The Euclidean distance between the two points is calculated and divided by the sliding step size to obtain the instantaneous motion speed. The instantaneous motion speed is divided by the historical average speed to obtain the second ratio. When the first Euclidean distance is greater than a first preset multiple of the average distance, a first enhancement coefficient is applied to the first base index to obtain a corrected first base index. When the instantaneous velocity is greater than a second preset multiple of the historical average velocity, a second enhancement coefficient is applied to the first base index to obtain a corrected first base index. Multiply the first base index or the modified first base index by the first weighting coefficient to obtain the first weighted component. Multiply the second ratio by the second weighting coefficient to obtain the second weighted component. The sum of the first weighting coefficient and the second weighting coefficient is 1. Add the first weighted component and the second weighted component to obtain the current cognitive load index.

[0026] It should be noted that the cognitive load index is determined based on the two-dimensional dynamic feature vectors of a preset number of time windows prior to the current moment. A set of historical trajectory points is constructed, and the coordinates of the center point of this set and the average distance of each historical point relative to the center point are calculated. Using the individual's recent physiological state distribution as a reference benchmark, personalized load assessment is achieved by comparing the deviation between the current state and historical states. This avoids individual differences caused by using a fixed threshold. Using the current moment as a benchmark, a preset number of time windows are traced back to obtain the two-dimensional dynamic feature vectors corresponding to these time windows. Each two-dimensional dynamic feature vector represents a historical trajectory point in the state space. All these historical trajectory points together constitute a historical trajectory point set. The preset number is the number of time windows pre-set according to the slow change characteristics of physiological state and the real-time tracking requirements, in order to balance the representativeness and timeliness of historical data. The arithmetic mean of the x-coordinate and y-coordinate of all points in the historical trajectory point set is calculated to obtain the x-coordinate and y-coordinate of the center point. The two together constitute the center point coordinate. The Euclidean distance from each historical trajectory point to the center point is calculated, and the arithmetic mean of the Euclidean distances of all historical trajectory points is calculated to obtain the average distance. The average distance quantitatively describes the degree of dispersion of the recent physiological state in the state space, that is, the normal fluctuation range of an individual in the current period. After obtaining the coordinates of the center point and the average distance, the first Euclidean distance between the current trajectory position and the center point is calculated. Based on the ratio of the first Euclidean distance to the average distance, a first basic index is generated to quantify the deviation of the current state from the historical baseline state. The larger the deviation, the more likely the cognitive load is in an abnormal or significantly changing state. The x-coordinate and y-coordinate values ​​of the current trajectory position are obtained, and the Euclidean distance between this point and the center point is calculated to obtain the first Euclidean distance. The first Euclidean distance reflects the deviation of the current competitive and cooperative combination state from the recent average state. Subsequently, the first Euclidean distance is divided by the average distance to obtain the ratio between the two. This ratio eliminates the differences in the physiological fluctuation amplitude between individuals, making the load index of different individuals or different time periods comparable. The ratio is used as the first basic index. The first basic index numerically represents the relative deviation of the current state from the historical normal fluctuation range: when the ratio is close to 1, it means that the current state is within the historical average fluctuation range; when the ratio is greater than 1, it means that the current state exceeds the historical average fluctuation range, and the larger the value, the more significant the deviation. Extract the instantaneous velocity of the current position point along the trajectory and calculate the second ratio of the instantaneous velocity to the historical average velocity. Changes in cognitive load are reflected not only in the spatial positional shift of state points but also in the speed of state change. Drastic state transitions often indicate rapid accumulation or release of load. Along the trajectory, obtain the current position point and the position point of the previous time window, calculate the Euclidean distance between the two points, and divide it by the sliding step size of the time window to obtain the instantaneous velocity at the current moment. The instantaneous velocity characterizes the rate of change of the combination of competitive and cooperative features in the state space at the current moment. At the same time, the velocity of all adjacent points in the historical trajectory point set is statistically averaged to obtain the historical average velocity. The historical average velocity reflects the average rate of state change of an individual in the recent period. Divide the instantaneous velocity by the historical average velocity to obtain the second ratio. The second ratio quantifies the multiple relationship between the intensity of the current state change and the historical average rate of change: when the ratio is close to 1, it indicates that the rate of change is at an average level; when the ratio is greater than 1, it indicates that the state change is more intense than the historical average. The current cognitive load index is obtained by weighting the first basic index and the second ratio. It integrates information from two dimensions: spatial deviation and rate of change over time, to comprehensively reflect the current level and dynamic trend of cognitive load. The first basic index and the second ratio are assigned a first weight coefficient and a second weight coefficient, respectively. The sum of the first weight coefficient and the second weight coefficient is 1. The specific values ​​of the two can be preset according to the importance attached to spatial deviation and rate of change. The first basic index is multiplied by the first weight coefficient to obtain the first weighted component. The second ratio is multiplied by the second weight coefficient to obtain the second weighted component. The first weighted component and the second weighted component are added together to obtain the current cognitive load index. The cognitive load index is a dimensionless continuous value. The larger the value, the higher the level of cognitive load or the more drastic the change. This realizes the real-time quantitative output of the cognitive load status. In the process of generating the cognitive load index, when the first Euclidean distance is greater than a first multiple of the average distance, the first basic index is enhanced and corrected. When the instantaneous movement speed is greater than a second multiple of the historical average speed, the first basic index is enhanced and corrected. If the deviation or rate of change of the current state significantly exceeds the historical normal range, it indicates that the cognitive load may have entered a special high-load state, and this anomaly needs to be highlighted through enhancement correction. The first preset multiple is a pre-set deviation threshold multiple, such as 2 or 3 times. When the first Euclidean distance exceeds the first preset multiple of the average distance, it is determined that the current state has significantly deviated from the historical normal fluctuation range. At this time, a first enhancement coefficient is applied to the first basic index, such as multiplying by a coefficient greater than 1 or adding a fixed increment to obtain the corrected first basic index. The second preset multiple is a pre-set rate threshold multiple. When the instantaneous movement speed exceeds the second preset multiple of the historical average speed, it is determined that the current state changes abnormally drastically. At this time, a second enhancement coefficient is applied to the first basic index. The enhancement correction can be performed before weighted combination to increase the contribution of abnormal states in the final cognitive load index, thereby more sensitively capturing abrupt cognitive load events.

[0027] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0028] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0029] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0030] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0031] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the described technical solution.

Claims

1. A real-time cognitive load assessment system for multimodal psychophysiological signals, characterized in that, include: EEG signal acquisition module: acquires bilateral EEG signals of the subject within a continuous sliding time window, wherein the bilateral EEG signals include at least a first EEG signal from a first brain region and a second EEG signal from a second brain region; Feature extraction module: Based on the first and second EEG signals, analyze the situational feature sequences used to characterize the bilateral activity of the brain; Collaborative feature extraction module: Based on the first and second EEG signals, analyze the collaborative feature sequence used to characterize the degree of coordination of bilateral brain activity; Feature fusion module: Combines the situation feature sequence and the cooperative feature sequence within the same time window to construct a two-dimensional dynamic feature vector, and connects the two-dimensional dynamic feature vectors of multiple consecutive time windows in chronological order to form a motion trajectory in a two-dimensional state space; Load index generation module: Based on the current position of the motion trajectory and the statistical distribution relationship between the position and the historical trajectory point set, determine the current cognitive load index of the tested object.

2. The real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 1, characterized in that, The EEG signal acquisition module acquires bilateral EEG signals of the test subject within a continuous sliding time window. Based on the first acquisition channel, the module senses the electrophysiological activity of the frontal cortex of the brain in a non-contact capacitive coupling manner through a first flexible thin-film electrode sensor placed at a first preset electrode site in the frontal lobe region of the test subject, and converts the sensed analog electrical signals into a digital first raw EEG sequence. Based on the second acquisition channel, a second flexible thin-film electrode sensor is placed in the frontal lobe region of the test subject and is symmetrical with respect to the midline of the skull with the second preset electrode site. The electrophysiological activity of the frontal cortex of the brain is sensed in a non-contact capacitive coupling manner, and the sensed analog electrical signals are converted into a digital second raw EEG sequence. A bandpass filter is performed on the first original EEG sequence to remove frequency components outside the preset passband range, resulting in a first filtered sequence. A bandpass filter is then performed on the second original EEG sequence to remove frequency components outside the preset passband range, resulting in a second filtered sequence. The first filtered sequence is input to the ultra-low frequency extraction branch. The ultra-low frequency extraction branch has a built-in cutoff frequency, which is lower than the lower limit frequency of the preset passband range of the bandpass filter. The ultra-low frequency extraction branch performs low-pass filtering on the first filtered sequence, filtering out frequency components higher than the cutoff frequency and retaining frequency components lower than the cutoff frequency, and outputting the first ultra-low frequency trend term. The first filtered sequence and the first low-frequency trend term are input into a subtractor. The subtractor performs point-by-point subtraction to remove the first low-frequency trend term from the first filtered sequence, thus obtaining the first EEG signal. Baseline drift correction is then performed on the second filtered sequence to obtain the second EEG signal. The first EEG signal and the second EEG signal constitute the bilateral EEG signal.

3. The real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 1, characterized in that, The feature extraction module acquires bilateral EEG signals of the tested object within a continuous sliding time window. The bilateral EEG signals include a first EEG signal and a second EEG signal. The first EEG signal is divided into segments according to continuous sliding timestamps to obtain a set of first EEG signal segments. Each EEG signal segment in the set of first EEG signal segments corresponds to a time window. The second EEG signal is divided into segments according to continuous sliding time windows to obtain a set of second EEG signal segments. Each second EEG signal segment in the set of second EEG signal segments corresponds to a time window. For each time window, a short-time Fourier transform is performed on the first EEG signal segment to obtain a first time-frequency distribution matrix. From the first time-frequency distribution matrix, frequency components corresponding to the first preset frequency band are extracted, and the energy amplitudes of all frequency components within the first preset frequency band are summed to obtain the first instantaneous power value of the time window. Arrange the first instantaneous power values ​​of all time windows in chronological order to generate the first instantaneous power sequence.

4. The real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 3, characterized in that, For each time window, a short-time Fourier transform is performed on the second EEG signal segment to obtain a second time-frequency distribution matrix. From the second time-frequency distribution matrix, the frequency components corresponding to the first preset frequency band are extracted, and the energy amplitudes of all frequency components within the first preset frequency band are summed to obtain the second instantaneous power value of that time window. The second instantaneous power values ​​of all time windows are arranged in chronological order to generate a second instantaneous power sequence.

5. The real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 4, characterized in that, For each time window, the power value at the first instant is used as the numerator and the power value at the second instant is used as the denominator. A division operation is performed to obtain the power ratio of that time window. The power ratios of all time windows are arranged in chronological order to generate a power ratio sequence. Each power ratio in the power ratio sequence is traversed. For each current power ratio, its previous power ratio and next power ratio are obtained. If the current power ratio is greater than or equal to the previous power ratio and the current power ratio is greater than or equal to the next power ratio, mark the current power ratio as a local maximum and record the position index of the local maximum in the power ratio sequence; If the current power ratio is less than or equal to the previous power ratio and less than or equal to the next power ratio, mark the current power ratio as a local minimum and record the position index of the local minimum in the power ratio sequence. Based on the location indices of local maxima and local minima, all extreme points are traversed alternately in chronological order. The difference in the number of time windows between adjacent local maxima and local minima is calculated. The difference in the number of time windows is multiplied by the step size of the sliding time window to obtain the time interval. The total number of alternations of all local maxima and local minima within the time length of the power ratio sequence is counted. This total number is taken as the alternation frequency. The time interval is reciprocally calculated to obtain the alternation frequency. For each pair of adjacent local maxima and local minima, the absolute value of the difference between the local maxima and local minima is calculated to obtain the amplitude of a single alternation. All single alternation amplitudes are averaged to obtain the alternation amplitude. The alternation frequency and alternation amplitude are combined to generate a situational feature sequence that characterizes the bilateral activity of the brain.

6. The real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 1, characterized in that, The collaborative feature extraction module, based on the first and second EEG signals, analyzes collaborative feature sequences used to characterize the degree of coordination of bilateral brain activity, specifically including: Based on the first EEG signal, the first instantaneous phase sequence of the first EEG signal within the second preset frequency band is extracted by Hilbert transform; based on the second EEG signal, the second instantaneous phase sequence of the second EEG signal within the second preset frequency band is extracted by Hilbert transform. Calculate the difference between the first instantaneous phase value in the first instantaneous phase sequence and the second instantaneous phase value in the second instantaneous phase sequence at each sampling point to obtain the instantaneous phase difference sequence. For each time window, statistically analyze the concentration of all instantaneous phase differences in the instantaneous phase difference sequence to obtain the phase lock value corresponding to that time window.

7. A real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 6, characterized in that, Obtaining the phase lock value specifically includes the following steps: The instantaneous phase difference sequence is obtained. The instantaneous phase difference matrix consists of several instantaneous phase difference values ​​arranged in the order of sampling time. For each timestamp, the instantaneous phase difference values ​​within the time range corresponding to the time window are extracted from the instantaneous phase difference sequence to form the in-window phase difference subsequence of that time window. For each instantaneous phase difference, calculate the cosine value of the instantaneous phase difference to obtain the abscissa component corresponding to the instantaneous phase difference; for each instantaneous phase difference, calculate the sine value of the instantaneous phase difference to obtain the ordinate component corresponding to the instantaneous phase difference. Summing the abscissa components corresponding to all instantaneous phase differences within the time window yields the sum of abscissas. Summing the ordinate components corresponding to all instantaneous phase differences within the time window yields the sum of ordinates. Calculating the sum of the squares of the sum of abscissas and the sum of the squares of the sum of ordinates yields the sum of squares. Performing the square root operation on the sum of squares yields the vector and its magnitude. Divide the vector and magnitude by the total number of instantaneous phase differences within the time window to obtain the average vector length. The average vector length is the phase lock value corresponding to the time window. Repeat the above steps until all time windows have been processed to obtain the phase lock value corresponding to each time window. Arrange the phase-locked values ​​corresponding to all time windows in chronological order to form a phase-locked value sequence. Calculate the mean and standard deviation of the phase-locked value sequence. Based on the mean and standard deviation, generate a synergistic feature that characterizes the degree of coordination between the two sides of the brain.

8. The real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 1, characterized in that, The feature fusion module normalizes the situation features and cooperative features calculated within the same time window to obtain two-dimensional coordinate points corresponding to the time window. The horizontal coordinate of the two-dimensional coordinate points is determined by the normalized situation features, and the vertical coordinate of the two-dimensional coordinate points is determined by the normalized cooperative features. For the current time window, obtain the current two-dimensional coordinate point corresponding to the current time window, and obtain the previous two-dimensional coordinate point corresponding to the previous time window adjacent to the current time window. Use the previous two-dimensional coordinate point and the current two-dimensional coordinate point as the two endpoints of the line segment and connect them to form the current line segment. For each time window, the steps of obtaining the current two-dimensional coordinate point, obtaining the previous two-dimensional coordinate point, and connecting them to form the current line segment are repeated to obtain a number of line segments equal to the number of time windows minus the number of line segments. All line segments are connected end to end according to the order of the time windows, that is, the end point of the previous line segment is used as the starting point of the next line segment, and the cumulative process is used to construct a motion trajectory that evolves continuously in the two-dimensional state space.

9. A real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 1, characterized in that, The load index generation module constructs a historical trajectory point set based on the two-dimensional coordinate points corresponding to a preset number of time windows before the current moment, calculates the average horizontal and vertical coordinates of all historical trajectory points in the historical trajectory point set, and obtains the coordinates of the center point. Calculate the Euclidean distance from each historical trajectory point in the historical trajectory point set to the center point coordinates, and average all Euclidean distances to obtain the average distance. Calculate the Euclidean distance from the current position point on the trajectory to the center point coordinates to obtain the first Euclidean distance.

10. A real-time cognitive load assessment system for multimodal psychophysiological signals according to claim 9, characterized in that, Divide the first Euclidean distance by the average distance to obtain the first basic index. For each pair of adjacent time windows in the historical trajectory point set, calculate the Euclidean distance between the two historical trajectory points and divide it by the sliding step size of the time window to obtain the motion speed corresponding to the adjacent time windows. The historical average speed is obtained by averaging the motion speeds corresponding to all adjacent time windows. The position point on the motion trajectory at the current moment and the position point corresponding to the previous time window are obtained. The Euclidean distance between the two points is calculated and divided by the sliding step size to obtain the instantaneous motion speed. The instantaneous motion speed is divided by the historical average speed to obtain the second ratio. When the first Euclidean distance is greater than a first preset multiple of the average distance, a first enhancement coefficient is applied to the first base index to obtain a corrected first base index. When the instantaneous velocity is greater than a second preset multiple of the historical average velocity, a second enhancement coefficient is applied to the first base index to obtain a corrected first base index. Multiply the first base index or the modified first base index by the first weighting coefficient to obtain the first weighted component. Multiply the second ratio by the second weighting coefficient to obtain the second weighted component. The sum of the first weighting coefficient and the second weighting coefficient is 1. Add the first weighted component and the second weighted component to obtain the current cognitive load index.

Images