Cognitive saturation workload recognition method, system, and model training method

Abstract

This invention provides a method, system, and model training method for recognizing cognitive saturation, which can be applied to the field of cognitive saturation recognition technology. The method includes: performing spectral analysis on the EEG signal data of the object to be identified to obtain frequency domain EEG feature data; for any EEG channel in at least one EEG channel, obtaining graph domain EEG feature data based on the intra-channel visibility relationship between multiple EEG sample values ​​in the EEG time-series data corresponding to the EEG channel; obtaining graph domain physiological feature data based on the cross-channel visibility relationship between multiple physiological sample values ​​in the physiological time-series data corresponding to each of the multiple physiological channels included in the physiological signal data; fusing the frequency domain EEG feature data, graph domain EEG feature data, and graph domain physiological feature data to obtain multimodal feature fusion data; and determining the cognitive saturation level of the object to be identified based on the multimodal feature fusion data.

Description

Technical Field

[0001] This invention relates to the field of cognitive saturation level recognition technology, and more specifically, to a cognitive saturation level recognition method, system, and model training method. Background Technology

[0002] Cognitive saturation (CS) refers to the external manifestation of an individual's cognitive performance under sustained cognitive tasks, resulting from the dynamic integration of fatigue, diminishing attentional resources, and excessive cognitive load. In repetitive tasks (such as assembly line assembly and precision component inspection) and long-term continuous tasks (such as central control room monitoring and air traffic control), the cognitive load of the task is continuously high. If the cognitive demand consistently exceeds the recovery and compensation threshold of the task's resources, it will lead to an accelerated increase in cognitive saturation. This results in behavioral changes such as prolonged reaction time, increased error rate, decreased working memory capacity, and deteriorated decision-making quality. Subjectively, it is often accompanied by feelings of burnout, decreased motivation, and reduced emotional regulation ability.

[0003] Therefore, accurate identification of cognitive fatigue levels can help maintain the health of individuals; however, current methods for identifying cognitive fatigue levels have low accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, system, and model training method for identifying cognitive saturation.

[0005] One aspect of this invention provides a method for identifying cognitive saturation, comprising: performing spectral analysis on electroencephalogram (EEG) signal data of a subject to be identified to obtain frequency-domain EEG feature data, wherein the EEG signal data includes EEG time-series data corresponding to at least one EEG channel, and the frequency-domain EEG feature data characterizes the energy distribution of the EEG signal data in multiple predetermined physiological frequency bands; for any one of the at least one EEG channels, obtaining channel-map-domain EEG feature data corresponding to the EEG channel based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel; and obtaining map-domain EEG feature data based on the channel-map-domain EEG feature data corresponding to at least one of the at least one EEG channels, wherein the channel-map-domain EEG feature data characterizes the energy distribution of the EEG signal data in multiple predetermined physiological frequency bands. The connectivity of each sample value's corresponding EEG sampling time point is determined, where the connectivity of the aforementioned EEG sampling time point is the number of other EEG sampling times that can be connected, as determined by the visibility relationship within the aforementioned channels. Based on the cross-channel visibility relationship between multiple physiological sample values ​​in the physiological time-series data corresponding to multiple physiological channels included in the physiological signal data, graph-domain physiological feature data is obtained. Here, the graph-domain physiological feature data characterizes the connectivity of each of the aforementioned physiological sample values, where the connectivity of the aforementioned physiological sample values ​​is the number of other physiological sample values ​​that can be connected, as determined by the cross-channel visibility relationship. The frequency-domain EEG feature data, the graph-domain EEG feature data, and the graph-domain physiological feature data are fused to obtain multimodal feature fusion data. Based on the multimodal feature fusion data, the cognitive saturation level of the object to be identified is determined.

[0006] Another aspect of this invention provides a model training method, comprising: performing spectral analysis on sample EEG signal data of a sample identification object to obtain sample frequency domain EEG feature data, wherein the sample EEG signal data includes sample EEG time-series data corresponding to at least one sample EEG channel, and the sample frequency domain EEG feature data characterizes the energy distribution of the sample EEG signal data in multiple predetermined physiological frequency bands; for any sample EEG channel among the at least one sample EEG channel, obtaining sample channel map domain EEG feature data corresponding to the sample EEG channel based on the intra-channel visibility relationship between multiple sample EEG sampling values ​​in the sample EEG time-series data corresponding to the sample EEG channel; and obtaining sample map domain EEG feature data based on the sample channel map domain EEG feature data corresponding to at least one sample EEG channel, wherein the sample channel map domain EEG feature data characterizes the continuity of sample EEG sampling times corresponding to the multiple sample EEG sampling values. The connectivity of the sample EEG sampling time points is defined as the number of sample EEG sampling times that can be connected to other sample EEG sampling times, determined based on the visibility relationships within the sample channels. Based on the cross-channel visibility relationships between multiple sample physiological sampling values ​​in the sample physiological time-series data corresponding to multiple physiological channels, sample map-domain physiological feature data is obtained. This data characterizes the connectivity of each of the aforementioned physiological sampling values, and the connectivity of these values ​​is defined as the number of other sample physiological sampling values ​​that can be connected to, determined based on the cross-channel visibility relationships. The sample frequency domain EEG feature data, the sample map-domain EEG feature data, and the sample map-domain physiological feature data are fused to obtain sample multimodal feature fusion data. Based on this multimodal feature fusion data, the sample cognitive saturation level of the sample identification object is determined. A deep learning model is trained based on the loss function, the sample cognitive saturation level, and the sample labels.

[0007] Another aspect of this invention provides a cognitive saturation level recognition system, comprising: a physiological signal data acquisition module configured to acquire physiological signal data of an object to be identified, wherein the physiological signal data includes physiological time-series data corresponding to each of multiple physiological channels; an electroencephalogram (EEG) signal data acquisition module configured to acquire EEG signal data of the object to be identified, wherein the EEG signal data includes EEG time-series data corresponding to each of at least one EEG channel; an electronic device comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the cognitive saturation level recognition method described above; and an electrical stimulation module configured to generate an electrical stimulation signal based on the cognitive saturation level output by the electronic device.

[0008] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0009] Another aspect of the present invention provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0010] According to embodiments of the present invention, frequency-domain EEG feature data based on the frequency domain can be obtained by performing spectral analysis on the EEG signal data of the object to be identified. This frequency-domain EEG feature data characterizes the energy distribution of the EEG signal data across multiple predetermined physiological frequency bands, thereby reflecting the neurophysiological activity patterns of the brain of the object to be identified under different states of consciousness and cognitive activities. Based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel, channel-map EEG feature data corresponding to the EEG channel can be obtained. This channel-map EEG feature data can reflect the connection relationship between the EEG sampling time corresponding to the EEG sampling value of the EEG channel and other EEG sampling times, thereby uncovering deeper features in the EEG signal data. Based on the channel-map EEG feature data corresponding to at least one EEG channel, map-domain EEG feature data is obtained. More comprehensive map-domain EEG feature data can be obtained through the corresponding channel-map EEG feature data of multiple EEG channels.

[0011] Furthermore, based on the cross-channel visibility relationship between multiple physiological sampling values ​​in the physiological time series data corresponding to the physiological channel, the graph domain physiological feature data corresponding to the physiological channel can be obtained. The graph domain physiological feature data corresponding to the physiological channel can reflect the connection relationship between the physiological sampling time corresponding to the physiological sampling value of the physiological channel and other physiological sampling times across channels, thereby mining deeper features in the physiological signal data and obtaining graph domain physiological feature data based on the graph domain.

[0012] Furthermore, frequency domain EEG feature data, image domain EEG feature data, and image domain physiological feature data can be fused to obtain multimodal feature fusion data. Image domain EEG feature data can more deeply reflect the correlation between EEG sample values ​​in EEG signal data, and image domain physiological feature data can more deeply reflect the correlation between physiological sample values ​​in physiological signal data. By fusing processed data based on EEG signal data with processed data based on physiological signal data (which has a different modality than EEG signal data), data features can be enhanced from different modalities to obtain multimodal feature fusion data, thereby improving the accuracy of the cognitive saturation level determined based on multimodal feature fusion data. Attached Figure Description

[0013] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0014] Figure 1 A block diagram of a cognitive saturation level recognition system according to an embodiment of the present invention is shown.

[0015] Figure 2 A flowchart of a cognitive saturation level identification method according to an embodiment of the present invention is shown.

[0016] Figure 3 A block diagram of a cognitive saturation fatigue recognition device according to an embodiment of the present invention is shown.

[0017] Figure 4 A flowchart of a model training method according to an embodiment of the present invention is shown.

[0018] Figure 5 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of the present invention, is shown. Detailed Implementation

[0019] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0022] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0023] In the embodiments of this invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to maintain the security of user personal information and network security.

[0024] In the embodiments of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0025] From a health perspective, prolonged periods of high cognitive saturation are a significant risk factor for psychosomatic problems such as chronic fatigue, burnout, anxiety, and depression, directly impacting long-term work capacity and quality of life. From a safety perspective, in fields with high reliability requirements, such as nuclear power plant operation, chemical process control, and transportation scheduling, momentary errors caused by cognitive saturation can have a cascading effect, leading to serious consequences. Therefore, precise and quantifiable monitoring and assessment of cognitive saturation levels can not only improve individual health but also enhance workplace safety in certain scenarios.

[0026] Electroencephalogram (EEG) signal analysis methods can be categorized into two types: traditional machine learning methods and deep learning transfer learning. Traditional machine learning methods, such as Random Forest and Support Vector Machines, rely on manually designed features such as frequency band energy, entropy, and asymmetry. While these models offer good interpretability, their decoding accuracy is relatively lower compared to newer methods. Deep learning transfer learning provides a solution to these shortcomings. For example, Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTMs) automatically extract features from data through end-to-end learning, exhibiting stronger modeling capabilities. However, these methods require large amounts of labeled data for training, and data collection costs are high in field applications. Furthermore, the black-box nature of these models makes it difficult to meet traceability requirements.

[0027] Among methods for identifying cognitive fatigue, physiological signals provide the most direct and reliable evidence. The collection and analysis of physiological signals have become crucial tools for accurately assessing cognitive fatigue levels. Existing research indicates that when a subject's cognitive fatigue is excessively high, the level of cortical arousal in the brain significantly changes, manifested as a decrease in power corresponding to alpha waves (8–13 Hz), an increase in power corresponding to beta waves (13–30 Hz), and an increase in power corresponding to theta waves (4–8 Hz) in EEG signals. These changes are closely related to cognitive fatigue, especially during high-intensity decision-making tasks, where these changes in EEG signals often provide early indication of the subject's cognitive fatigue level. In addition, under conditions of high cognitive saturation and fatigue, the cardiovascular system, dominated by the autonomic nervous system, will also exhibit regular changes, including a flattening of the pulse waveform, blurred characteristic points, an increase in heart rate at rest and a significant decrease in heart rate variability, weakened peripheral microcirculation, a decrease in perfusion index, and a shift in reference blood pressure from an increase during acute fatigue to increased dysregulation and fluctuations under long-term fatigue. These signals together reflect the body's continuous fatigue state from stress to exhaustion.

[0028] Methods that analyze physiological signal data such as pulse waveform, heart rate, blood oxygen saturation, microcirculation, and reference blood pressure are mainly based on traditional machine learning. These methods heavily rely on expert knowledge and require manual extraction of time-domain features (such as waveform amplitude, interval, and variability), frequency-domain features (such as power spectrum and oscillation components), and nonlinear features (such as entropy and fractal dimension) from the physiological signal data. These features have clear physiological significance, and the model therefore has good interpretability and is easy to trace diagnostic evidence. However, its representation ability is limited, and there are bottlenecks in overall decoding accuracy and generalization ability.

[0029] Both EEG signal data analysis and physiological signal data analysis are single-modal analysis methods. Single-modal analysis methods suffer from limitations such as limited features, poor robustness across individuals, high data noise, small sample sizes, and significant drift across time periods. Therefore, multimodal analysis methods were proposed to further address these issues. Multimodal analysis methods can more comprehensively present the physiological micro-changes of the analyzed object. However, multimodal methods also have the following problems in application: the modality feature fusion method is simple and cannot fully utilize the deep information of complex signal data such as EEG signals; different modalities contain information of varying levels and importance, requiring modality fusion weight design and allocation; furthermore, the data acquisition equipment has a low degree of integration and always affects the normal behavior of the collected object to some extent.

[0030] In view of this, this application proposes a method for recognizing cognitive fatigue saturation. It processes two different modalities of data: electroencephalogram (EEG) signal data and physiological signal data. The processing of EEG signal data yields not only frequency-domain EEG feature data but also map-domain EEG feature data that reflects deeper information within the EEG signal data. Similarly, the processing of physiological signal data yields map-domain physiological feature data. By fusing the frequency-domain EEG feature data, map-domain EEG feature data, and map-domain physiological feature data, a multimodal feature fusion data is obtained. Since both map-domain EEG and map-domain physiological feature data are derived from deeper features, this method increases the amount of deeper features available in the multimodal feature fusion data, thereby enhancing the accuracy of the cognitive fatigue saturation recognition method.

[0031] Figure 1 A block diagram of a cognitive saturation level recognition system according to an embodiment of the present invention is shown.

[0032] like Figure 1 As shown, the cognitive saturation and fatigue recognition system 100 includes a physiological signal data acquisition module 110, an electroencephalogram (EEG) signal data acquisition module 120, an electronic device 130, and an electrical stimulation module 140.

[0033] The physiological signal data acquisition module 110 is configured to acquire physiological signal data of the object to be identified, wherein the physiological signal data includes physiological time series data corresponding to multiple physiological channels.

[0034] The EEG signal data acquisition module 120 is configured to acquire EEG signal data of the object to be identified, wherein the EEG signal data includes EEG timing data corresponding to at least one EEG channel.

[0035] The electronic device 130 includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the cognitive saturation fatigue recognition method of the present invention.

[0036] The electrical stimulation module 140 is configured to generate an electrical stimulation signal based on the cognitive saturation level output by the electronic device.

[0037] A communication link can be provided between the physiological signal data acquisition module 110, the electroencephalogram (EEG) signal data acquisition module 120, the electronic device 130, and the electrical stimulation module 140 via a network. The network can include various connection types, such as wired and / or wireless communication links. For example, physiological signal data and EEG signal data can be transmitted to the electronic device 130 via the network, and the electronic device 130 can process the physiological signal data and EEG signal data to obtain the cognitive fatigue level of the subject to be identified.

[0038] The physiological signal data acquisition module 110 can be, for example, a multispectral physiological data measurement module. The physiological signal data acquisition module 110 may also include a physiological data acquisition submodule, a data processing submodule, a data output submodule, and a physiological power supply module. The physiological data acquisition submodule can use a wide-spectrum, high-sensitivity optical sensor to acquire pulse wave signals from the surface of the object to be identified as raw physiological signal data. The data processing submodule can process the raw physiological signal data into directly usable physiological signal data through built-in signal conditioning technology and algorithms. The physiological signal data can include pulse signal data, heart rate signal data, blood oxygen saturation signal data, microcirculation signal data, and reference blood pressure signal data, etc., with each data type serving as a physiological channel. The data output submodule can output the physiological signal data as data packets to the electronic device 130 in real time through a Universal Asynchronous Receiver / Transmitter (UART) serial interface. The physiological power supply module can be in the form of a rechargeable lithium battery to provide a stable operating voltage for the physiological signal data acquisition module 110, for example, 4.5V~5.5V.

[0039] The EEG signal data acquisition module 120 can be integrated into a portable EEG cap. The EEG signal data acquisition module 120 may include an EEG signal acquisition submodule, an EEG signal processing submodule, a wireless transmission submodule, and an EEG power supply module. The EEG signal acquisition submodule can acquire the raw electrode signals of the object to be identified through EEG acquisition electrodes. The EEG acquisition electrodes and connecting devices are integrated into the portable EEG cap, and the distribution of the EEG acquisition electrodes can be a 4-channel electrode arrangement based on standard lead electrode distribution.

[0040] The EEG signal processing submodule converts raw electrode signals into EEG signal data. It connects to the EEG signal acquisition submodule via a flexible flat cable. The submodule features a high common-mode rejection ratio analog input unit, a low-noise programmable gain amplifier, and a high-resolution synchronous sampling analog-to-digital converter. It acquires and amplifies the raw electrode signals, as well as converting analog signals to digital signals. The submodule can be controlled by a 400MHz processor and outputs EEG signal data via a UART serial interface.

[0041] The wireless transmission submodule can transmit EEG signal data to the electronic device 130. The EEG power supply module can use the output voltage of the lithium battery of the portable EEG cap as the input voltage, and provide the different operating voltages required by the EEG signal data acquisition module 120 through a voltage conversion chip.

[0042] The electrical stimulation module 140 may include a transcranial microcurrent stimulation submodule, an electrical stimulation processor, and an electrical stimulation power supply module. The transcranial microcurrent stimulation submodule may include a BOOST boost circuit (a type of switched DC boost circuit), an H-bridge drive circuit, an electrode dislodgement detection circuit, etc. The electrical stimulation processor may be a chip with a main frequency of 72MHz. The electrical stimulation power supply module may include a power isolation unit, a system power supply unit, and an electrical stimulation signal source power supply unit. The electrical stimulation power supply module can be powered by a lithium battery.

[0043] The electronic device 130 may include a first obtaining submodule, a second obtaining submodule, a third obtaining submodule, a fourth obtaining submodule, and a determining submodule.

[0044] The first submodule is used to perform spectral analysis on the EEG signal data of the object to be identified to obtain frequency domain EEG feature data. The EEG signal data includes EEG timing data corresponding to at least one EEG channel. The frequency domain EEG feature data characterizes the energy distribution of the EEG signal data in multiple predetermined physiological frequency bands.

[0045] The second submodule is used to obtain channel map domain EEG feature data corresponding to any one of the at least one EEG channels, based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel, and to obtain map domain EEG feature data based on the channel map domain EEG feature data corresponding to each of the at least one EEG channels. The channel map domain EEG feature data characterizes the connectivity of the EEG sampling times corresponding to each of the multiple EEG sampling values. The connectivity of the EEG sampling times is the number of other EEG sampling times that can be connected, determined based on the intra-channel visibility relationship.

[0046] The third submodule is used to obtain graph domain physiological feature data based on the cross-channel visibility relationship between multiple physiological sample values ​​in the physiological time series data corresponding to multiple physiological channels included in the physiological signal data. The graph domain physiological feature data characterizes the connectivity of each of the multiple physiological sample values. The connectivity of the physiological sample values ​​is the number of other physiological sample values ​​that can be connected, determined based on the cross-channel visibility relationship.

[0047] The fourth submodule is used to fuse frequency domain EEG feature data, map domain EEG feature data, and map domain physiological feature data to obtain multimodal feature fusion data.

[0048] The determination submodule is used to determine the cognitive saturation level of the object to be identified based on the fusion of multimodal feature data.

[0049] The specific implementations of the first obtaining submodule, the second obtaining submodule, the third obtaining submodule, the fourth obtaining submodule, and the determining submodule can be found in the descriptions of other embodiments of the present invention.

[0050] The EEG signal data acquisition module collects EEG signal data of the subject to be identified; the physiological signal data acquisition module collects physiological signal data of the subject to be identified. The EEG signal data and physiological signal data are processed in a multimodal manner by electronic devices to obtain the cognitive saturation level. The electrical stimulation module provides early warning feedback based on the cognitive saturation level. When an early warning is required, an electrical stimulation signal is generated to increase the excitability of the nervous system of the subject to be identified and to induce rest.

[0051] Figure 2 A flowchart of a cognitive saturation level identification method according to an embodiment of the present invention is shown.

[0052] like Figure 2 As shown, the method includes operations S210 to S250.

[0053] In operation S210, the EEG signal data of the object to be identified is subjected to spectral analysis to obtain frequency domain EEG feature data.

[0054] The EEG signal data includes EEG timing data corresponding to at least one EEG channel, and frequency domain EEG feature data characterizes the energy distribution of the EEG signal data in multiple predetermined physiological frequency bands.

[0055] In operation S220, for any one of the at least one EEG channels, channel map domain EEG feature data corresponding to the EEG channel is obtained based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel, and map domain EEG feature data is obtained based on the channel map domain EEG feature data corresponding to each of the at least one EEG channel.

[0056] Channel-map domain EEG feature data characterizes the connectivity of EEG sampling moments corresponding to multiple EEG sampling values. The connectivity of EEG sampling moments is the number of other EEG sampling moments that can be connected, determined based on the visibility relationship within the channel.

[0057] In operation S230, based on the cross-channel visibility relationship between multiple physiological sampling values ​​in the physiological time series data corresponding to multiple physiological channels, the graph domain physiological feature data is obtained.

[0058] Graph domain physiological feature data characterizes the connectivity of multiple physiological sample values, where the connectivity of a physiological sample value is the number of other physiological sample values ​​that can be connected, determined based on cross-channel visibility relationships.

[0059] In operation S240, frequency domain EEG feature data, map domain EEG feature data, and map domain physiological feature data are fused to obtain multimodal feature fusion data.

[0060] In operation S250, the cognitive saturation level of the object to be identified is determined based on the multimodal feature fusion data.

[0061] The physiological frequency bands of electroencephalogram (EEG) signals typically include Delta ( ) wave (0.5-4Hz), Theta ( ) wave (4-8Hz), Alpha ( ) wave (8-13Hz), Beta ( ) wave (13-30Hz) and Gamma ( ) wave (30-49Hz). Physiological frequency bands associated with cognitive fatigue include Wave, Wave, wave and The frequency band where the wave is located, therefore, can Wave, Wave, wave and The frequency bands corresponding to each wave are designated as the physiological frequency bands.

[0062] Before performing spectral analysis on EEG signal data, the EEG signal data can be preprocessed. Preprocessing can include filtering, removal of electrooculogram (EOG) signals, etc. Filtering can be done by using a zero-phase Butterworth filter for bandpass filtering of 1–50 Hz and a notch filter of 50 Hz.

[0063] This embodiment of the invention is described using a number of EEG channels of 4, but it is not limited to this; the number of EEG channels can also be 8, 12, etc. EEG signal data can be represented as... ,in, This represents the T sampling times of EEG timing data from 4 EEG channels, with a sampling duration of 10 seconds and a sampling frequency of [missing information]. Sampling is performed at 250Hz, and T can be 2500.

[0064] For the preprocessed EEG signal data, a Fast Fourier Transform (FFT) with a step size of 512 and 1024 points can be performed to obtain the energy distribution of EEG signal data of at least one EEG channel in multiple predetermined physiological frequency bands. Flattening the energy distribution of EEG signal data corresponding to multiple EEG channels in multiple predetermined physiological frequency bands yields frequency domain EEG feature data, as shown below.

[0065] (1);

[0066] in, This represents frequency domain EEG feature data. This indicates the energy distribution of brainwave signal data corresponding to a brainwave channel across multiple predetermined physiological frequency bands. The data dimension of the frequency domain EEG feature data is 32, as stated in the following text. These all represent the data dimensions of the corresponding data, and will not be explained further.

[0067] EEG time-series data is typically acquired continuously. Therefore, there is a certain correlation between multiple EEG sample values ​​in the time-series data corresponding to each EEG channel, which can be considered as an intra-channel visibility relationship. Thus, channel-domain EEG feature data corresponding to each EEG channel can be obtained based on this intra-channel visibility relationship. By concatenating the channel-domain EEG feature data corresponding to at least one EEG channel, the map-domain EEG feature data is obtained. This can be represented as follows.

[0068] (2);

[0069] in, Represents graph domain EEG feature data, This represents the channel map domain EEG feature data corresponding to at least one EEG channel.

[0070] The acquisition of frequency domain EEG feature data and image domain EEG feature data can be a parallel processing process. For example, after acquiring EEG signal data, the data can be segmented into 10-second non-overlapping windows with a total of 4 EEG channels. Frequency domain EEG feature data and image domain EEG feature data can then be obtained through parallel dual-track processing, thereby improving data processing efficiency.

[0071] Since both frequency domain EEG data and map domain EEG feature data are derived from EEG signal data, they can be regarded as feature pools of EEG signal data, as shown below.

[0072] (3);

[0073] in, Feature pool representing EEG signal data.

[0074] Before processing physiological signal data to obtain map-domain physiological feature data, the physiological signal data can be preprocessed. Preprocessing can include filtering, amplitude normalization, etc. Filtering can be done by using a zero-phase Butterworth filter for 0.05–45Hz bandpass filtering and power frequency notch filtering.

[0075] This embodiment of the invention is described using 5 physiological channels, but it is not limited to this; the number of physiological channels can also be 6, 7, etc. Physiological channels may include channels processing pulse signal data, channels processing heart rate signal data, channels processing blood oxygen saturation signal data, channels processing microcirculation signal data, or channels processing reference blood pressure signal data. Physiological signal data can be represented as follows: ,in, Represents physiological time-series data in 5 physiological channels At each physiological sampling time point, with a sampling duration of 10 seconds and a sampling frequency of [missing information], Taking 100Hz as an example, It can be 1000.

[0076] Cross-channel visibility relationships can be the association between multiple physiological sample values ​​in different physiological channels. Based on the cross-channel visibility relationships between multiple physiological sample values ​​in the physiological time series data corresponding to each of the multiple physiological channels, graph domain physiological feature data can be obtained, as shown below.

[0077] (4);

[0078] in, Represents the physiological characteristics data of the graph domain. This represents the cross-channel map domain physiological feature data corresponding to at least one physiological channel.

[0079] Frequency domain EEG feature data, map domain EEG feature data, and map domain physiological feature data can be fused according to certain dimensions to obtain multimodal feature fusion data.

[0080] Multimodal feature fusion data can be input into a pre-trained network to determine the cognitive saturation level of the object to be identified.

[0081] According to embodiments of the present invention, frequency-domain EEG feature data based on the frequency domain can be obtained by performing spectral analysis on the EEG signal data of the object to be identified. This frequency-domain EEG feature data characterizes the energy distribution of the EEG signal data across multiple predetermined physiological frequency bands, thereby reflecting the neurophysiological activity patterns of the brain of the object to be identified under different states of consciousness and cognitive activities. Based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel, channel-map EEG feature data corresponding to the EEG channel can be obtained. This channel-map EEG feature data can reflect the connection relationship between the EEG sampling time corresponding to the EEG sampling value of the EEG channel and other EEG sampling times, thereby uncovering deeper features in the EEG signal data. Based on the channel-map EEG feature data corresponding to at least one EEG channel, map-domain EEG feature data is obtained. More comprehensive map-domain EEG feature data can be obtained through the corresponding channel-map EEG feature data of multiple EEG channels.

[0082] Furthermore, based on the cross-channel visibility relationship between multiple physiological sampling values ​​in the physiological time series data corresponding to the physiological channel, the graph domain physiological feature data corresponding to the physiological channel can be obtained. The graph domain physiological feature data corresponding to the physiological channel can reflect the connection relationship between the physiological sampling time corresponding to the physiological sampling value of the physiological channel and other physiological sampling times across channels, thereby mining deeper features in the physiological signal data and obtaining graph domain physiological feature data based on the graph domain.

[0083] Furthermore, frequency domain EEG feature data, image domain EEG feature data, and image domain physiological feature data can be fused to obtain multimodal feature fusion data. Image domain EEG feature data can more deeply reflect the correlation between EEG sample values ​​in EEG signal data, and image domain physiological feature data can more deeply reflect the correlation between physiological sample values ​​in physiological signal data. By fusing processed data based on EEG signal data with processed data based on physiological signal data (which has a different modality than EEG signal data), data features can be enhanced from different modalities to obtain multimodal feature fusion data, thereby improving the accuracy of the cognitive saturation level determined based on multimodal feature fusion data.

[0084] According to an embodiment of the present invention, channel map domain EEG feature data corresponding to an EEG channel is obtained based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel. This includes: determining the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel based on a predetermined EEG penetration parameter, wherein the EEG penetration parameter represents the number of EEG sampling moments that prevent the connection between EEG sampling moments and other EEG sampling moments; and obtaining channel map domain EEG feature data corresponding to the EEG channel based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel and the EEG connectivity sequence length, wherein the EEG connectivity sequence length represents the sequence length of the connectivity of EEG sampling moments in the channel map domain EEG feature data.

[0085] The preset EEG penetration parameter can be set to 1, meaning that there can be one EEG sampling time that blocks other EEG sampling times. Each element in the EEG connectivity sequence length represents the connectivity of the EEG sampling time corresponding to the respective EEG sampling value. The EEG connectivity sequence length can be set to 64, meaning that the nodes in the channel map domain EEG feature data can include 64.

[0086] By setting predetermined EEG penetration parameters, we can avoid inaccurate construction results of channel map EEG feature data due to excessive connectivity at the EEG sampling time. By setting the EEG connectivity sequence length, we can avoid excessive data in the channel map EEG feature data, which would lead to overly complex calculations.

[0087] According to an embodiment of the present invention, graph domain physiological feature data is obtained based on the cross-channel visibility relationship between multiple physiological sample values ​​in the physiological time series data corresponding to multiple physiological channels included in the physiological signal data. This includes obtaining graph domain physiological feature data based on the cross-channel visibility relationship between multiple physiological sample values ​​in the physiological time series data corresponding to multiple physiological channels included in the physiological signal data and the physiological connectivity sequence length, wherein the physiological connectivity sequence characterizes the sequence length of connectivity at the physiological sampling time.

[0088] Each element in the physiological connectivity sequence length represents the connectivity of the physiological sampling time corresponding to the physiological sampling value. The physiological connectivity sequence length can be set to 64, that is, the nodes in the graph domain physiological feature data can include 64 nodes.

[0089] By setting the length of the physiological connectivity sequence, we can avoid excessive computational complexity caused by too much data in the graph domain physiological feature data.

[0090] According to an embodiment of the present invention, frequency domain EEG feature data, image domain EEG feature data, and image domain physiological feature data are fused to obtain multimodal feature fusion data, including: channel attention fusion of frequency domain EEG feature data and image domain EEG feature data to obtain EEG feature fusion data; and weighted fusion of feature dimension data in the EEG feature fusion data and feature dimension data in the image domain physiological feature data to obtain multimodal feature fusion data.

[0091] Since both frequency domain EEG feature data and graph domain EEG feature data are obtained by performing different operations on EEG signal data, channel attention fusion can be performed on the corresponding EEG channel data to obtain EEG feature fusion data.

[0092] EEG feature fusion data and map-domain physiological feature data can be fused within the same dimension, determining the data and weights of each feature dimension. Based on the data and weights of each feature dimension of the EEG feature fusion data and the map-domain physiological feature data, a weighted fusion can be performed on the feature dimension data from the EEG feature fusion data and the map-domain physiological feature data to obtain multimodal feature fusion data.

[0093] Since both frequency domain EEG feature data and map domain EEG feature data are EEG-related data, the correlation between them is high. Therefore, channel attention fusion can be performed on the frequency domain EEG feature data and map domain EEG feature data to obtain EEG feature fusion data. Then, the EEG feature fusion data and physiological feature data can be fused to obtain multimodal feature fusion data. Since both EEG feature fusion data and physiological feature data include deep feature acquisition through the map domain, the obtained multimodal feature fusion data also includes deep multimodal features, thereby improving the feature richness of the multimodal feature fusion data.

[0094] According to an embodiment of the present invention, channel attention fusion is performed on frequency domain EEG feature data and image domain EEG feature data to obtain EEG feature fusion data, including: performing convolution processing on frequency domain EEG feature data and image domain EEG feature data to obtain intermediate frequency domain EEG feature data and intermediate image domain EEG feature data, wherein the feature dimension of the intermediate image domain EEG feature data is the same as the feature dimension of the intermediate frequency domain EEG feature data; and fusing the intermediate frequency domain EEG feature data and the intermediate image domain EEG feature data according to the feature dimension of the intermediate frequency domain EEG feature data and the feature dimension of the intermediate image domain EEG feature data to obtain EEG feature fusion data.

[0095] Convolutional processing can be performed on frequency-domain EEG feature data and image-domain EEG feature data to obtain intermediate frequency-domain EEG feature data and intermediate image-domain EEG feature data. The convolutional processing of the frequency-domain EEG feature data is as follows, with the frequency-domain EEG feature data as input.

[0096] (5);

[0097] in, This indicates the first one-dimensional convolutional layer, where 16 represents the number of convolutional kernels (number of output channels) and 3 represents the size of the convolutional kernels (window size). This indicates the Batch Normalization layer. This represents the corrected linear unit. This indicates the second one-dimensional convolutional layer, where 32 represents the number of output channels and 3 represents the kernel size. This represents the Global Max Pooling layer. This represents the intermediate frequency domain EEG characteristic data.

[0098] The convolutional processing depth for graph domain EEG feature data and frequency domain EEG feature data can be the same. The input is graph domain EEG feature data, and the output is as follows.

[0099] (6);

[0100] in, This represents the EEG feature data of the intermediate map domain.

[0101] The feature dimensions of intermediate map domain EEG feature data and intermediate frequency domain EEG feature data can be the same, so the two can be fused in the same feature dimension to obtain EEG feature fusion data.

[0102] By performing convolution processing on frequency domain EEG feature data and graph domain EEG feature data, spatial dimensionality reduction and information compression can be further performed on the features in the data, and intermediate graph domain EEG feature data and intermediate frequency domain EEG feature data with the same feature dimension can be output so that the two can be fused on the same feature dimension to obtain EEG feature fusion data.

[0103] According to an embodiment of the present invention, fusing intermediate frequency domain EEG feature data and intermediate map domain EEG feature data based on the feature dimensions of intermediate frequency domain EEG feature data and intermediate map domain EEG feature data to obtain EEG feature fusion data may include: determining the weight of the EEG feature fusion dimension based on the correlation data of intermediate frequency domain EEG feature data and intermediate map domain EEG feature data in the feature dimensions; and performing feature fusion on frequency domain EEG feature convolution data and channel map domain EEG feature convolution data based on the weight of the EEG feature fusion dimension to obtain EEG feature fusion data.

[0104] Intermediate frequency domain EEG feature data and intermediate image domain EEG feature data can be correlated along the feature dimension to obtain correlated data. The correlated data can be the convolutional fusion result of intermediate frequency domain EEG feature data and intermediate image domain EEG feature data. Based on this correlated data, the weights of the EEG feature fusion dimension are determined, as shown below.

[0105] (7);

[0106] in, Indicates the weights of the EEG feature fusion dimensions. Indicates related data. The weight matrix represents the associated data. Indicates the bias term. This represents the second-layer weight matrix. This represents the sigmoid activation function.

[0107] The weights of the EEG feature fusion dimensions can represent the output components of the intermediate frequency domain EEG feature data and the intermediate map domain EEG feature data in each feature dimension, thereby performing feature fusion on the intermediate frequency domain EEG feature data and the intermediate map domain EEG feature data to obtain the EEG feature fusion data, as shown below.

[0108] (8);

[0109] in, This represents the fusion data of EEG features.

[0110] By using the correlation data between intermediate frequency domain EEG feature data and intermediate image domain EEG feature data in the feature dimension, the weight of the feature dimension can be calculated, and then the intermediate frequency domain EEG feature data and intermediate image domain EEG feature data can be fused according to the weight of the EEG feature fusion dimension.

[0111] According to an embodiment of the present invention, the feature dimension data in the EEG feature fusion data and the feature dimension data in the graph domain physiological feature data are weighted and fused to obtain multimodal feature fusion data, including: performing convolution processing on the graph domain physiological feature data to obtain intermediate graph domain physiological feature data; performing weighted fusion of the feature dimension data in the EEG feature fusion data and the feature dimension data in the intermediate graph domain physiological feature data to obtain EEG physiological feature connectivity data; and performing edge attention convolution on the EEG physiological feature connectivity data to obtain multimodal feature fusion data.

[0112] Convolutional processing can be applied to graph-domain physiological feature data to achieve dimensionality reduction and feature extraction. The convolutional processing of graph-domain physiological feature data can be a single convolutional layer sharing weight values ​​with the graph-domain EEG feature data, with the input being the graph-domain physiological feature data, as shown below.

[0113] (9);

[0114] in, This represents the physiological characteristic data of the intermediate map domain.

[0115] The output of the weighted fusion of EEG feature fusion data and intermediate map domain physiological feature data can be under the same feature dimension. Therefore, based on the feature dimension of the output, the feature dimension data in the EEG feature fusion data and the feature dimension data in the intermediate map domain physiological feature data can be weighted and fused to obtain EEG physiological feature connectivity data, as shown below.

[0116] (10);

[0117] in, This represents the connection data of electroencephalographic characteristics. This indicates a padding operation, which can be used to add extra elements (e.g., 0 values) to the edges of input data.

[0118] Edge attention convolution on EEG physiological feature connectivity data can be implemented using an Edge Graph Attention Network (EGAT) to further highlight the features in the EEG physiological feature connectivity data and obtain multimodal feature fusion data, as shown below.

[0119] (11);

[0120] in, This represents multimodal feature fusion data. Indicates global average pooling. This represents the data after performing limb attention convolution on the electroencephalogram (EEG) feature connectivity data.

[0121] Convolution of graph-domain physiological feature data facilitates weighted fusion of graph-domain physiological feature data and EEG feature fusion data. Edge attention convolution of EEG physiological feature connection data can further extract features from EEG physiological feature connection data to obtain multimodal feature fusion data.

[0122] According to an embodiment of the present invention, determining the cognitive saturation level of an object to be identified based on multimodal feature fusion data may include: performing spatiotemporal evolution coding on the multimodal feature fusion data to obtain multimodal feature fusion coded data; performing convolution and global average pooling on the multimodal feature fusion coded data to obtain multimodal feature fusion coded output data; and performing fully connected processing and threshold mapping on the multimodal feature fusion coded output data to determine the cognitive saturation level of the object to be identified.

[0123] Spatiotemporal evolution encoding of multimodal feature fusion data can be achieved by inputting the multimodal feature fusion data into a Transformer encoding network in a temporal sequence. The Transformer encoding network can be based on a multi-head self-attention mechanism and may include two layers with a hidden layer dimension of 64 and an 8-head multi-head self-attention mechanism. The dropout rate during training can be 0.15. Specifically, the multimodal feature fusion data can be represented as temporal data. ,in, This represents a sequence of multimodal feature fusion data. This means that five consecutive time-series data points are selected from the sequence and concatenated as a group each time. This is because hardware sampling frequency is usually relatively high and the segments are short. Therefore, selecting a certain number of time-series data points for concatenation can increase the length of the input data and improve processing efficiency. The output multimodal feature fusion encoded data is represented as follows.

[0124] (12);

[0125] in, This represents multimodal feature fusion encoded data.

[0126] One-dimensional convolution (Depthwise(1D)-Conv) and global average pooling (GAP) can be performed on the multimodal feature fusion encoded data to obtain the multimodal feature fusion encoded output data, represented as follows: .

[0127] The multimodal feature fusion encoding output data is processed by a fully connected classification head to obtain the probability of cognitive saturation level, as shown below.

[0128] (13);

[0129] in, This represents the probability of cognitive saturation level. This represents the weights of the multimodal feature fusion encoding output data.

[0130] Different probabilities correspond to different levels. Therefore, threshold mapping can be performed based on the probability of cognitive saturation level to determine the cognitive saturation level of the object to be identified. The cognitive saturation level can include three levels, as shown below.

[0131] (14);

[0132] in, This indicates the level of cognitive saturation of the object to be identified.

[0133] Figure 3 A block diagram of a cognitive saturation fatigue recognition device according to an embodiment of the present invention is shown.

[0134] like Figure 3 As shown, the cognitive saturation level recognition device 300 may include a graph-frequency dual-branch enhancement module 310, a cross-modal graph attention fusion module 320, a temporal evolution modeling module 330, and a cognitive saturation level output module 340.

[0135] The image-frequency dual-branch enhancement module 310 includes a frequency domain processing branch and two image domain processing branches, which process EEG signal data in parallel to obtain frequency domain EEG feature data, image domain EEG feature data and image domain physiological feature data. It can also perform channel attention fusion on the frequency domain EEG feature data and image domain EEG feature data to obtain EEG feature fusion data.

[0136] The cross-modal graph attention fusion module 320 can align EEG feature fusion data with graph domain physiological feature data at the second-level timestamp, and then fuse them to achieve cross-modal complementarity and spatial topology consistency, thereby obtaining multimodal feature fusion data.

[0137] The temporal evolution modeling module 330 can perform temporal evolution encoding on multimodal feature fusion data to obtain multimodal feature fusion encoded output data.

[0138] The cognitive saturation level output module 340 processes the multimodal feature fusion encoding output data by aggregating temporal context through global depthwise separable convolution. After passing through a fully connected layer, it outputs a three-level cognitive saturation level probability, which can be compared with a threshold through 5-point Hanning smoothing to output an end-to-end, interpretable, and time-stable cognitive saturation level.

[0139] The cognitive saturation fatigue identification method of this invention can be implemented through a trained deep learning model, and the training method of the deep learning model is as follows.

[0140] Figure 4 A flowchart of a model training method according to an embodiment of the present invention is shown.

[0141] like Figure 4 As shown, the model training method includes operations S410 to S460.

[0142] In operation S410, the sample EEG signal data of the sample identification object is subjected to spectral analysis to obtain sample frequency domain EEG feature data.

[0143] The sample EEG signal data includes sample EEG time-series data corresponding to at least one sample EEG channel, and sample frequency domain EEG feature data characterizes the energy distribution of the sample EEG signal data in multiple predetermined physiological frequency bands.

[0144] In operation S420, for any sample EEG channel in at least one sample EEG channel, based on the intra-channel visibility relationship between multiple sample EEG sampling values ​​in the sample EEG time series data corresponding to the sample EEG channel, sample channel map domain EEG feature data corresponding to the sample EEG channel is obtained, and sample map domain EEG feature data is obtained based on the sample channel map domain EEG feature data corresponding to each of the at least one EEG channel.

[0145] The sample channel map domain EEG feature data characterizes the connectivity of the sample EEG sampling times corresponding to each of the multiple sample EEG sampling values. The connectivity of the sample EEG sampling times is the number of other sample EEG sampling times that can be connected, determined based on the visibility relationship within the sample channel.

[0146] In operation S430, based on the cross-channel visibility relationship between sample physiological sampling values ​​in the sample physiological time series data corresponding to multiple physiological channels, sample map domain physiological feature data is obtained.

[0147] The physiological feature data of the sample map domain characterizes the connectivity of multiple sample physiological sample values. The connectivity of a sample physiological sample value is the number of other sample physiological sample values ​​that can be connected, determined based on the cross-channel visibility relationship of the sample.

[0148] In operation S440, the sample frequency domain EEG feature data, sample map domain EEG feature data, and sample map domain physiological feature data are fused to obtain sample multimodal feature fusion data.

[0149] In operation S450, the sample cognitive saturation level of the object being identified is determined based on the multimodal feature fusion data of the sample.

[0150] In the S460 operation, a deep learning model is trained based on a loss function, according to the sample cognitive saturation level and sample labels.

[0151] The processing procedures of operations S410 to S450 in the model training method of this embodiment are similar to the application process of the cognitive saturation fatigue identification method of this embodiment. Please refer to the description of the cognitive saturation fatigue identification method of this embodiment.

[0152] The loss function is as follows.

[0153] (15);

[0154] in, Represents the loss function. This represents the probability predicted by the model that the element belongs to class c. Indicates category index, This represents the weight coefficient of category c. Represents the regularization coefficient. This represents the output data of multimodal feature fusion encoding. This indicates the indicator value of category c in the sample label. This represents the total number of training samples. This represents the number of samples belonging to class c in the training samples.

[0155] The initial learning rate during the training of a deep learning model can be 1*10. -3 The training results of deep learning models can be verified by using 5-fold cross-period validation.

[0156] The fusion of sample frequency domain EEG feature data and sample map domain EEG feature data can be a spatial linear mixture. This allows us to obtain fusion data of the sample's electroencephalogram (EEG) features.

[0157] According to an embodiment of the present invention, determining the sample cognitive saturation level of the sample recognition object based on the sample multimodal feature fusion data includes: deleting the target number of fusion sub-data to be deleted from the sample multimodal feature fusion data to obtain sample multimodal feature fusion update data; and determining the sample cognitive saturation level of the sample recognition object based on the sample multimodal feature fusion update data.

[0158] To enhance the robustness of training, the target number of frames can be removed from the multimodal feature fusion data to obtain updated multimodal feature fusion data. Based on this updated data, the cognitive saturation level of the target object can be determined. The target number of frames can be, for example, 1, 2, or 3, and is not limited here.

[0159] Figure 5 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of the present invention, is shown. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0160] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 or a program loaded from storage portion 508 into RAM 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0161] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0162] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0163] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of the present invention. According to embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0164] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0165] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0166] For example, according to embodiments of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0167] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of the present invention.

[0168] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0169] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0170] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0172] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for identifying cognitive saturation level, characterized in that, include: Spectral analysis is performed on the EEG signal data of the object to be identified to obtain frequency domain EEG feature data, wherein the EEG signal data includes EEG time-series data corresponding to at least one EEG channel, and the frequency domain EEG feature data characterizes the energy distribution of the EEG signal data in multiple predetermined physiological frequency bands; For any one of the at least one EEG channels, channel map domain EEG feature data corresponding to the EEG channel is obtained based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel. Furthermore, map domain EEG feature data is obtained based on the channel map domain EEG feature data corresponding to each of the at least one EEG channel. The channel map domain EEG feature data characterizes the connectivity of EEG sampling moments corresponding to each of the multiple EEG sampling values, and the connectivity of the EEG sampling moments is the number of other EEG sampling moments that can be connected, determined based on the intra-channel visibility relationship. Based on the cross-channel visibility relationship between multiple physiological sample values ​​in the physiological time series data corresponding to multiple physiological channels included in the physiological signal data, graph domain physiological feature data is obtained, wherein the graph domain physiological feature data characterizes the connectivity of each of the multiple physiological sample values, and the connectivity of the physiological sample values ​​is the number of other physiological sample values ​​that can be connected, determined according to the cross-channel visibility relationship. The frequency domain EEG feature data, the map domain EEG feature data, and the map domain physiological feature data are fused to obtain multimodal feature fusion data; Based on the multimodal feature fusion data, the cognitive saturation level of the object to be identified is determined.

2. The method according to claim 1, characterized in that, The process of fusing the frequency domain EEG feature data, the map domain EEG feature data, and the map domain physiological feature data to obtain multimodal feature fusion data includes: Channel attention fusion is performed on the frequency domain EEG feature data and the map domain EEG feature data to obtain EEG feature fusion data. The feature dimension data in the EEG feature fusion data and the feature dimension data in the map domain physiological feature data are weighted and fused to obtain the multimodal feature fusion data.

3. The method according to claim 2, characterized in that, The process of performing channel attention fusion on the frequency domain EEG feature data and the map domain EEG feature data to obtain fused EEG feature data includes: The frequency domain EEG feature data and the graph domain EEG feature data are convolved to obtain intermediate frequency domain EEG feature data and intermediate graph domain EEG feature data, wherein the feature dimension of the intermediate graph domain EEG feature data is the same as the feature dimension of the intermediate frequency domain EEG feature data. Based on the feature dimensions of the intermediate frequency domain EEG feature data and the feature dimensions of the intermediate map domain EEG feature data, the intermediate frequency domain EEG feature data and the intermediate map domain EEG feature data are fused to obtain the fused EEG feature data.

4. The method according to claim 3, characterized in that, The step of fusing the intermediate frequency domain EEG feature data and the intermediate map domain EEG feature data according to the feature dimensions of the intermediate frequency domain EEG feature data and the feature dimensions of the intermediate map domain EEG feature data to obtain the EEG feature fusion data includes: The weights of the EEG feature fusion dimension are determined based on the correlation data between the intermediate frequency domain EEG feature data and the intermediate map domain EEG feature data in the feature dimension. The intermediate frequency domain EEG feature data and the intermediate map domain EEG feature data are fused according to the weighted dimensions of the EEG feature fusion to obtain the fused EEG feature data.

5. The method according to any one of claims 2 to 4, characterized in that, The step of weightedly fusing the feature dimension data from the EEG feature fusion data with the feature dimension data from the map-domain physiological feature data to obtain the multimodal feature fusion data includes: The physiological feature data of the map domain is convolved to obtain the physiological feature data of the intermediate map domain. The feature dimension data in the EEG feature fusion data is weighted and fused with the feature dimension data in the intermediate map domain physiological feature data to obtain EEG physiological feature connection data. The multimodal feature fusion data is obtained by performing edge attention convolution on the electroencephalogram (EEG) feature connectivity data.

6. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining channel map domain EEG feature data corresponding to the EEG channel based on the intra-channel visibility relationship among multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel includes: Based on predetermined EEG penetration parameters, the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel is determined, wherein the EEG penetration parameters characterize the number of EEG sampling moments that prevent the connection between the EEG sampling moment and other EEG sampling moments; Based on the intra-channel visibility relationship between multiple EEG sampling values ​​in the EEG time-series data corresponding to the EEG channel and the EEG connectivity sequence length, the channel map domain EEG feature data corresponding to the EEG channel is obtained, wherein the EEG connectivity sequence length characterizes the sequence length of connectivity at the EEG sampling time.

7. The method according to any one of claims 1 to 4, characterized in that, The process of obtaining map-domain physiological feature data based on the cross-channel visibility relationships between multiple physiological sample values ​​in the physiological time-series data corresponding to multiple physiological channels, including: The graph domain physiological feature data is obtained based on the cross-channel visibility relationship and physiological connectivity sequence length among multiple physiological sampling values ​​in the physiological time series data corresponding to multiple physiological channels, which are included in the physiological signal data. The physiological connectivity sequence represents the sequence length of connectivity at each physiological sampling time.

8. A model training method, characterized in that, include: Spectral analysis is performed on the sample EEG signal data of the sample identification object to obtain sample frequency domain EEG feature data, wherein the sample EEG signal data includes sample EEG time-series data corresponding to at least one sample EEG channel, and the sample frequency domain EEG feature data characterizes the energy distribution of the sample EEG signal data in multiple predetermined physiological frequency bands. For any one of the at least one sample EEG channels, based on the intra-channel visibility relationship between multiple sample EEG sampling values ​​in the sample EEG time series data corresponding to the sample EEG channel, sample channel map domain EEG feature data corresponding to the sample EEG channel is obtained; and based on the sample channel map domain EEG feature data corresponding to each of the at least one sample EEG channel, sample map domain EEG feature data is obtained, wherein the sample channel map domain EEG feature data characterizes the connectivity of sample EEG sampling times corresponding to each of the multiple sample EEG sampling values, and the connectivity of sample EEG sampling times is the number of other sample EEG sampling times that can be connected based on the intra-channel visibility relationship; Based on the cross-channel visibility relationship between multiple sample physiological sampling values ​​in the sample physiological time series data corresponding to multiple physiological channels, sample map domain physiological feature data is obtained. The sample map domain physiological feature data characterizes the connectivity of each of the multiple physiological sampling values. The connectivity of the sample physiological sampling values ​​is the number of other sample physiological sampling values ​​that can be connected, determined according to the cross-channel visibility relationship. The sample frequency domain EEG feature data, the sample map domain EEG feature data, and the sample map domain physiological feature data are fused to obtain sample multimodal feature fusion data; Based on the multimodal feature fusion data of the samples, the sample cognitive saturation level of the sample identification object is determined; Based on the loss function, a deep learning model is trained according to the cognitive saturation level of the samples and the sample labels.

9. The method according to claim 8, characterized in that, The step of determining the sample cognitive saturation level of the sample identification object based on the sample multimodal feature fusion data includes: The target number of fused sub-data to be deleted is removed from the sample multimodal feature fusion data to obtain the sample multimodal feature fusion update data; Based on the updated data fused from the multimodal features of the samples, the sample cognitive saturation level of the identified object is determined.

10. A cognitive saturation level recognition system, characterized in that, include: A physiological signal data acquisition module is configured to acquire physiological signal data of an object to be identified, wherein the physiological signal data includes physiological time-series data corresponding to multiple physiological channels. The electroencephalogram (EEG) signal data acquisition module is configured to acquire EEG signal data of the object to be identified, wherein the EEG signal data includes EEG timing data corresponding to at least one EEG channel; Electronic devices, including: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 9; The electrical stimulation module is configured to generate an electrical stimulation signal based on the cognitive saturation level output by the electronic device.

Images