Psychological load detection method, device and system

Abstract

The invention discloses a psychological load detection method, device and system, and the method comprises the steps: obtaining an electroencephalogram signal and an electrocardiosignal of a detected person; performing feature extraction on the electroencephalogram signals and the electrocardiosignals to obtain brain-heart interaction features; wherein the brain-heart interaction characteristics reflect the interaction between the brain activity and the cardiovascular system of the detected person; and inputting the brain-heart interaction characteristics into a psychological load detection model to obtain a psychological load category of the detected person.

Description

Technical Field

[0001] The present invention relates to the field of mental load technology, and more specifically, to a mental load detection method, device and system. Background Art

[0002] Mental load refers to the amount of psychological burden an individual bears when performing a task or activity. It includes not only the load on the individual's cognitive information processing and decision-making, but also the emotional and volitional load. Mental load is closely related to factors such as ability, personality, task difficulty, physiological variables, and social pressure. Long-term high mental load may lead to a decrease in an individual's attention, working memory, and executive ability, greatly increasing the probability of an individual making wrong decisions. Therefore, it is necessary to evaluate mental load.

[0003] Existing psychological load assessment methods include subjective assessment method, behavioral measurement method, performance measurement method, etc. Although they can reflect the individual's psychological state to a certain extent, their accuracy is low and it is difficult to objectively reflect the individual's psychological load level. Summary of the invention

[0004] One purpose of the embodiments of the present invention is to provide a new technical solution for mental load detection.

[0005] According to a first aspect of the present invention, there is provided a method for detecting mental load, comprising:

[0006] Obtaining the EEG signal and ECG signal of the subject;

[0007] Extracting features from the EEG signal and the ECG signal to obtain brain-heart interaction features; wherein the brain-heart interaction features reflect the interaction between the brain activity and the cardiovascular system of the subject;

[0008] The brain-heart interaction feature is input into a psychological load detection model to obtain the psychological load category of the subject.

[0009] Optionally, extracting features from the EEG signal and the ECG signal to obtain brain-heart interaction features includes:

[0010] Determining a first parameter of a heart rate variability model according to the EEG signal;

[0011] Updating the heart rate variability model using the first parameter to obtain an updated heart rate variability model;

[0012] Inputting the electrocardiogram signal into the updated heart rate variability model for feature extraction to obtain a heart rate variability sequence;

[0013] The heart rate variability sequence and the electroencephalogram signal are input into a brain-heart feature extraction model to obtain the brain-heart interaction feature.

[0014] Optionally, extracting features from the EEG signal and the ECG signal to obtain brain-heart interaction features includes:

[0015] Performing bandpass filtering on the EEG signal at a first set frequency to obtain a filtered EEG signal;

[0016] Correlating the electrocardiogram signal with the filtered electroencephalogram signal to obtain a correlation signal;

[0017] Extracting features from the correlation signal to obtain the brain-heart interaction features;

[0018] Before inputting the brain-heart interaction feature into the mental load detection model, the method further includes:

[0019] The brain-heart interaction feature is subjected to median compression processing to obtain a compressed brain-heart interaction feature as a new brain-heart interaction feature.

[0020] Optionally, the step of training to obtain the mental load detection model includes:

[0021] Acquire a training sample set; wherein the training sample set includes a plurality of sets of training EEG signals and training ECG signals corresponding to each category of mental load;

[0022] Extract features from each set of training EEG signals and training ECG signals to obtain a brain-heart interaction feature set;

[0023] The initial mental load detection model is trained by using the brain-heart interaction feature set to obtain a trained mental load detection model;

[0024] When the trained mental load detection model meets a preset condition, the trained mental load detection model is used as the mental load detection model.

[0025] Optionally, obtaining a training sample set includes:

[0026] Acquire multiple sets of training data for each subject under stimulation of different task difficulty levels; wherein each set of training data includes the training EEG signal and the training ECG signal, as well as a task difficulty assessment value;

[0027] Classifying the training data according to the difficulty level of the task;

[0028] For any category of training data, the psychological load category of the training data of the category is determined according to the task difficulty level and task difficulty evaluation value corresponding to the training data of the category, and the training sample set is obtained.

[0029] Optionally, the preset condition is that the evaluation index value of the trained mental load detection model is greater than or equal to a preset index value, and the evaluation index value includes at least one of average accuracy, precision, recall rate and F1 score.

[0030] Optionally, during the process of training the model, the method further includes:

[0031] Obtaining an estimated mental load category by inputting the brain-heart interaction feature set into the trained mental load detection model;

[0032] interpreting and analyzing the brain-heart interaction features corresponding to the same estimated mental workload category to determine the brain-heart interaction features associated with the estimated mental workload category;

[0033] According to the brain-heart interaction characteristics related to the estimated mental workload category, model parameters of the mental workload detection model during training are adjusted.

[0034] According to a second aspect of the present disclosure, there is provided a mental load detection device, comprising:

[0035] An acquisition module, used to acquire the EEG signal and ECG signal of the subject;

[0036] A feature extraction module, used to extract features from the EEG signal and the ECG signal to obtain brain-heart interaction features; wherein the brain-heart interaction features reflect the interaction between the brain activity and the cardiovascular system of the subject;

[0037] The model detection module is used to input the brain-heart interaction characteristics into the psychological load detection model to obtain the psychological load category of the subject.

[0038] According to a third aspect of the present disclosure, a mental load detection device is provided, comprising a memory and a processor, wherein the memory is used to store executable instructions; and the processor is used to operate according to the control of the instructions to execute the method as described in the first aspect.

[0039] According to a fourth aspect of the present disclosure, a mental load detection system is provided, comprising: an electroencephalogram sensor, an electrocardiogram sensor, and the mental load detection device according to the second aspect or the third aspect;

[0040] The electroencephalogram sensor is used to collect the electroencephalogram signal of the subject and send it to the mental load detection device;

[0041] The electrocardiogram sensor is used to collect the electrocardiogram signal of the subject and send it to the mental load detection device.

[0042] One beneficial effect of the present invention is that, by analyzing EEG signals and ECG signals, brain-heart interaction characteristics that reflect the interaction between the brain activity and cardiovascular system of the subject are obtained, which can more comprehensively reflect the psychological load state of the subject, and the brain-heart interaction characteristics are input into the psychological load detection model to obtain the psychological load category of the subject, which can improve the accuracy of psychological load detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0044] Figure 1 is a schematic diagram of the hardware structure of a mental load detection system according to an embodiment of the present invention;

[0045] Figure 2 is a flow chart of a method for detecting mental load according to an embodiment of the present invention;

[0046] Figure 3 is a flow chart of an ecological N-back paradigm experiment according to one embodiment of the present invention;

[0047] Figure 4 is a schematic diagram of stimuli at one task difficulty level according to the present invention;

[0048] Figure 5 is a global explanatory feature map of an estimated mental load category according to the present invention;

[0049] Figure 6 is a schematic structural diagram of a mental load detection device according to an embodiment of the present invention;

[0050] Figure 7 is a structural schematic diagram of a mental load detection device according to another embodiment of the present invention;

[0051] Figure 8 1 is a schematic diagram of the structure of a mental load detection system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0053] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0054] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0055] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0056] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0057] With the rapid development of aviation technology, the assessment of pilots' psychological load has become particularly important. When performing complex flight missions, pilots often face high-intensity cognitive load and psychological pressure, which not only affects flight safety, but may also have long-term effects on pilots' physical and mental health. Existing working memory load assessment methods, such as subjective assessment methods, behavioral measurement methods, and performance measurement methods, can reflect the psychological state of pilots to a certain extent, but they still have limitations in accuracy and applicability.

[0058] The embodiment of the present application obtains brain-heart interaction features for the interaction between the brain activity and the cardiovascular system of the subject by analyzing EEG signals and ECG signals, which can more comprehensively reflect the psychological load state of the pilot, and inputs the brain-heart interaction features into the psychological load detection model to obtain the pilot's psychological load category, which can improve the accuracy of psychological load detection.

[0059] The principle of this embodiment is described below by taking the detection of the psychological load of pilots as an example, but it can be understood that the specific application of the psychological load detection is not limited thereto. In fact, the psychological load detection method of this embodiment can be used to detect the psychological load of any object in any scenario. Without departing from the principle described in this embodiment, the detection method of this embodiment can be applied to the psychological load detection of any object in any scenario.

[0060] <Hardware Configuration>

[0061] Figure 1 is a block diagram of the hardware configuration of the mental load detection system 100 according to an embodiment of the present invention.

[0062] like Figure 1As shown, the mental load detection system 100 includes an electroencephalogram sensor 1000 and an electrocardiogram sensor 2000 , and a mental load detection device 3000 .

[0063] The EEG sensor may be an EEG (electroencephalogram) sensor, an ECoG (electrocorticogram) sensor, a SEEG (stereoencephalogram) sensor, etc., which is not limited here.

[0064] In some embodiments, the electroencephalographic sensor is an EEG sensor.

[0065] In this embodiment, EEG measures the electrical signals generated by the brain by placing electrodes on the scalp, wherein the electrodes may be arranged according to the international 10-20 system.

[0066] The electroencephalogram sensor collects the electroencephalogram signal of the subject and sends it to the mental load detection device 3000.

[0067] In this embodiment, refer to Figure 1 As shown, the EEG sensor 1000 may include a processor 1100 , a memory 1200 , an interface device 1300 , a communication device 1400 , a display device 1500 , an input device 1600 , and the like.

[0068] The processor 1100 may be a mobile processor. The memory 1200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 1300 may include, for example, a USB interface, a headphone interface, etc. The communication device 1400 may, for example, be capable of wired or wireless communication. The communication device 1400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 1400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 1500 may be, for example, a liquid crystal display, a touch display, etc., and the display device 1500 is used to display the collected EEG signals. The input device 1600 may include, for example, a touch screen, a keyboard, etc.

[0069] In this embodiment, the memory 1200 of the EEG sensor 1000 is used to store instructions, which are used to control the processor 1100 to operate to at least perform the mental load detection method implemented by the EEG sensor according to any embodiment of the present invention. Technicians can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0070] Despite Figure 1 , multiple devices of the EEG sensor 1000 are shown; however, the present invention may only involve some of the devices, for example, the EEG sensor 1000 only involves the memory 1200 and the processor 1100 .

[0071] The ECG sensor may be an electrocardiogram (ECG) sensor, a Holter sensor, etc., which is not limited here.

[0072] The electrocardiogram sensor collects the electrocardiogram signal of the subject and sends it to the mental stress detection device 3000 .

[0073] In this embodiment, refer to Figure 1 As shown, the electrocardiographic sensor 2000 may include a processor 2100 , a memory 2200 , an interface device 2300 , a communication device 2400 , a display device 2500 , an input device 2600 , and the like.

[0074] The processor 2100 may be a mobile version processor. The memory 2200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 2300 may include, for example, a USB interface, a headphone interface, etc. The communication device 2400 may, for example, be capable of wired or wireless communication. The communication device 2400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 2400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 2500 may be, for example, a liquid crystal display, a touch display, etc., and the display device 2500 is used to display the collected electrocardiogram signal. The input device 2600 may include, for example, a touch screen, a keyboard, etc.

[0075] In this embodiment, the memory 2200 of the psychological sensor 2000 is used to store instructions, and the instructions are used to control the processor 2100 to operate to at least perform the psychological load detection method performed by the electrocardiogram sensor 2000 according to any embodiment of the present invention. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0076] Despite Figure 1, multiple devices of the electrocardiogram sensor 2000 are shown; however, the present invention may only involve some of the devices, for example, the electrocardiogram sensor 2000 only involves the memory 2200 and the processor 2100 .

[0077] The mental load detection device 3000 is used to receive the EEG signals collected by the EEG sensor 1000 and the ECG signals collected by the ECG sensor 2000, and perform mental load detection based on the EEG signals and the ECG signals.

[0078] The mental load detection device 3000 may include a processor 3100, a memory 3200, an interface device 3300, a communication device 3400, a display device 3500, an input device 3600, and the like.

[0079] The processor 3100 may be a mobile version processor. The memory 3200 may include, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, etc. The interface device 3300 may include, for example, a USB interface, a headphone interface, etc. The communication device 3400 may, for example, be capable of wired or wireless communication. The communication device 3400 may include a short-range communication device, for example, any device for short-range wireless communication based on short-range wireless communication protocols such as Hilink protocol, WiFi (IEEE 802.11 protocol), Mesh, Bluetooth, ZigBee, Thread, Z-Wave, NFC, UWB, LiFi, etc. The communication device 3400 may also include a remote communication device, for example, any device for WLAN, GPRS, 2G / 3G / 4G / 5G remote communication. The display device 3500 may be, for example, a liquid crystal display, a touch display, etc. The input device 3600 may include, for example, a touch screen, a keyboard, etc.

[0080] In this embodiment, the memory 3200 of the mental load detection device 3000 is used to store instructions, and the instructions are used to control the processor 3100 to operate to at least execute the mental load detection method performed by the mental load detection device 3000 according to any embodiment of the present invention. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0081] Despite Figure 1 , multiple devices of the mental load detection device 3000 are shown, but the present invention may only involve some of the devices, for example, the mental load detection device 3000 only involves the memory 3200 and the processor 3100.

[0082] It should be understood that although Figure 1Only one EEG sensor 1000, one ECG sensor 2000, and one mental load detection device 3000 are shown, but this does not mean to limit their respective numbers. The mental load detection system 100 may include multiple EEG sensors 1000 and / or ECG sensors 2000 and / or mental load detection devices 3000.

[0083] <Method Example>

[0084] Figure 2 3 is a flow chart of a method for detecting mental load according to an embodiment of the present invention. The method can be implemented by a mental load detection device 3000.

[0085] according to Figure 2 As shown, the mental load detection method of this embodiment may include the following steps S2100 to S2300:

[0086] Step S2100, obtaining the electroencephalogram (EEG) signal and the electrocardiogram (ECG) signal of the subject.

[0087] In this embodiment, corresponding to the flight driving scenario, the person being detected may be a pilot, and corresponding to the high-speed rail driving scenario, the person being detected may be a high-speed rail driver, etc.

[0088] The EEG sensor collects the EEG signal of the subject, and the EEG sensor collects the EEG signal of the subject, and the EEG sensor collects the EEG signal of the subject, and the sampling frequency of the EEG sensor is the same as the sampling frequency of the EEG sensor.

[0089] Step S2200, extracting features from the EEG signal and the ECG signal to obtain brain-heart interaction features.

[0090] In this embodiment, the brain-heart interaction features (BHI) reflect the interaction between the brain activity and the cardiovascular system of the subject, that is, the complex mutual influence relationship between the heart and the brain.

[0091] Brain-heart interaction features are extracted through joint analysis, fusion technology, machine learning, deep learning and other methods, which are not limited here.

[0092] The joint analysis method can be used to extract brain-heart interaction features by extracting features from EEG and ECG signals at the same time, and then analyzing the correlation, synchronization or causal relationship between the two. The fusion technology can be used to extract brain-heart interaction features by using data fusion technology to fuse the features of EEG and ECG signals, and then extract comprehensive features that can reflect brain-heart interaction. Machine learning and deep learning can be used to extract brain-heart interaction features by applying machine learning methods, such as support vector machines, neural networks, etc., to classify and identify EEG and ECG signals, and then extract feature vectors that are sensitive to brain-heart interaction features through training models. The model-based method can be used to extract brain-heart interaction features by first establishing a physiological model or mathematical model of brain-heart interaction, and then extracting brain-heart interaction features through model parameters or output variables.

[0093] Those skilled in the art should understand that there are many ways to extract features from EEG signals and ECG signals to obtain brain-heart interaction features, which are not limited here.

[0094] In some embodiments, step S2200 performs feature extraction on the EEG signal and the ECG signal to obtain brain-heart interaction features, including: steps S2200.1 to S2200.4.

[0095] Step S2200.1, determining a first parameter of a heart rate variability model based on the EEG signal.

[0096] In this embodiment, the EEG signal is generated by the electrical activities of different brain regions. In the simulation of the EEG signal, the electrical activities of different brain regions can be modeled as a synthesis of multiple oscillators, each oscillator representing a specific frequency component. The amplitude of the multiple oscillators (i.e., the strength of the signal) is modeled as a first-order autoregressive signal of exogenous cardiac function, that is, the amplitude of the EEG signal is regarded as a quantity related to the heart rate variability, and then the first parameter of the heart rate variability model can be determined according to the amplitude of the EEG signal.

[0097] The heart rate variability (HRV) model is a model used to extract the heart rate variability sequence in the electrocardiogram signal. In the HRV model, its first parameter may include the basic heart rate, the long-term and short-term variability of the heart rate, the regulation parameters of the autonomic nervous system, etc.

[0098] For example, if the amplitude of the EEG signal is large, that is, it means that the subject is under high stress or high cognitive load, and the parameter value representing the activity of the sympathetic nervous system in the HRV model needs to be increased.

[0099] Step S2200.2: Update the heart rate variability model using the first parameter to obtain an updated heart rate variability model.

[0100] Step S2200.3, inputting the ECG signal into the updated heart rate variability model for feature extraction to obtain a heart rate variability sequence.

[0101] In this embodiment, the HRV model is updated by the first parameter, and the electrocardiogram signal is input into the updated HRV model to obtain a heart rate variability sequence, wherein the heart rate variability sequence reflects the influence of brain activity on the cardiovascular system.

[0102] Step S2200.4, inputting the heart rate variability sequence and the EEG signal into a brain-heart feature extraction model to obtain the brain-heart interaction feature.

[0103] In this embodiment, the brain-heart feature extraction model can be a statistical analysis model, a machine learning model, etc., which is not limited here.

[0104] In some embodiments, before determining the first parameter of the heart rate variability model according to the EEG signal in step S2200.1, the method further includes: performing signal processing on the EEG signal.

[0105] In this embodiment, the purpose of EEG signal processing is to obtain an artifact-free signal. The signal processing may specifically include frequency filtering, removal of large artifacts (such as eye movement and heart activity artifacts), interpolation of contaminated channels, and the like.

[0106] In some embodiments, the step S2200 extracts features from the EEG signal and the ECG signal to obtain brain-heart interaction features, including steps SA1 to SA3.

[0107] Step SA1: bandpass filter the EEG signal at a first set frequency to obtain a filtered EEG signal.

[0108] In this embodiment, specific frequency bands are associated with specific cognitive and physiological states. Typically, theta waves (4-8Hz), which are associated with relaxation, light sleep, meditation, and emotional regulation. Alpha waves (8-12Hz), which are associated with relaxation, alertness, and concentration, are enhanced when resting with eyes closed. Changes in alpha waves can reflect changes in psychological relaxation and cognitive load. Beta waves (12-30Hz), which are associated with positive thinking, attention, anxiety, and tension. The enhancement of beta waves is associated with high cognitive load or stress states. Gamma waves (30-45Hz), which are associated with advanced cognitive functions, information processing, and states of consciousness. Gamma waves play an important role in perception, attention, and memory integration.

[0109] The first set frequency may be θ(4-8] Hz, α(8-12] Hz, β(12-30] Hz, and γ[30-45] Hz.

[0110] By performing bandpass filtering of the first set frequency on the EEG signal, unnecessary frequency components can be removed and the effective components of the EEG signal can be increased.

[0111] Step SA2, correlating the electrocardiogram signal with the filtered electroencephalogram signal to obtain a correlation signal.

[0112] In this embodiment, the ECG signal and EEG signal in the associated signal are both signals of the first set frequency, that is, the associated signals are EEG signals and ECG signals of the first set frequency, so that when performing subsequent feature extraction, only the ECG signal and EEG signal of the first set frequency are focused on, thereby improving the efficiency of feature extraction.

[0113] Step SA3: extracting features from the correlation signal to obtain the brain-heart interaction features.

[0114] In this embodiment, brain-heart interaction features can be extracted through joint analysis, fusion technology, machine learning, deep learning and other methods, which are not limited here.

[0115] It should be noted that the method of extracting features from the associated signal in step SA3 may be the steps described in steps S2200.1 to S2200.4 above, or other methods, which are not limited here.

[0116] According to the embodiment of the present application, by performing bandpass filtering on the EEG signal at a first set frequency, unnecessary frequency components can be removed and the effective components of the EEG signal can be improved. Then, by correlating the ECG signal with the filtered EEG signal to obtain a correlated signal, and performing feature extraction on the correlated signal, brain-heart interaction features can be obtained, thereby improving the accuracy of feature extraction.

[0117] In some embodiments, before inputting the brain-heart interaction feature into the mental load detection model in step S2300, the method further includes:

[0118] The brain-heart interaction feature is subjected to median compression processing to obtain a compressed brain-heart interaction feature as a new brain-heart interaction feature.

[0119] In this embodiment, the time-resolved information of the directional BHI feature is compressed by calculating the median of the brain-heart interaction feature in non-overlapping 2-second windows to obtain a new brain-heart interaction feature for subsequent mental workload detection.

[0120] Compression can be some form of compression or dimensionality reduction of data to retain the most critical information.

[0121] According to the embodiment of the present application, by performing median compression processing on the filtered brain-heart interaction features to obtain compressed brain-heart interaction features, data redundancy and noise can be further reduced, and the data structure can be simplified to improve data processing efficiency. It also helps to reduce the complexity of the mental load detection model and reduce the risk of overfitting.

[0122] Step S2300: input the brain-heart interaction feature into a psychological load detection model to obtain the psychological load category of the subject.

[0123] In this embodiment, the mental load detection model can be obtained by training the training sample set. By inputting the brain-heart interaction feature into the mental load detection model, the mental load category of the subject can be obtained, for example, the mental load category is high load, medium load, and low load.

[0124] In some embodiments, the mental load detection model is a neural network (CNN) model, and its model structure is shown in Table 1:

[0125] As shown in Table 1, the CNN architecture takes a 2×2 block matrix of shape 25×21×1 as input and processes it using two convolutional layers with depths of 32 and 64, respectively, and a rectified linear unit (ReLU) activation function. The size of the convolutional filter is set to 3×3 to prevent overlapping electrode locations for different frequency blocks. Then, a flattening layer is inserted before the last two fully connected layers (containing 512 and 3 neurons, respectively). The last layer contains 3 neurons and uses a softmax activation function based on the number of categories to be recognized. In order to prevent overfitting during training, a dropout layer is added after the flattening layer.

[0126] Table 1 Neural network CNN architecture

[0127]

[0128]

[0129] In some embodiments, the step of training the mental load detection model includes: step S4100 to step S4400.

[0130] Step S4100, obtaining a training sample set.

[0131] In this embodiment, the training sample set includes a plurality of sets of training EEG signals and training ECG signals corresponding to each category of mental load.

[0132] For example, the mental workload includes three categories: high workload, medium workload, and low workload. Each category of mental workload includes 200 sets of training data, that is, the training sample set includes 600 sets of training data, and each set of training data includes training EEG signals and training ECG signals.

[0133] In some embodiments, the step S4100 of acquiring the training sample set includes: steps S4100.1 to S4100.3.

[0134] Step S4100.1, obtaining multiple sets of training data for each subject under stimulation of different task difficulty levels.

[0135] In this embodiment, the subjects have experience in operating flight driving games, so as to simulate the real flight process through flight driving games, and an ecological N-back paradigm experiment is designed to induce the psychological load of the subjects, and multiple sets of training data of the subjects under the stimulation of different task difficulty levels (3-back, 1-back, 2-back) are obtained. Each set of training data includes training EEG signals and the training ECG signals, as well as task difficulty evaluation values.

[0136] For example, Figure 3 As shown, the ecological N-back paradigm experimental design is as follows:

[0137] 1. Experimental Preparation

[0138] 1. Mission scenario simulation: In the experiment, a dynamic background of a real flight scenario is used to increase the realism and complexity of the mission. The background contains dynamically changing flight instruments and navigation information to simulate the operating environment of the pilot in actual flight.

[0139] 2. Multi-sensory stimulation: In addition to visual stimulation, unified synthetic audio cues are added, such as aircraft engine noise and communication conversations, to increase the multi-sensory load of the mission and make the mission closer to real flight situations.

[0140] 3. Subjects: 50 subjects had experience in operating flight driving games so as to simulate the real flight process through flight driving games.

[0141] 4. Ecological N-back paradigm experiment: The ecological N-back paradigm experiment includes two task blocks, each of which includes the same 100 aircraft model pictures. There is a 2-minute break between each task block. Each task block contains three N-back tasks of random difficulty levels (such as 2, 1, and 3-back), so the task difficulty level can be controlled by adjusting the value of N (such as 1-back, 2-back, and 3-back), that is, the larger the N value, the higher the task difficulty level.

[0142] 5. Signal monitoring: Set up ECG sensors and EEG sensors to monitor the subjects’ ECG signals and EEG signals.

[0143] 2. Experimental process

[0144] like Figure 4 As shown in the figure, it is the experimental process corresponding to a task difficulty level, as follows:

[0145] 1. Present pictures of aircraft models on the screen one by one, with each picture displayed for 500 milliseconds.

[0146] 2. After each airplane model image was presented, the screen displayed a “+” sign (fixation point) for 2 seconds to allow the subject to calm down and respond.

[0147] 3. The subject responds based on whether the currently presented aircraft model picture is the same as the previous N aircraft model pictures. If it is the same, press the left button; if it is different, press the right button.

[0148] 4. After the 100th aircraft model image of the task difficulty level is presented, the subject needs to input the task difficulty assessment value, that is, the task difficulty assessment value is the task difficulty score input subjectively by the subject. For example, the subject can input the task difficulty assessment value according to the task difficulty quantification table.

[0149] 5. After the first task block ends and before the second task block begins, the subjects can be prompted to adjust their status and improve their task completion based on their performance in the first task block, such as accuracy, reaction time and other indicators.

[0150] 6. During the ecological N-back paradigm experiment, the subjects' ECG signals and EEG signals are collected through ECG sensors and EEG sensors as training ECG signals and training EEG signals.

[0151] In this way, 600 sets of training data corresponding to each subject can be obtained, wherein each set of training data includes a training EEG signal and a training ECG signal, as well as a task difficulty assessment value.

[0152] Step S4100.2, classifying the training data according to the task difficulty level.

[0153] Continuing the above example, for any subject, 200 sets of training data corresponding to the three categories of mental load can be obtained, that is, 600 sets of training data. According to the task difficulty level, that is, the different N values, the training data of 50 subjects are classified.

[0154] Step S4100.3, for any category of training data, determine the psychological load category of the training data of the category according to the task difficulty level and task difficulty evaluation value corresponding to the training data of the category, and obtain the training sample set.

[0155] For example, for the training data of the category where N is 1 in the N-back paradigm experiment, according to the value of N and the task difficulty assessment value input by the subject, for example, the task difficulty assessment value is also 1, the psychological load category of the training data of this category is marked as low load.

[0156] For another example, for the training data of the category where N is 3 in the N-back paradigm experiment, based on the value of N and the task difficulty assessment value input by the subject, for example, the task difficulty assessment value is 9 (extremely high task difficulty), the psychological load category of the training data of this category is marked as high load.

[0157] For another example, for the training data of the category where N is 2 in the N-back paradigm experiment, based on the value of N and the task difficulty assessment value input by the subject, for example, the task difficulty assessment value is 4 (medium task difficulty), the psychological load category of the training data of this category is marked as medium load.

[0158] Step S4200: extract features from each set of training EEG signals and training ECG signals to obtain a brain-heart interaction feature set.

[0159] The method of executing step S4200 to extract brain-heart interaction features of the training EEG signal and the training ECG signal is the same as step S2200 and will not be elaborated here.

[0160] Step S4300: training the initial mental load detection model using the brain-heart interaction feature set to obtain a trained mental load detection model.

[0161] In this embodiment, the mental load detection model can be a three-classification model, and a suitable machine learning algorithm can be selected as the initial model. Common choices include logistic regression, support vector machine (SVM), random forest, neural network, etc.

[0162] The initial mental load detection model is trained using the brain-heart interaction feature set so that the estimated mental load of the mental load detection model approaches the mental load category of the training data marked by step S4100.3.

[0163] Step S4400: When the trained mental load detection model meets a preset condition, the trained mental load detection model is used as the mental load detection model.

[0164] In this embodiment, the preset condition may be that the number of model iterations is greater than or equal to a threshold number, or the preset condition may be that the evaluation index value of the model meets the requirements, which is not limited here.

[0165] In some embodiments, the preset condition is that the evaluation index value of the trained mental load detection model is greater than or equal to a preset index value, and the evaluation index value includes at least one of average accuracy, precision, recall rate and F1 score.

[0166] In this embodiment, after the trained mental load detection model is obtained through model iteration, the trained mental load detection model can be evaluated through the test sample set to obtain an evaluation index value, and then the trained mental load detection model is judged whether it meets the preset conditions according to the evaluation index value to determine whether the model training is completed. When the evaluation index value of the trained mental load detection model is greater than or equal to the preset index value, the model training is terminated to obtain the final mental load detection model, which is the mental load detection model in step S2300.

[0167] The method for acquiring the test sample set may be the same as the method for acquiring the training sample set in step S4100, which will not be elaborated here.

[0168] The model is evaluated by at least one of the average accuracy, precision, recall, and F1 score, which is not limited here.

[0169] Those skilled in the art should understand that average accuracy, precision, recall rate and F1 score are all commonly used indicator values ​​for model evaluation, and their specific meanings are common knowledge in the field and will not be elaborated here.

[0170] In order to verify the beneficial effects of brain-heart interaction features in psychological workload detection, the inventors designed a comparative experiment, namely, training the initial psychological workload detection model through EEG signals, and obtaining a first detection model when the number of model iterations is greater than or equal to a threshold number, and training the initial psychological workload detection model through brain-heart interaction features, and obtaining a second detection model when the number of model iterations is greater than or equal to a threshold number, which is the psychological workload detection model in step S2300 above, and then evaluating the first detection model and the second detection model respectively through a test sample set to verify the superiority of brain-heart interaction features in psychological workload detection.

[0171] It should be noted that, during the process of training the first detection model and the second detection model, the number threshold is set to the same value.

[0172] In the comparative experiment, the method of obtaining the second detection model through brain-heart interaction feature training is basically the same as that recorded in steps S4100 to S4400 above. The difference is that the preset condition in step S4400 is that the number of iterations of the model is greater than or equal to the number threshold, and the specific training process will not be repeated here.

[0173] Exemplarily, the process of training the mental load detection model through EEG signals is as follows:

[0174] Step S1, obtaining a training sample set through the ecological N-back paradigm experiment described above, each sample in the training sample set includes a training EEG signal and a task difficulty evaluation value.

[0175] Step S2: for any sample, perform signal processing on the training EEG signal to obtain a new training EEG signal.

[0176] The signal processing in this step is aimed at obtaining an artifact-free signal, and the signal processing includes frequency filtering, removal of large artifacts (such as eye movement and cardiac activity artifacts), and interpolation of contaminated channels.

[0177] Step S3, extracting the EEG power spectrum density from the new training EEG signal, and performing bandpass filtering on the EEG power spectrum density at a first set frequency to obtain a filtered EEG power spectrum density.

[0178] In this step, Welch's method was used to integrate the EEG signals every 2 seconds without overlapping to extract the EEG power spectral density (PSD).

[0179] Those skilled in the art should understand that Welch's method is a commonly used method for calculating PSD, and the detailed process of calculating PSD is not described here.

[0180] The first set frequency is θ(4-8]Hz, α(8-12]Hz, β(12-30]Hz, and γ[30-45]Hz.

[0181] Step S4, generating a grayscale image set according to the filtered EEG power spectrum density.

[0182] In this step, a spatial matrix is ​​constructed for each frequency band (i.e., θ, α, β, and γ), and then they are combined together to form a 2×2 block matrix. A padding space of 3 empty elements is placed between each block to avoid overlap between electrodes of different blocks during the convolution process. Finally, the matrix is ​​interpreted as a grayscale image.

[0183] Step S5, inputting the grayscale image set into the initial mental load detection model for training, and obtaining the first detection model when the number of model iterations is greater than or equal to a number threshold.

[0184] Step S6, using the test sample set, model evaluation is performed on the first detection model and the second detection model to obtain evaluation index values. As shown in Table 2, it is a table of evaluation index values ​​for the first detection model and the second detection model.

[0185] Table 2 Evaluation index values ​​of the first detection model and the second detection model

[0186] Input Features Average accuracy Accuracy Recall F1 score EEG power 78.02％ 78.45％ 78.09％ 78.09％ BHI 87.20％ 87.08％ 87.12％ 87.12％

[0187] Among them, the input feature is EEG power (brain electroencephalogram power spectrum density) corresponding to the first detection model, and the input feature is BHI corresponding to the second detection model. It can be seen that the mental load detection based on brain-heart interaction features shows superior performance in multiple evaluation index values ​​and has outstanding beneficial effects.

[0188] In some embodiments, after executing step S4100.3 for any category of training data, the psychological load category of the training data of the category is determined according to the task difficulty level and task difficulty evaluation value corresponding to the training data of the category, a sample set is obtained, and then a part of the sample set is divided into a training sample set and a part into a test sample set.

[0189] In some embodiments, during the process of training the model, the method further includes: steps S5100 to S5300.

[0190] Step S5100, obtaining an estimated mental load category by inputting the brain-heart interaction feature set into the trained mental load detection model.

[0191] In this embodiment, the mental load detection model obtained in each training is the trained mental load detection model. In the process of training the model, the trained mental load detection model can obtain an estimated mental load category according to any brain-heart interaction feature in the brain-heart interaction feature set. The estimated mental load category can be high load, medium load, and low load.

[0192] Step S5200 , interpreting and analyzing the brain-heart interaction features corresponding to the same estimated mental load category, and determining the brain-heart interaction features related to the estimated mental load category.

[0193] In this embodiment, the brain-heart interaction features corresponding to the same estimated mental load category can be classified into one category, and then the brain-heart interaction features of each category are interpreted and analyzed through the Grad-CAM algorithm, and the obtained global interpretation is expressed in the form of a topographic map of feature importance.

[0194] Those skilled in the art should understand that the interpretation and analysis of brain-heart interaction features by the Grad-CAM algorithm is well known in the art, and the algorithm will not be described in detail here.

[0195] For example, Figure 5 As shown in Figure 1, it is a global explanation feature map for the estimated mental load category of low load. Among them, the dark area in the image represents features that are more important for predicting valence (the estimated mental load category is low load), while the bright area represents features that are less important for predicting valence, that is, in the dark area, the brain-heart interaction features in a specific frequency band have a great influence on the classification results of the low load category, while in the bright area, the influence is small. In addition, in the wide dark area, the mental load is distributed over the entire scalp, while in the restricted area, the model only focuses on a small part of the brain-heart interaction features and ignores other parts. In the α and θ bands, there are large areas of dark areas in the frontal lobe, temporal lobe, and occipital lobe. In the β and γ bands, the dark areas are relatively restricted, which means that the mental load is only concentrated in certain parts of the frontal lobe and occipital lobe. In addition, by comparing these frequency bands, it can be clearly seen that when the estimated mental load category is low, the mental load detection model selects the BHI features in the θ and α bands as more relevant features than the β and γ bands.

[0196] Step S5300: adjusting the model parameters of the mental load detection model during the training process according to the brain-heart interaction characteristics related to the estimated mental load category.

[0197] For example, during the model training process, different weights can be assigned to features according to their importance. For another example, a classification threshold can be set according to the statistical distribution of the brain-heart interaction features. For another example, a suitable model structure can be selected according to the complexity of the brain-heart interaction features. The loss function can be modified according to the characteristics of the brain-heart interaction features to optimize the performance of the model, etc., which are not limited here.

[0198] According to an embodiment of the present application, by analyzing EEG signals and ECG signals, brain-heart interaction characteristics that reflect the interaction between the brain activity and the cardiovascular system of the subject are obtained, which can more comprehensively reflect the psychological load state of the subject. The brain-heart interaction characteristics are input into the psychological load detection model to obtain the psychological load category of the subject, which can improve the accuracy of psychological load detection.

[0199] <Device Example>

[0200] Figure 6 4 is a principle block diagram of a mental load detection device according to an embodiment of the present invention.

[0201] In this embodiment, if Figure 6 As shown, the mental load detection device 600 includes: an acquisition module 610, a feature extraction module 620, and a model detection module 630.

[0202] An acquisition module 610 is used to acquire the EEG signal and the ECG signal of the subject;

[0203] A feature extraction module 620 is used to extract features from the EEG signal and the ECG signal to obtain brain-heart interaction features; wherein the brain-heart interaction features reflect the interaction between the brain activity and the cardiovascular system of the subject;

[0204] The model detection module 630 is used to input the brain-heart interaction feature into the mental load detection model to obtain the mental load category of the subject.

[0205] In some embodiments, the feature extraction module 620 is specifically used to determine a first parameter of a heart rate variability model based on the EEG signal; update the heart rate variability model using the first parameter to obtain an updated heart rate variability model; input the EEG signal into the updated heart rate variability model for feature extraction to obtain a heart rate variability sequence; input the heart rate variability sequence and the EEG signal into a brain-heart feature extraction model to obtain the brain-heart interaction feature.

[0206] In some embodiments, the feature extraction module 620 is further used to perform bandpass filtering on the EEG signal at a first set frequency to obtain a filtered EEG signal; associate the ECG signal with the filtered EEG signal to obtain a correlated signal; perform feature extraction on the correlated signal to obtain the brain-heart interaction feature;

[0207] The device 600 also includes a compression module 640, which is further used to perform median compression processing on the brain-heart interaction feature before inputting the brain-heart interaction feature into the mental load detection model to obtain the compressed brain-heart interaction feature as a new brain-heart interaction feature.

[0208] In some embodiments, the device 600 also includes a model training module 650, which is used to obtain a training sample set; wherein the training sample set includes multiple groups of training EEG signals and training ECG signals corresponding to each category of psychological load; feature extraction is performed on each group of training EEG signals and training ECG signals to obtain a brain-heart interaction feature set; the initial psychological load detection model is trained by the brain-heart interaction feature set to obtain a trained psychological load detection model; when the trained psychological load detection model meets preset conditions, the trained psychological load detection model is used as the psychological load detection model.

[0209] In some embodiments, the model training module 650 is also used to obtain multiple groups of training data for each subject under stimulation of different task difficulty levels; wherein each group of training data includes the training EEG signal and the training ECG signal, as well as a task difficulty assessment value; the training data is classified according to the task difficulty level; for any category of training data, the psychological load category of the training data of the category is determined according to the task difficulty level and the task difficulty assessment value corresponding to the training data of the category, to obtain the training sample set.

[0210] In some embodiments, the preset condition is that the evaluation index value of the trained mental load detection model is greater than or equal to a preset index value, and the evaluation index value includes at least one of average accuracy, precision, recall rate and F1 score.

[0211] In some embodiments, the model training module 650 is also used to obtain an estimated psychological load category during the model training process by inputting the brain-heart interaction feature set into the trained psychological load detection model; interpreting and analyzing the brain-heart interaction features corresponding to the same estimated psychological load category to determine the brain-heart interaction features related to the estimated psychological load category; and adjusting the model parameters of the psychological load detection model during the training process according to the brain-heart interaction features related to the estimated psychological load category.

[0212] Figure 7 4 is a principle block diagram of a mental load detection device according to an embodiment of the present invention.

[0213] In this embodiment, if Figure 7 As shown, the mental load detection device 700 includes a processor 710 and a memory 720, wherein the memory 720 stores programs or instructions that can be executed on the processor, and when the programs or instructions are executed by the processor 710, the steps of the mental load detection method as described in any of the above method embodiments are implemented.

[0214] Figure 8 4 is a principle block diagram of a mental load detection system according to an embodiment of the present invention.

[0215] In this embodiment, if Figure 8 As shown, the mental load detection system 800 includes: an electroencephalogram sensor 810, an electrocardiogram sensor 820 and a mental load detection device 830.

[0216] The EEG sensor 810 is used to collect EEG signals of the subject and send them to the mental load detection device 830;

[0217] The electrocardiogram sensor 820 is used to collect the electrocardiogram signal of the subject and send it to the mental load detection device 830 .

[0218] In some embodiments, the mental load detection device 830 is as follows Figure 1 The mental load detection device or Figure 6 The mental stress test device shown, or Figure 7 The mental stress testing device shown.

[0219] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0220] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0221] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0222] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0223] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0224] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0225] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0226] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0227] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for detecting mental load, characterized in that: include: Obtaining the EEG signal and ECG signal of the subject; Extracting features from the EEG signal and the ECG signal to obtain brain-heart interaction features; wherein the brain-heart interaction features reflect the interaction between the brain activity and the cardiovascular system of the subject; The brain-heart interaction feature is input into a psychological load detection model to obtain the psychological load category of the subject.

2. The method according to claim 1, characterized in that The extracting features of the EEG signal and the ECG signal to obtain brain-heart interaction features includes: Determining a first parameter of a heart rate variability model according to the EEG signal; Updating the heart rate variability model using the first parameter to obtain an updated heart rate variability model; Inputting the electrocardiogram signal into the updated heart rate variability model for feature extraction to obtain a heart rate variability sequence; The heart rate variability sequence and the electroencephalogram signal are input into a brain-heart feature extraction model to obtain the brain-heart interaction feature.

3. The method according to claim 1, characterized in that The feature extraction of the EEG signal and the ECG signal to obtain the brain-heart interaction feature includes: Performing bandpass filtering on the EEG signal at a first set frequency to obtain a filtered EEG signal; Correlating the electrocardiogram signal with the filtered electroencephalogram signal to obtain a correlation signal; Extracting features from the correlation signal to obtain the brain-heart interaction features; Before inputting the brain-heart interaction feature into the mental load detection model, the method further includes: The brain-heart interaction feature is subjected to median compression processing to obtain a compressed brain-heart interaction feature as a new brain-heart interaction feature.

4. The method according to claim 1, characterized in that: The steps of training and obtaining the mental load detection model include: Acquire a training sample set; wherein the training sample set includes a plurality of sets of training EEG signals and training ECG signals corresponding to each category of mental load; Extract features from each set of training EEG signals and training ECG signals to obtain a brain-heart interaction feature set; The initial mental load detection model is trained by using the brain-heart interaction feature set to obtain a trained mental load detection model; When the trained mental load detection model meets a preset condition, the trained mental load detection model is used as the mental load detection model.

5. The method according to claim 4, characterized in that The obtaining of the training sample set comprises: Acquire multiple sets of training data for each subject under stimulation of different task difficulty levels; wherein each set of training data includes the training EEG signal and the training ECG signal, as well as a task difficulty assessment value; Classifying the training data according to the difficulty level of the task; For any category of training data, the psychological load category of the training data of the category is determined according to the task difficulty level and task difficulty evaluation value corresponding to the training data of the category, and the training sample set is obtained.

6. The method according to claim 5, characterized in that The preset condition is that the evaluation index value of the trained mental load detection model is greater than or equal to a preset index value, and the evaluation index value includes at least one of average accuracy, precision, recall rate and F1 score.

7. The method according to claim 4, characterized in that During the process of training the model, the method further includes: Obtaining an estimated mental load category by inputting the brain-heart interaction feature set into the trained mental load detection model; interpreting and analyzing the brain-heart interaction features corresponding to the same estimated mental workload category to determine the brain-heart interaction features associated with the estimated mental workload category; According to the brain-heart interaction characteristics related to the estimated mental workload category, model parameters of the mental workload detection model during training are adjusted.

8. A psychological load detection device, characterized in that: include: An acquisition module, used to acquire the EEG signal and ECG signal of the subject; A feature extraction module, used to extract features from the EEG signal and the ECG signal to obtain brain-heart interaction features; wherein the brain-heart interaction features reflect the interaction between the brain activity and the cardiovascular system of the subject; The model detection module is used to input the brain-heart interaction characteristics into the psychological load detection model to obtain the psychological load category of the subject.

9. A psychological load detection device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the mental load detection method according to any one of claims 1 to 7 are implemented.

10. A psychological load detection system, characterized in that: include: An electroencephalogram sensor, an electrocardiogram sensor, and a mental load detection device as claimed in claim 9 or 8; The electroencephalogram sensor is used to collect the electroencephalogram signal of the subject and send it to the mental load detection device; The electrocardiogram sensor is used to collect the electrocardiogram signal of the subject and send it to the mental load detection device.

Images