Pilot mental load assessment system based on emotion calculation

Abstract

The invention discloses a pilot mental load assessment system based on emotion calculation. The pilot mental load assessment system comprises an information acquisition and processing part and a psychological assessment and intervention report generation part. Specifically, a wearable physiological signal acquisition device is provided for a pilot in traditional simulated flight training, electroencephalogram, electrocardio and skin electrical signals of the pilot are acquired in the training process, and the emotion level of the pilot during training is judged through comprehensive processing of a common person emotion model and a data algorithm, so that the training accuracy of the pilot is improved. According to the judgment result, mental load and pressure level indexes of the pilot at a certain moment in training are obtained through comprehensive analysis, and the analyzed oscillogram of each index changing according to the time axis is taken as an output result; and finally generating physiological data, flight training data and an emotion evaluation result in training collected in training, and formulating a targeted training intervention scheme according to the individual emotion state of the pilot. According to the method, the psychological quality problem of the pilot can be well corrected, and finally an ideal training effect is achieved.

Description

Technical Field

[0001] The present invention relates to the fields of biomedical engineering and precision medicine, brain science and brain-like computing, and specifically is a pilot mental workload assessment system based on affective computing. Background Art

[0002] Flying is a special profession that requires pilots to have good psychological qualities. In recent years, personal mistakes by pilots have led to a large number of flight accidents. Accident analysis shows that psychological abnormalities, emotional instability, poor stress tolerance, improper operation, and crew collaboration loopholes are all important factors that cause flight accidents. Therefore, establishing a pilot psychological training model that meets the needs of the development of the modern aviation industry is of great practical significance for enhancing the psychological quality of pilots and promoting the improvement of aviation safety levels.

[0003] At present, in addition to theoretical training and physical training, pilot flight training methods include real aircraft training and flight simulation training. Compared with traditional real aircraft training, flight simulation training can avoid the fuel consumption, daily maintenance and other costs brought by real aircraft training, and is more economical. It can simulate different flight scenarios through simulators, improve the pertinence of training, and simulated flight also increases the safety of training and greatly improves the fault tolerance of training. Therefore, it is widely used in flight training.

[0004] Current flight simulators roughly include four types: driving technology training simulators, combat technology training simulators, full-mission flight simulators, and distributed mission training simulation systems. They can be used for training in driving technology, combat practical technology, various airborne equipment operations, multi-aircraft contract tactics and other subjects. Flight simulators are of great help in the development of basic driving skills at the basic level in the initial training stage, as well as in the later advanced and high-level basic driving skills training.

[0005] However, current flight training is mainly based on universal training models and lacks personalized training plans. It is difficult to develop personalized training plans based on the physiological and psychological differences of individual flight trainees. The psychological quality problems exposed by the pilots themselves cannot be corrected well, and ultimately the ideal training effect cannot be achieved. Summary of the invention

[0006] The purpose of the present invention is to provide a wearable physiological signal acquisition device for pilots in traditional simulated flight training, collect the pilot's electroencephalogram, electrocardiogram, and skin conduction signals during the training process, and judge the pilot's emotional level during training through the comprehensive processing of ordinary people's emotional model + data algorithm. According to the judgment result, the mental load and stress level index of the pilot at a certain moment in the training are obtained through comprehensive analysis, and the waveform diagram of the analyzed indicators changing along the time axis is used as the output result, and finally the physiological data collected during the training, flight training data, emotional evaluation results during the training, and targeted training intervention plans formulated according to the individual emotional state of the pilot are generated.

[0007] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a pilot mental load assessment system based on affective computing, characterized by including two parts: information collection and processing and psychological assessment and intervention report generation;

[0008] The information collection and processing includes flight training data collection and physiological data collection, wherein the collected flight training data includes environmental information and aircraft data during flight training, and the collected physiological data includes electrocardiogram, electrodermal conductivity, and electroencephalogram data of the flight trainee in a calm state and during training;

[0009] The psychological assessment and intervention report generation includes training records, assessment results, intervention suggestions, and physiological data;

[0010] The training record includes basic information and training information of the flight trainee;

[0011] The evaluation results include: outputting the pilot's psychological state level and stress level index under stress, as well as a stress level waveform chart changing along the time axis; outputting the pilot's fear, anger, and tension index under stress, the three emotion levels, and three emotion index waveform charts changing along the time axis; outputting the pilot's mental load index under stress, as well as a mental load waveform chart changing along the time axis; flight data changing along the time axis; judging the stress level by analyzing the emotional level and mental load index of the trainees under stress;

[0012] The intervention suggestion is a quantitative result of emotions obtained based on the evaluation result, and a corresponding intervention suggestion is given;

[0013] The physiological data are the original data of electrocardiogram, electrodermal conductivity and electroencephalogram of the trainees in this training.

[0014] Furthermore, the environmental information includes weather, time, and duration, and the aircraft data is recorded in real time along a timeline, including changes in instrument data, flight attitude, flight altitude, and flight speed.

[0015] Furthermore, the ECG signal is acquired by collecting and analyzing the characteristics of the ECG signal to obtain the parasympathetic nerve activity, including:

[0016] (1) Average RR duration

[0017]

[0018] The average RR duration is the arithmetic mean of all RR intervals measured in ms; RRi is the i-th RR interval, and N is the number of all RR intervals measured;

[0019] (2) Average heart rate

[0020] The average heart rate is calculated from the electrocardiogram:

[0021] HR(BPM)=60*(N R -1) / ((S2-S1) / F s )

[0022] Where Fs is the sampling rate of the signal in Hz, the sample number of the first R wave detected is S1 and the sample number of the last R wave detected is S2, N R is the total number of R waves between S1 and S2, including the waves at S1 and S2, and BPM is the number of heart beats per minute;

[0023] (3)SDNN

[0024]

[0025] SDNN is the standard deviation of the RR intervals within the measurement time range, in milliseconds. The standard deviation describes the dispersion of the individual RR intervals around their mean. The greater the dispersion, the higher the standard deviation. On the contrary, if all measurements are the same, the standard deviation will be 0.

[0026] (4)RMSSD

[0027]

[0028] RMSSD is the square root of the mean of all squares of the differences between consecutive RR intervals, expressed in milliseconds; RMSSD measures the short-term variation in consecutive RR intervals and is a reasonable measure of parasympathetic activation;

[0029] (5)pNN50

[0030]

[0031] pNN50 is the fraction of all consecutive RR interval pairs that differ from each other by more than 50 ms; to calculate this parameter, determine the number of consecutive RR interval pairs that differ by more than 50 ms and divide this value by the number of all consecutive RR interval pairs; pNN50 is an informed measure of parasympathetic activation;

[0032] (6) Frequency domain analysis

[0033] The discrete Fourier transform is used to transform the time series of heartbeat intervals into the frequency domain and express it in the form of power spectral density or spectrum distribution.

[0034] Furthermore, the GSR signal is obtained by acquisition, and the GSR signal includes the skin conductance level SCL and the skin conductance response SCR:

[0035] (1) Skin conductance level (SCL)

[0036] SCL is the absolute value of skin conductance between two points on the skin. An increase in SCL reflects an increase in attention / emotional arousal, while a decrease in SCL represents a decrease in attention / emotional arousal.

[0037] (2) Skin conductance response (SCR)

[0038] SCR is the phased sympathetic nerve discharge, which is a physiological and psychological activation state caused by stimulation. It reflects the index of short-term brain processes and is used as a positioning marker for the emotional arousal point caused by external novel events.

[0039] Furthermore, GSR analysis focuses on event-related SCRs and extracts three Hiorth features: activity A, mobility M, and complexity C, where activity A is used to measure the average power of the GSR signal and is the variance of the GSR signal X; mobility M is used to measure the proximity of the signal to the average frequency and is defined as the ratio of the standard deviation of the first-order derivative of the variance to the standard deviation; complexity C is used to measure the frequency change and the proximity to the GSR signal bandwidth and is defined as the ratio of the mobility of the first-order derivative of the function to the mobility of the function.

[0040] Furthermore, the calculation formulas for activity A, mobility M, and complexity C are:

[0041]

[0042]

[0043]

[0044] σx is the standard deviation of the GSR signal x, σx2 is the variance of x, σd corresponds to the standard deviation of the first derivative of the variance of x, and σdd corresponds to the standard deviation of the second derivative of the variance of x.

[0045] Furthermore, the EEG signal is acquired by collection, and then the following features of the EEG signal are extracted, as follows:

[0046] (1) Event-related potential (ERP): The EEG voltage fluctuations caused by discrete stimulation events reflect the process of cognitive processing. The characteristics of ERP include latency, amplitude and polarity, and are analyzed by averaging multiple EEG signals under the same stimulation.

[0047] (2) Differential entropy DE: It is a measure of signal complexity and is used to represent the complexity of signals in a specific frequency band. For a specific EEG signal x with a length range of |a, b| and approximately following a Gaussian distribution, which can be expressed as N(μ,σi2)), its differential entropy is expressed as:

[0048]

[0049] (3) Differential symmetry DASM: measures the difference in power spectrum or other signal characteristics between symmetrical electrode pairs (Xileft, Xiright):

[0050] and

[0051]

[0052] (4) Ratio symmetry RASM:

[0053]

[0054] (5) Frequency domain features: represents the power distribution of the EEG signal in each frequency band. The power spectrum density of the EEG signal x(n) with a length of n is as follows:

[0055]

[0056] Among them, the autocorrelation function r(k)=R[x(n)x * (n+k)], E is the expectation of the function, and x* is the complex conjugate of x.

[0057] Furthermore, by preprocessing and extracting features from EEG, ECG, and galvanic skin signals, we obtain the indexes and levels of the three emotions of fear, anger, and anxiety. The levels and indexes correspond to low, medium, and high levels, respectively. When any of the three emotions appears to be medium or high, its emotional level is marked as negative, otherwise it is marked as neutral.

[0058] Furthermore, EEG signals are used to judge the mental load status, which is divided into no load, low load, medium load, and high load. Judgment and classification are carried out based on the identification results of emotional level and mental load to obtain the stress level of the subjects. According to the stress level, the psychological state of the pilots under stress is divided into 5 levels, namely: very poor, poor, acceptable, good, and excellent.

[0059] Furthermore, intervention recommendations include advising the student to strengthen stress relief training and practice progressive relaxation techniques, including: experiencing the feeling of relaxation through sequential alternating contraction and relaxation of skeletal muscles, reducing whole body tension, and relieving anxiety and mental stress.

[0060] The beneficial effects of the present invention are:

[0061] (1) The present invention can determine the emotional level of pilots during training, and comprehensively analyze the mental load and stress level indicators of the pilots at a certain moment during training based on the determination results. The waveform diagram of the analyzed indicators changing along the time axis is used as the output result, and finally the physiological data collected during training, flight training data, emotional evaluation results during training, and targeted training intervention plans formulated according to the individual emotional state of the pilots are generated.

[0062] (2) During the training process, the pilot's electroencephalogram, electrocardiogram, and electrical skin signals are collected. After processing and analysis, the time axis waveforms of the three indicators of the pilot's emotional level, mental workload, and stress level are obtained. These can objectively reflect the changes in the pilot's psychological state during training and the deficiencies in his psychological quality. This presentation of the differences in individual pilot qualities is something that traditional flight simulators cannot do.

[0063] (3) While presenting the trainees’ psychological qualities in the training report, a training intervention plan can also be developed to compensate for these deficiencies. The plan includes precautions before training and specific steps during training. It is scientific and effective, and makes up for the deficiency of traditional flight simulators in general training that lacks specific training for individual differences.

[0064] (4) It can be used to track and analyze historical training results after multiple training sessions and to conduct horizontal analysis of the training results of multiple students. It helps aviation schools and instructors understand the psychological quality and emotional level of students, facilitates the improvement of student information databases, and facilitates the planning and arrangement of future training and flight missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of the operation of the evaluation system of the present invention;

[0066] Figure 2 It is a 3-lead or 5-lead ECG wiring diagram;

[0067] Figure 3It is a schematic diagram of ECG signal preprocessing and feature extraction;

[0068] Figure 4 This is the GSR wiring diagram;

[0069] Figure 5 It is a schematic diagram of GSR signal preprocessing and feature extraction;

[0070] Figure 6 This is a wiring diagram for EEG signal collection;

[0071] Figure 7 It is the characteristic graph of ERP;

[0072] Figure 8 This is a schematic diagram of 14-lead EEG signals;

[0073] Fig. 9 It is the analysis frame diagram of the evaluation results;

[0074] Fig.10 It is the block diagram of the modal psychology algorithm;

[0075] Fig.11 This is a sample diagram of an evaluation report. DETAILED DESCRIPTION

[0076] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0077] A pilot mental workload assessment system based on affective computing includes two parts: information collection and processing, and psychological assessment and intervention report generation.

[0078] 1. Flight Collection and Processing

[0079] 1. Flight training data

[0080] Record environmental information during flight training, including weather, time, and duration.

[0081] Aircraft data during flight training is recorded in real time on a timeline, including but not limited to instrument data changes, flight attitude, flight altitude, flight speed, etc.

[0082] 2. Physiological data collection

[0083] The present invention will collect and analyze the pilot's electrocardiogram, electrodermal conductivity and electroencephalogram data during the training process. The system needs to measure the degree of change of continuous heart rate, electrodermal conductivity and electroencephalogram, especially the physiological signal data before and after the stressor acts, so it is also necessary to collect calm state data.

[0084] (1) ECG data

[0085] Soothing music is played for 5 minutes before the flight training begins, and the ECG data of the student in a calm state is collected. After the flight training begins, the ECG signals are collected according to the flight training timeline.

[0086] Collect 3 channels of ECG signals. The collection principle is as follows: Figure 2 As shown, the wiring diagram can produce a total of 7-lead ECG leads I, II, III, V (usually V1), aVR, aVL, and aVF.

[0087] The specific conversion relationship is as follows:

[0088] Lead I is the potential difference between the left arm (positive electrode, LA) and the right arm (RA);

[0089] Lead II is the potential difference between the left leg (positive electrode, LL) and the right arm (RA):

[0090] Lead III is the potential difference between the left leg (positive, LL) and the left arm (LA):

[0091] Lead V (usually V1) is the potential difference between the chest (positive, V) and the hybrid electrode (negative, Wilson's central terminal); the Wilson's central terminal is created by connecting the RA, LA, and LL electrodes through a resistor network and represents the average body voltage.

[0092] Right upper limb pressurized unipolar limb lead (aVR): The positive electrode (white) is located in the right arm, and the negative electrode is the synthesis of the left arm electrode (black) and the left leg electrode (red), which can enhance (i.e. pressurize) the signal strength of the positive electrode of the right arm:

[0093] Left upper limb angiographic unipolar limb lead (aVL): The positive electrode (black) is located in the left arm, and the negative electrode is the synthesis of the right arm electrode (white) and the left leg electrode (red), which can enhance the signal strength of the positive electrode in the left arm:

[0094] Left lower limb pressurized unipolar limb lead (aVF): The positive electrode (red) is located in the left leg, and the negative electrode is the synthesis of the right arm electrode (white) and the left arm electrode (black), which can enhance the signal strength of the positive electrode of the left leg:

[0095] The obtained ECG signals are first screened for high signal quality for analysis. Heart rate variability (HRV) analysis helps to understand how the autonomic nervous system (ANS) is associated with different emotional states. HRV measurements show that negative emotions such as tension and fear are inversely proportional to parasympathetic nerve activity. For example, if parasympathetic nerve activity decreases, the degree of negative emotions such as tension and fear increases.

[0096] By analyzing the characteristics of ECG signals, we can obtain the parasympathetic nerve activity, such as the average RR duration, average heart rate, SDNN, etc., as follows:

[0097] 1) Average RR duration

[0098] The average RR duration is mathematically considered as the arithmetic mean (in [ms]) of all RR intervals measured. RRi is the i-th RR interval and N is the number of all RR intervals measured.

[0099]

[0100] 2) Average heart rate

[0101] The average heart rate can be calculated from the electrocardiogram. Fs is the sampling rate of the signal in (Hz), the sample number of the first R wave detected is (S1) and the sample number of the last R wave detected is (S2), (NR) is the total number of R waves between S1 and S2, including the waves at S1 and S2, and BPM is the number of beats per minute.

[0102] HR(BPM)=60*(N R -1) / ((S2-S1) / F s )

[0103] 3) SDNN (standard deviation of RR interval)

[0104] SDNN is the standard deviation of the RR intervals in the measurement time range, in [ms]. The standard deviation describes how dispersed the individual RR intervals are around their mean. The greater the dispersion, the higher the standard deviation. Conversely, if all measurements were identical, the standard deviation would be 0.

[0105]

[0106] SDNN is a measure of overall autonomic nervous system activation, the sum of sympathetic and parasympathetic activation. It indicates the ability of the autonomic nervous system to regulate processes in the body.

[0107] 4) RMSSD (root mean square distance)

[0108] Mathematically, RMSSD is the square root of the mean of all squares of the differences between consecutive RR intervals, also expressed in [ms].

[0109]

[0110] RMSSD measures the short-term variation in consecutive RR intervals. It is considered a reasonable measure of parasympathetic activation and is often interpreted as an indicator of resilience, health, and wellness.

[0111] The RMSSD value shows how quickly the body reacts to stress. A high value means that the body copes well with the alternation between load and relief. Thus, conclusions can be drawn about the stress load.

[0112] 5) pNN50

[0113] pNN50 is the fraction of all consecutive RR interval pairs that differ from each other by more than 50ms. To calculate this parameter, the number of consecutive RR interval pairs with a difference greater than 50ms is determined and this value is divided by the number of all consecutive RR interval pairs.

[0114]

[0115] pNN50 is closely related to RMSSD and is also considered a valid measure of parasympathetic activation. However, it is more sustainable and more representative of general power reserve than RMSSD. Initial deterioration can be read more quickly from it.

[0116] 6) Frequency domain analysis

[0117] The discrete Fourier transform is used to convert the time series of heartbeat intervals into the frequency domain, which is expressed in the form of power spectrum density or spectral distribution. It is mainly divided into high-frequency area (0.15-0.40 Hz) and low-frequency area (0.04-0.15 Hz). The high-frequency area usually reflects the activity of the parasympathetic nerves, and the low-frequency area is regulated by both the sympathetic and parasympathetic nervous systems.

[0118] (2) Skin electrical data

[0119] Soothing music was played for 5 minutes before the flight training began, and the skin conduction data of the trainee in a calm state were collected. After the flight training began, the skin conduction signals were collected according to the flight training timeline.

[0120] Collect 1 channel of Galvanic Skin Response (GSR) signal. Skin resistance changes with the state of the skin sweat glands. Sweating is controlled by the sympathetic nervous system, and skin conductance is an indicator of psychological or physiological arousal. If the sympathetic branch of the autonomic nervous system is highly aroused, then sweat gland activity will also increase, thereby increasing skin conductance. GSR is the only physiological indicator that is not affected by the parasympathetic nervous system. In this way, skin conductance can be used as a measure of emotions and sympathetic responses, and is therefore widely used in the study of different psychological states such as stress, emotions, fatigue, and cognitive load.

[0121] Since the hands and feet have the largest number of sweat glands, GSR is usually collected by placing sensors on the hands. Figure 4 shown.

[0122] The most commonly used time domain features of GSR are signal statistical parameters. The GSR signal is mainly composed of slowly changing basal (Tonic) activity - skin conductance level (SCL) and rapidly changing phasic (Phasic) activity - skin conductance response (SCR).

[0123] 1) Skin conductance level SCL

[0124] SCL is the absolute value of skin conductance between two points on the skin, also known as "basal skin conductance". It is often used to reflect the level of emotional arousal at different stages. An increase in SCL generally reflects an increase in attention / emotional arousal, while a decrease in SCL represents a decrease in attention / emotional arousal.

[0125] SCL varies among individuals and is related to personality traits: those with higher basic levels tend to be more introverted, nervous, anxious, emotionally unstable, and overly sensitive; while those with lower basic levels tend to be more cheerful, extroverted, have a more balanced mentality, be more confident, and have better psychological adaptability.

[0126] 2) Skin conductance response (SCR)

[0127] SCR is a phased sympathetic nerve discharge, a physiological and psychological activation state caused by stimulation, an indicator of short-term brain processes, and can be used as a marker for the location of emotional arousal points caused by external novel events. SCR is sensitive to specific emotional stimulation events and generally occurs between 1 and 5 seconds after the emotional stimulation occurs. GSR analysis focuses on event-related SCRs because they can be interpreted as a direct measure of the subject's arousal and involvement.

[0128] 3) Hjorth characteristics

[0129] The concept of Hjorth parameters is applied to feature extraction and analysis of physiological signals such as EEG. Hjorth parameters consist of three descriptors: activity, mobility, and complexity.

[0130] Activity is used to measure the average power of the GSR signal; it is the variance of the GSR signal X.

[0131] Mobility is used to measure the proximity of the signal to the average frequency; it is defined as the ratio of the standard deviation of the first derivative of the variance to the standard deviation.

[0132] Complexity is used to measure the frequency variation and the proximity to the GSR signal bandwidth; it is defined as the ratio of the mobility of the first derivative of the function to the mobility of the function. If the signal is similar to a pure sine wave, the complexity value converges to 1.

[0133]

[0134]

[0135]

[0136] σx is the standard deviation of the GSR signal x, σx2 is the variance of x, σd corresponds to the standard deviation of the first derivative of the variance of x, and σdd corresponds to the standard deviation of the second derivative of the variance of x.

[0137] (3) EEG data

[0138] Soothing music was played for 5 minutes before the flight training began, and the EEG data of the trainee in a calm state was collected. After the flight training began, the EEG signals were collected according to the flight training timeline.

[0139] Collect 14 channels of EEG signals. According to the International Clinical Neurophysiology Federation EEG electrode placement standard guidelines, the EEG signal collection wiring is as follows Figure 6 shown.

[0140] Extract relevant features of EEG signals, including event-related potential (ERP), differential entropy (DE), etc., to monitor emotions, mental workload and stress levels. The details are as follows:

[0141] 1) Event-related potential (ERP)

[0142] Definition: It refers to the change in potential caused by applying a specific stimulus to a sensory system or a part of the brain, giving or removing the stimulus. ERP reflects the process of cognitive processing.

[0143] Formula: The characteristics of ERP include latency, amplitude and polarity, and are usually analyzed by averaging multiple EEG signals under the same stimulus. The positive component that reaches a peak about 100 milliseconds after the stimulus is presented is called "P100" and the negative component is called "N100".

[0144] Application scenarios and advantages: ERP is widely used in the field of neuroscience to understand cognitive processes, especially in accurately measuring specific sensory, cognitive and motor processes.

[0145] 2) Differential Entropy (DE)

[0146] Definition: A measure of signal complexity, often used to represent the complexity of signals in a specific frequency band.

[0147] Formula: For an EEG signal x of a certain length (in the range of |a, b|) that approximately obeys a Gaussian distribution (expressed as N(μ,σi2)), its differential entropy can be expressed as:

[0148]

[0149] Application scenarios and advantages: When used as a feature in emotion recognition, DE provides a quantitative measure of signal complexity and outperforms other EEG features in some scenarios.

[0150] 3) Differential Symmetry (DASM)

[0151] Definition and formula: Measures the difference in power spectrum or other signal characteristics between a symmetrical electrode pair (Xileft, Xiright).

[0152]

[0153] Application scenarios and advantages: Useful in the study of emotional and cognitive processes, especially when assessing asymmetries between the two brain hemispheres.

[0154] 4) Ratio Symmetry (RASM)

[0155] Definition and formula: Similar to DASM, but uses characteristic ratios instead of differences.

[0156]

[0157] Application scenarios and advantages: Also used in emotion and cognitive research, providing relative measurements between symmetrical electrode pairs.

[0158] 5) Frequency domain characteristics (such as power spectral density)

[0159] Definition: It represents the power distribution of each frequency band of EEG signal.

[0160] Formula: Usually the signal is decomposed into its frequency components using methods such as Fourier transform for calculation. The power spectral density of an EEG signal x(n) of length n is as follows:

[0161]

[0162] Among them, the autocorrelation function r(k)=E[x(n)x * (n+k)], E is the expectation of the function, and x* is the complex conjugate of x.

[0163] Application scenarios and advantages: Frequency domain features are the basis of EEG analysis, which helps to understand the oscillatory characteristics of brain activity and its association with different states or tasks.

[0164] 2. Psychological Assessment and Intervention Report

[0165] After each training session, students can view and print the training evaluation report immediately. After logging out, they can view the report on the instructor's end.

[0166] The psychological assessment intervention report includes four parts: training records, assessment results, intervention suggestions and physiological data.

[0167] (1) Training records

[0168] Including basic information: student name, student number, instructor name, training time; training information: training aircraft model, weather, simulation time, training duration, fault type, fault time, fault duration.

[0169] (2) Evaluation results

[0170] The evaluation results of this program include the following parts:

[0171] Output the pilot's psychological state level and stress level index under stress, as well as the stress level waveform chart changing along the time axis;

[0172] Output the pilot's fear, anger, and tension indexes under stress, the levels of the three emotions, and the waveforms of the three emotion indexes changing along the time axis;

[0173] Output the pilot's mental workload index under stress and the mental workload waveform chart that changes along the time axis;

[0174] Flight data changing on a timeline;

[0175] The correlation analysis of several waveform graphs and flight data of the same time unit is carried out. By analyzing the three emotional levels and mental load index of the trainees under stress state, the stress level is objectively judged and the analysis conclusion is drawn.

[0176] The following describes how to obtain these evaluation results one by one;

[0177] 1) Identification of three levels of emotions

[0178] Usually under the influence of stressors, pilots' emotions, cognition, and behavior will undergo a series of changes, mainly manifested in:

[0179] ① Fear: The emotional response directly associated with acute stress is fear. When a disaster occurs, people's first reaction is fear, or even panic. When fear occurs, the organism will mobilize energy to cope with the stressor and produce behavioral responses such as escaping and avoiding danger. Excessive fear will cause individuals to lose their rationality and become confused.

[0180] ②Anger: Anger is also an emotional response closely related to acute stress. When a disaster or other stressors occur, when individuals believe that the stressful events are caused by human factors that can be controlled, they will feel strong anger, lose their rationality, and engage in verbal aggression.

[0181] ③Anxiety: Anxiety is a common psychological stress response. It is generally believed that anxiety includes three characteristics: one is tension and fear; the second is irritability and restlessness; the third is worry and anxiety. These characteristics constitute a vague premonition that misfortune and danger are approaching, and the resulting complex negative emotional state is anxiety. After anxiety occurs, sympathetic nervous system hyperactivity often occurs, such as increased heart rate, increased blood pressure, deeper breathing, sweating, trembling limbs, etc. Further development may lead to increased parasympathetic nervous system activity, such as diarrhea.

[0182] By preprocessing and extracting features from EEG, ECG and galvanic skin signals and analyzing them through algorithms, we can derive the indexes and levels of the three emotions: fear, anger and anxiety. The levels and indexes correspond to low (0), medium (1) and high (2), respectively.

[0183] 2) Emotional level recognition

[0184] The emotional level can be determined based on the levels of the three emotions. When any of the three emotions appears to be medium (1) or high (2), its emotional level is marked as negative (2), otherwise it is marked as neutral (1).

[0185] 3) Mental workload identification

[0186] The EEG signal is used to judge the mental load status, which is divided into no load 0, low load 1, medium load 2, and high load 3.

[0187] 4) Acquisition of stress level and psychological state

[0188] Based on the identification results of emotional level and mental load, judgment and classification are carried out to obtain the stress level of the subjects. According to the stress level, the psychological state of pilots under stress can be divided into five levels, namely: very poor / poor / acceptable / good / excellent. The judgment criteria of stress level and psychological state are shown in Table 1.

[0189] Table 1 Criteria for judging mental state

[0190]

[0191] Evaluation results: ① Output the pilot's psychological state level and stress level index under stress, as well as the stress level waveform chart changing along the time axis; ② Output the pilot's fear, anger, and tension index under stress, the three emotion levels, and the three emotion index waveform chart changing along the time axis; ③ Output the pilot's mental load index under stress, as well as the mental load waveform chart changing along the time axis; ④ Flight data changing along the time axis; ⑤ Correlate and analyze several waveform charts with the flight data of the same time unit. By analyzing the three emotion levels and mental load index of the trainees under stress, the stress level can be objectively judged and the analysis conclusion can be drawn.

[0192] (3) Intervention recommendations

[0193] Based on the emotional quantification results obtained from the assessment, corresponding intervention suggestions are given.

[0194] Examples of intervention recommendations:

[0195] It is recommended that the student strengthen stress relief training and use progressive relaxation techniques to do the following exercises:

[0196] Progressive relaxation training is to reduce whole body tension, relieve anxiety and mental stress by alternately contracting and relaxing skeletal muscles in sequence and experiencing the feeling of relaxation.

[0197] 1) Notes:

[0198] You should be in a comfortable, quiet, dimly lit place. Before training, go to the bathroom and undress. Sit comfortably on a sofa or lie on the bed. Contract each muscle for 5 to 7 seconds, then relax for 20 to 30 seconds. There should be enough space between contraction and relaxation to create a sharp contrast between the feeling of relaxation and tension.

[0199] Each training session should last 20 to 30 minutes and should generally be done before going to bed at night, once a day.

[0200] If you keep training, you will achieve full body relaxation after 2 weeks.

[0201] 2) Specific steps:

[0202] Relaxation of arm muscles: Place both hands flat on the sofa armrests, clench your fists with your palms facing up, and tense the muscles of both hands and forearms. Hold for 5 to 7 seconds, then relax for 20 to 30 seconds (the tension and relaxation time is omitted below). Bend both elbows to tense the biceps, then relax.

[0203] Relaxation of head muscles: frown, tense the frontal muscles, and then relax. Close your eyes, tighten the eye muscles, and then relax. Wrinkle your nose and facial muscles, and then relax. Clench your teeth, pay attention to the tension of the lower jaw, and then relax. Use your tongue to press against the upper jaw, tense the tongue, and then relax. Close your lips tightly, tense the muscles, and then relax.

[0204] Relax the neck muscles: bend your head downward with your chin against your chest, then relax. Then lean your head back as far as you can, turn your head to the right, then to the left, experience the shift in tension points, then relax.

[0205] Relax chest muscles: Bring your shoulders forward and together, tense the muscles around your chest, and then relax them. You will feel a comfortable and relaxed feeling in your chest.

[0206] Relax the shoulder muscles: Stretch your arms and legs outward so that they float in the air. Try your best to lift your shoulders upward, then relax them, and pay attention to the feeling of heat and heaviness.

[0207] Relax the back muscles: bend your back backwards, trying to make your chest and abdomen protrude, hold for a while, and then relax. Expand your shoulders behind you, bring them together as much as possible to tighten the upper back muscles, and then relax.

[0208] Relax your abdominal muscles: Raise your legs to tense your abdominal muscles, then relax them. Pay attention to the changes in your abdominal sensation from tense to relaxed.

[0209] Relax the buttocks muscles: straighten your legs and lay them flat on the ground, press down your calves and heels to tense your buttocks muscles, then relax. Then clamp your buttocks, try to lift the pelvis, then relax. At this time, you will feel the buttocks muscles begin to heat up and feel heavy.

[0210] Relaxation of thigh muscles: Tighten your legs, lift your heels off the ground, and then relax. Straighten your legs and tighten your knees, and then relax. Pay attention to the feeling of slight warmth.

[0211] Calf muscle relaxation: Bend your feet toward your knees to tense your calf muscles, then slowly relax them. Bend your feet forward and downward with force, then relax them. Notice how the tension disappears.

[0212] Relaxation of toe muscles: bend your feet upwards slowly and forcefully, without moving your ankles and legs, and gradually relax them. Then bend your feet downwards slowly and forcefully, and then relax. During the training, you can silently say: "I feel very comfortable, very quiet, very relaxed, very relaxed..." Let this idea of ​​relaxation spread throughout the body along with the relaxation of every part of the body.

[0213] (4) Physiological data

[0214] The original data of ECG, electrodermal and EEG of the trainees in this training.

[0215] The technical indicators involved in the present invention are:

[0216] Hardware indicators:

[0217] The ShimmerGSR+ galvanic skin response acquisition system produced by Shimmer is compatible with the Shimmer3 platform.

[0218] The Shimmer ECG & EMG acquisition system produced by Shimmer is compatible with the shimmer3 platform.

[0219] Emotiv EPOC X from Emotiv.

[0220] Software indicators:

[0221] GSR and ECG signal acquisition and processing use the Consensys software carried by the Shimmer3 platform. We use the most advanced Consensys v1.6.0PRO version of the platform.

[0222] EEG signal acquisition and processing uses the data acquisition software EmotivPRO produced by Emotiv and matched with Emotiv EPOC X.

[0223] Performance indicators:

[0224] Physiological signal acquisition delay: ≤20ms;

[0225] Micro SD memory card storage: ≤32G;

[0226] GSR three-lead, 128Hz, 12-bit resolution;

[0227] GSR measurement range: 10KΩ-4.7MΩ(.2uS-100uS)+ / -10%, 22KΩ-680KΩ(1.5-45uS)+ / -3%;

[0228] GSR frequency measurement range: DC-15.9Hz;

[0229] ECG five-lead, 256Hz, 12-bit resolution;

[0230] ECG bandwidth: 8.4KHz;

[0231] ECG storage memory: 2048 bytes;

[0232] EEG 14 leads, 128Hz, 14bit resolution;

[0233] Mental load level: 0, 1, 2, 3;

[0234] Three basic emotions: fear, anger, and anxiety;

[0235] Emotion level rating: 0: positive, 1: neutral, 2: negative;

[0236] The evaluation results showed that: within 5 minutes after the training;

[0237] Emotional level judgment accuracy: 72%.

[0238] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0239] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. A pilot mental workload assessment system based on affective computing, characterized in that: It includes two parts: information collection and processing, and psychological assessment and intervention report generation; The information collection and processing includes flight training data collection and physiological data collection, wherein the collected flight training data includes environmental information and aircraft data during flight training, and the collected physiological data includes electrocardiogram, electrodermal conductivity, and electroencephalogram data of the flight trainee in a calm state and during training; The psychological assessment and intervention report generation includes training records, assessment results, intervention suggestions, and physiological data; The training record includes basic information and training information of the flight trainee; The evaluation results include: outputting the pilot's psychological state level and stress level index under stress, as well as a stress level waveform chart changing along the time axis; outputting the pilot's fear, anger, and tension index under stress, the three emotion levels, and three emotion index waveform charts changing along the time axis; outputting the pilot's mental load index under stress, as well as a mental load waveform chart changing along the time axis; flight data changing along the time axis; judging the stress level by analyzing the emotional level and mental load index of the trainees under stress; The intervention suggestion is a quantitative result of emotions obtained based on the evaluation result, and a corresponding intervention suggestion is given; The physiological data are the original data of electrocardiogram, electrodermal conductivity and electroencephalogram of the trainees in this training.

2. A pilot mental load assessment system based on affective computing according to claim 1, characterized in that: The environmental information includes weather, time, and duration, and the aircraft data is recorded in real time along a timeline, including instrument data changes, flight attitude, flight altitude, and flight speed.

3. The pilot mental load assessment system based on affective computing according to claim 1, characterized in that: By collecting ECG signals, the characteristics of the ECG signals are analyzed to obtain the parasympathetic nerve activity, including: (1) Average RR duration The average RR duration is the arithmetic mean of all RR intervals measured in ms; RRi is the i-th RR interval, and N is the number of all RR intervals measured; (2) Average heart rate The average heart rate is calculated from the electrocardiogram: HR(BPM)=60*(N R -1) / ((S2-S1 / F s ) Where Fs is the sampling rate of the signal in Hz, the sample number of the first R wave detected is S1 and the sample number of the last R wave detected is S2, N R is the total number of R waves between S1 and S2, including the waves at S1 and S2, and BPM is the number of heart beats per minute; (3)SDNN SDNN is the standard deviation of the RR intervals within the measurement time range, in milliseconds. The standard deviation describes the dispersion of the individual RR intervals around their mean. The greater the dispersion, the higher the standard deviation. On the contrary, if all measurements are the same, the standard deviation will be 0. (4)RMSSD RMSSD is the square root of the mean of all squares of the differences between consecutive RR intervals, expressed in milliseconds; RMSSD measures the short-term variation in consecutive RR intervals and is a reasonable measure of parasympathetic activation; (5)pNN50 pNN50 is the fraction of all consecutive RR interval pairs that differ from each other by more than 50 ms; to calculate this parameter, determine the number of consecutive RR interval pairs that differ by more than 50 ms and divide this value by the number of all consecutive RR interval pairs; pNN50 is an informed measure of parasympathetic activation; (6) Frequency domain analysis The discrete Fourier transform is used to transform the time series of heartbeat intervals into the frequency domain and express it in the form of power spectral density or spectrum distribution.

4. The pilot mental load assessment system based on affective computing according to claim 1, characterized in that: The GSR signal is obtained by acquisition, and the GSR signal includes the skin conductance level SCL and the skin conductance response SCR: (1) Skin conductance level (SCL) SCL is the absolute value of skin conductance between two points on the skin. An increase in SCL reflects an increase in attention / emotional arousal, while a decrease in SCL represents a decrease in attention / emotional arousal. (2) Skin conductance response (SCR) SCR is the phased sympathetic nerve discharge, which is a physiological and psychological activation state caused by stimulation. It reflects the index of short-term brain processes and is used as a positioning marker for the emotional arousal point caused by external novel events.

5. The pilot mental load assessment system based on affective computing according to claim 4, characterized in that: GSR analysis focuses on the SCR related to the event and extracts three Hiorth features: Activity A, mobility M, complexity C, where: Activity A is used to measure the average power of the GSR signal, which is the variance of the GSR signal X; Mobility M is used to measure the proximity of the signal to the average frequency and is defined as the ratio of the standard deviation of the first-order derivative of the variance to the standard deviation. Complexity C is used to measure the frequency variation and the proximity to the GSR signal bandwidth and is defined as the ratio of the mobility of the first-order derivative of the function to the mobility of the function.

6. A pilot mental workload assessment system based on affective computing according to claim 5, characterized in that: The calculation formulas for activity A, mobility M, and complexity C are: σx is the standard deviation of the GSR signal x, σx2 is the variance of x, σd corresponds to the standard deviation of the first derivative of the variance of x, and σdd corresponds to the standard deviation of the second derivative of the variance of x.

7. The pilot mental load assessment system based on affective computing according to claim 1, characterized in that: The EEG signal is obtained by acquisition, and then the following features of the EEG signal are extracted, as follows: (1) Event-related potential (ERP): The EEG voltage fluctuations caused by discrete stimulation events reflect the process of cognitive processing. The characteristics of ERP include latency, amplitude and polarity, and are analyzed by averaging multiple EEG signals under the same stimulation. (2) Differential entropy DE: It is a measure of signal complexity and is used to represent the complexity of signals in a specific frequency band. For a specific EEG signal x with a length range of |a, b| and approximately following a Gaussian distribution, which can be expressed as N(μ,σi2)), its differential entropy is expressed as: (3) Differential symmetry DASM: measures the difference in power spectrum or other signal characteristics between symmetrical electrode pairs (Xileft, Xiright): (4) Ratio symmetry RASM: (5) Frequency domain features: represents the power distribution of the EEG signal in each frequency band. The power spectrum density of the EEG signal x(n) with a length of n is as follows: Wherein, the autocorrelation function r(k)=E[x(n)x*(n+k)], E is the expectation of the function, and x* is the complex conjugate of x.

8. The pilot mental load assessment system based on affective computing according to claim 1, characterized in that: By preprocessing and extracting features from EEG, ECG, and galvanic skin signals, we obtain the indexes and levels of the three emotions: fear, anger, and anxiety. The levels and indexes correspond to low, medium, and high levels, respectively. When any of the three emotions appears to be medium or high, its emotional level is marked as negative, otherwise it is marked as neutral.

9. The pilot mental load assessment system based on affective computing according to claim 1, characterized in that: The EEG signals are used to judge the mental load status, which is divided into no load, low load, medium load and high load. The stress level of the subjects is obtained by judgment and classification based on the identification results of emotional level and mental load. According to the stress level, the psychological state of pilots under stress is divided into five levels, namely: very poor, poor, acceptable, good and excellent.

10. The pilot mental load assessment system based on affective computing according to claim 1, characterized in that: The intervention recommendations include advising the student to strengthen stress relief training and practice progressive relaxation techniques, including: experiencing the feeling of relaxation through sequential alternating contraction and relaxation of skeletal muscles, reducing whole body tension, and relieving anxiety and mental stress.

Images