Method and apparatus for pilot workload identification

Abstract

The present application relates to the technical field of computer, in particular to a pilot workload identification method and device, electronic equipment, storage medium and program product. Physiological time series data, electroencephalogram time series data, eye movement image data and face image data corresponding to a target pilot are acquired. The physiological time series data, electroencephalogram time series data, eye movement image data and face image data are input into a preset workload identification model, and a current workload level corresponding to the target pilot is output. Based on the current workload level, a warning information is output. Through timely warning, the risk of flight accidents caused by excessive workload of pilots can be effectively reduced, and flight safety can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a pilot workload identification method and device, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of aviation technology, modern aircraft systems are becoming increasingly complex and automated, but this has not reduced the workload of pilots, but rather has placed higher demands on their cognitive, decision-making and operational abilities. Pilots need to handle multiple tasks such as navigation, communication, aircraft state monitoring, and abnormal situation handling during flight, and the level of workload directly affects flight safety and efficiency. According to statistics from the International Civil Aviation Organization (ICAO), a significant proportion of aviation accidents are closely related to judgment errors and operational delays caused by excessive pilot workload. In addition, the pilot workload differs significantly under different flight stages (such as takeoff, cruising, and landing) and complex weather conditions, and how to accurately identify the workload state has become a key problem in ensuring flight safety, optimizing flight training, and improving human-machine interaction performance.

[0003] Traditional workload assessment methods based on questionnaires and subjective rating scales (such as the NASA-TLX scale) have obvious limitations. These methods rely on pilots' post-event subjective recall and evaluation, which is easily influenced by factors such as memory bias and emotional state, and cannot reflect the real workload during flight in real time and dynamically. At the same time, with the growing demand for intelligence and automation in the aviation industry, there is an urgent need for more objective, accurate and real-time workload identification technology to meet the operational and development needs of modern aviation systems.

[0004] Therefore, how to accurately identify the pilot workload has become a problem to be solved. SUMMARY

[0005] Therefore, the present application provides a pilot workload identification method and device to solve the problem of how to accurately identify the pilot workload.

[0006] In a first aspect, the present application provides a pilot workload identification method, which comprises:

[0007] acquiring physiological time series data, electroencephalogram time series data, eye movement image data and facial image data corresponding to a target pilot; the physiological time series data includes at least one of electrocardiogram data, electromyogram data and respiration data;

[0008] inputting the physiological time series data, electroencephalogram time series data, eye movement image data and facial image data into a preset workload identification model, and outputting a current workload level corresponding to the target pilot;

[0009] Output an early warning information based on the current workload level.

[0010] The pilot workload identification method provided by the embodiments of the present application acquires physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data corresponding to a target pilot; inputs the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the facial image data into a preset workload identification model, and outputs a current workload level corresponding to the target pilot. The pilot workload identification method comprehensively considers various data features and their mutual relationships, greatly improving the accuracy and stability of workload identification. In addition, the pilot workload identification method can adapt to various complex scenarios. Whether in the key stages of takeoff and landing, or in the face of adverse weather or unexpected situations, the model can accurately judge the workload of the pilot according to the input data, providing a reliable basis for subsequent decision-making. Based on the current workload level, an early warning information is output. The pilot himself, the flight management personnel, and the ground control personnel can timely understand the working state of the pilot. When the early warning information shows that the pilot is in a high workload state, the pilot can adjust his own work rhythm and attention allocation to avoid operation errors caused by excessive fatigue or information overload; the flight management personnel can reasonably arrange tasks according to the situation and provide support or adjust the flight plan if necessary; the ground control personnel can pay more attention and assistance to ensure the safety of the flight process. Through timely early warning, the risk of flight accidents caused by excessive workload of the pilot can be effectively reduced, and flight safety can be ensured.

[0011] In an optional implementation, the preset workload identification model includes a first feature recognition network and a second feature recognition network, the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the facial image data are input into the preset workload identification model, and the current workload level corresponding to the target pilot is output, including:

[0012] The physiological time series data and the electroencephalogram time series data are input into the first feature extraction network to generate time series modal features;

[0013] The facial image data and the eye movement image data are input into the second feature extraction network to generate visual modal features;

[0014] The time series modal features and the visual modal features are fused to generate target fusion features;

[0015] Based on the target fusion features, the current workload level corresponding to the target pilot is output.

[0016] In an optional implementation, the first feature extraction network includes a first sub-feature extraction branch, a second sub-feature extraction branch, and a first sub-feature fusion network, the physiological time series data and the electroencephalogram time series data are input into the first feature extraction network to generate time series modal features, including:

[0017] inputting the physiological time series data into a first sub-feature extraction branch and inputting the electroencephalogram time series data into a second sub-feature extraction branch;

[0018] The first sub-feature extraction branch extracts features from the physiological time series data to generate a physiological signal sequence, which includes at least one of respiratory data features, electrocardiogram data time domain features, electrocardiogram data frequency domain features, electrocardiogram data nonlinear features, and electromyogram data features.

[0019] The physiological signal sequence is extracted based on a sparse attention scoring method to generate a physiological signal matrix.

[0020] The physiological signal matrix is extracted based on a multi-cycle convolution kernel to output physiological modality features.

[0021] The second sub-feature extraction branch calculates the energy spectral density and the power spectral density corresponding to the electroencephalogram time series data.

[0022] The electroencephalogram time series data is extracted based on the energy spectral density and the power spectral density to output electroencephalogram modality features.

[0023] The first sub-feature fusion network fuses the physiological modality features and the electroencephalogram modality features to generate time series modality features.

[0024] In an optional embodiment, the physiological signal sequence is extracted based on a sparse attention scoring method to generate a physiological signal matrix, including:

[0025] The physiological signal sequence is linearly transformed to generate an initial query matrix, an initial key matrix, and an initial value matrix.

[0026] The similarity scores between each query vector in the initial query matrix and all key vectors in the initial key matrix are calculated.

[0027] The physiological signal-specific prior is introduced, and the sparsity measure values corresponding to each query vector in the initial query matrix are calculated based on the similarity scores.

[0028] According to the sparsity measure values, each query vector is sorted, and the top-u target query vectors with the highest sparsity measure values are selected.

[0029] The sparse query matrix is composed based on the top-u target query vectors.

[0030] The physiological signal matrix is generated based on the sparse query matrix, the initial key matrix, and the initial value matrix.

[0031] In an optional implementation, the electroencephalogram time series data is subjected to feature extraction based on energy spectral density and power spectral density, and electroencephalogram modality features are output, including:

[0032] Each frequency point in the electroencephalogram time series data is mapped to a first initial frequency band, a second initial frequency band, and a third initial frequency band;

[0033] Based on the energy spectral density and the power spectral density, the energy mean and the power mean corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band are calculated respectively;

[0034] Based on the energy mean and the power mean corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band respectively, weight information corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band respectively is calculated;

[0035] The first initial frequency band, the second initial frequency band, and the third initial frequency band are multiplied by the corresponding weight information respectively to generate a first target frequency band, a second target frequency band, and a third target frequency band;

[0036] The phase information corresponding to the electroencephalogram time series data is extracted by Hilbert transform to generate a phase matrix;

[0037] Based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, the information of the time dimension, the frequency dimension, and the phase dimension corresponding to the electroencephalogram time series data is integrated to construct a time-frequency-phase three-dimensional feature, and electroencephalogram modality features are generated.

[0038] In an optional implementation, the second feature extraction network includes a third sub-feature extraction branch, a fourth sub-feature extraction branch, and a second sub-feature fusion network. The face image data and the eye movement image data are input into the second feature extraction network to generate visual modality features, including:

[0039] The face image data is input into the third sub-feature extraction branch, and the eye movement image data is input into the fourth sub-feature extraction branch;

[0040] The dynamic feature extraction branch in the third sub-feature extraction branch performs optical flow feature extraction on the face image data to output face optical flow features corresponding to the face image data;

[0041] The static feature extraction branch in the third sub-feature extraction branch performs static feature extraction on the face image data to output face static features corresponding to the face image data;

[0042] The face optical flow features and the face static features are fused to generate face fusion features;

[0043] The dynamic feature extraction branch in the fourth sub-feature extraction branch performs optical flow feature extraction on the eye-tracking image data and outputs the eye-tracking optical flow features corresponding to the eye-tracking image data;

[0044] The static feature extraction branch in the fourth sub-feature extraction branch performs static feature extraction on the eye-tracking image data and outputs the static eye-tracking features corresponding to the eye-tracking image data;

[0045] Eye-movement optical flow features and eye-movement static features are integrated to generate eye-movement fusion features;

[0046] Facial fusion features and eye-tracking fusion features are fused together to generate visual modal features.

[0047] In one optional implementation, the facial image data consists of at least three frames. The third sub-feature extraction branch extracts optical flow features from the facial image data and outputs the facial optical flow features corresponding to the facial image data, including:

[0048] Target detection is performed on each frame of facial image data to determine the region of interest in each frame of facial image data; the region of interest includes the orbicularis oculi muscle, the orbicularis oris muscle, and the frontalis muscle.

[0049] Based on the preset streamer algorithm, the motion vector corresponding to each facial image data is calculated according to the position information of each region of interest in the corresponding facial image data; the motion vector includes horizontal displacement and vertical displacement.

[0050] Calculate the motion amplitude, direction of motion, and acceleration corresponding to the motion vector;

[0051] The motion vector, motion amplitude, motion direction, and motion acceleration are fused to generate facial optical flow features.

[0052] In one optional implementation, the static feature extraction branch includes multiple parallel branches; the static feature extraction branch in the third sub-feature extraction branch performs static feature extraction on the facial image data and outputs the facial static features corresponding to the facial image data, including:

[0053] The static feature extraction branch extracts features from the facial image data to obtain initial static features;

[0054] Input the initial static features into each of the first parallel branches;

[0055] The first parallel branch weights the initial static features based on two dimensions: channel and space, to obtain weighted static features;

[0056] Based on a preset number of multi-size local feature extraction modules, local feature extraction is performed on weighted static features to obtain local static features at various scales.

[0057] The local static features are spliced to generate multi-scale local features;

[0058] The global feature contraction attention module is used for contracting the multi-scale local features to generate target local features;

[0059] The target local features are input into the next parallel branch, and the process is repeated until the last parallel branch is processed;

[0060] The results output by each parallel branch are globally pooled, and the face static features are output based on the fully connected layer. In an optional implementation, the time-series modal features and the visual modal features are fused to generate target fusion features, including:

[0061] The visual modal features are used as the first query matrix, and the time-series modal features are used as the first key matrix and the first value matrix;

[0062] The first dependency weight of the visual modal features on the time-series modal features is calculated;

[0063] The time-series modal features are used as the second query matrix, and the visual modal features are used as the second key matrix and the second value matrix;

[0064] The second dependency weight of the time-series modal features on the visual modal features is calculated;

[0065] The first dependency weight is multiplied by the time-series modal features to obtain target time-series features;

[0066] The second dependency weight is multiplied by the visual modal features to obtain target visual features;

[0067] The target time-series features and the target visual features are combined to generate target fusion features.

[0068] In a second aspect, the present application provides a pilot workload recognition device, which comprises:

[0069] An acquisition module is configured to acquire physiological time-series data, electroencephalogram time-series data, eye movement image data, and face image data corresponding to a target pilot; the physiological time-series data includes at least one of electrocardiogram data, electromyogram data, and respiration data;

[0070] A first output module is configured to input the physiological time-series data, the electroencephalogram time-series data, the eye movement image data, and the face image data into a preset workload recognition model, and output a current workload level corresponding to the target pilot;

[0071] A second output module is configured to output a warning information based on the current workload level.

[0072] The pilot workload identification device provided by the embodiment of the application obtains physiological time sequence data, electroencephalogram time sequence data, eye movement image data and facial image data corresponding to a target pilot; inputs the physiological time sequence data, the electroencephalogram time sequence data, the eye movement image data and the facial image data into a preset workload identification model, and outputs a current workload level corresponding to the target pilot, thereby greatly improving the accuracy and stability of workload identification by comprehensively considering various data features and their mutual relationships. In addition, the workload identification can adapt to various complex scenarios. Whether in the key stages such as takeoff and landing, or in the face of adverse weather or unexpected situations, the model can accurately judge the workload of the pilot according to the input data, and provide a reliable basis for subsequent decision-making. Based on the current workload level, warning information is output. The pilot himself, the flight management personnel and the ground control personnel can timely understand the working state of the pilot. When the warning information shows that the pilot is in a high workload state, the pilot can adjust his own work rhythm and attention allocation to avoid operation errors caused by excessive fatigue or information overload; the flight management personnel can reasonably arrange tasks according to the situation, and provide support or adjust the flight plan if necessary; the ground control personnel can pay more attention and assistance to ensure the safety of the flight process. Through timely warning, the risk of flight accidents caused by excessive workload of the pilot can be effectively reduced, and the flight safety can be ensured. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0074] Figure 1 is a flowchart of a pilot workload identification method according to an embodiment of the present application;

[0075] Figure 2 is a schematic diagram of a data acquisition system according to an embodiment of the present application;

[0076] Figure 3 is a flowchart of another pilot workload identification method according to an embodiment of the present application;

[0077] Figure 4 is a processing flowchart of a preset workload identification model according to an embodiment of the present application;

[0078] Figure 5 is a structural block diagram of a pilot workload identification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0080] It should be noted that the pilot workload identification method provided in the embodiments of the present application can be executed by a pilot workload identification device. The pilot workload identification device can be implemented by software, hardware or a combination of software and hardware to become part or all of an electronic device. The electronic device can be a controller in an airplane or a controller in a flight simulator. The following method embodiments are described by taking the electronic device as an example.

[0081] According to the embodiments of the present application, a pilot workload identification method is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions. Although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0082] In the present embodiment, a pilot workload identification method is provided, which can be used in the electronic device described above, Figure 1 is a flowchart of the pilot workload identification method according to the embodiments of the present application, as Figure 1 shown, the flow includes the following steps:

[0083] In step S101, physiological time series data, electroencephalogram time series data, eye movement image data and facial image data corresponding to a target pilot are acquired.

[0084] The physiological time series data includes at least one of electrocardiogram data, electromyogram data and respiration data.

[0085] Specifically, the electronic device can receive physiological time series data, electroencephalogram time series data, eye movement image data and facial image data corresponding to a target pilot sent by a data acquisition system to the electronic device.

[0086] The data acquisition system can include a wearable ergonomics physiological recorder, a water electrode electroencephalogram system, a wearable eye tracker, and a camera device. The wearable ergonomics physiological recorder can be an ErgoLAB intelligent wearable ergonomics physiological recorder, which is a set of wearable multi-parameter vital sign comprehensive detectors that can be worn on the human body, can monitor the physiological indicators of the human body in real time, such as RESP breathing frequency, HR heartbeat speed, ECG heart change, EDA skin change, PPG pulse change, and EMG muscle electricity, and can also extract the posture change, GPS space-time behavior trajectory, and spatial position data of the human body in real time. Thus, the electronic device can acquire the physiological time series data of the target pilot based on the ErgoLAB intelligent wearable ergonomics physiological recorder.

[0087] The water electrode electroencephalogram system can be a Semi-Dry EEG water electrode electroencephalogram system. The Semi-Dry EEG water electrode electroencephalogram system is composed of 32 electroencephalogram electrodes, a reference electrode, a wireless electroencephalogram signal amplifier, an electroencephalogram cap, and an electrode cotton sheet. The sampling rate is as high as 32 kHz, the resolution is 24 bit, and the signal programmable amplification is supported by 10-1000 times. Thus, the electronic device can acquire the electroencephalogram time series data based on the Semi-Dry EEG water electrode electroencephalogram system.

[0088] The wearable eye tracker can be a Tobii Glasses2 wearable eye tracker, which is a wearable eye tracker with wireless real-time observation function, specially designed for research in real-world environment. The sampling rate of the eye tracker is 50 Hz or 100 Hz. Thus, the electronic device can acquire the eye movement image data of the target pilot based on the Tobii Glasses2 wearable eye tracker.

[0089] The electronic device can acquire the face image data of the target pilot by the camera device.

[0090] As shown in the example, Figure 2 FIG. 1 shows a schematic diagram of a data acquisition system.

[0091] In step S102, the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the face image data are input into a preset workload recognition model, and the current workload level corresponding to the target pilot is output.

[0092] Specifically, the electronic device can input the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the face image data into the preset workload recognition model. The preset workload recognition model respectively extracts features from the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the face image data, and outputs the current workload level corresponding to the target pilot based on the extracted features.

[0093] Step S103, outputting a warning information based on the current workload level.

[0094] Specifically, the electronic device can compare the current workload level with a preset workload level threshold, and if the current workload level is greater than or equal to the preset workload level threshold, output the warning information.

[0095] The pilot workload identification method provided by the embodiment of the present application acquires physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data corresponding to a target pilot, inputs the physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data into a preset workload identification model, and outputs a current workload level corresponding to the target pilot. The pilot workload identification method provided by the embodiment of the present application comprehensively considers various data features and their mutual relationships, greatly improves the accuracy and stability of workload identification, and is suitable for workload identification in various complex scenarios. Whether in a critical stage such as takeoff or landing, or in the face of adverse weather or unexpected situations, the model can accurately judge the workload of the pilot according to the input data, and provide a reliable basis for subsequent decision-making. Based on the current workload level, a warning information is output. The pilot himself, the flight management personnel, and the ground control personnel can timely understand the working state of the pilot. When the warning information shows that the pilot is in a high workload state, the pilot can adjust his own work rhythm and attention allocation to avoid operation errors caused by excessive fatigue or information overload; the flight management personnel can reasonably arrange tasks according to the situation and provide support or adjust the flight plan if necessary; the ground control personnel can pay more attention and assistance to ensure the safety of the flight process. Through timely warning, the risk of flight accidents caused by excessive workload of the pilot can be effectively reduced, and flight safety can be ensured.

[0096] In the embodiment, a pilot workload identification method is provided, which can be used in the electronic device described above, Figure 2 is a flowchart of the pilot workload identification method according to the embodiment of the present application, as Figure 2 shown, the flowchart includes the following steps:

[0097] Step S201, acquiring physiological time series data, electroencephalogram time series data, eye movement image data, and facial image data corresponding to a target pilot.

[0098] The physiological time series data includes at least one of electrocardiogram data, electromyogram data, and respiration data.

[0099] For this step, please refer to the description of step S101 above, which will not be repeated here.

[0100] In step S202, the physiological time series data, the electroencephalogram time series data, the eye movement image data, and the facial image data are input into the preset workload recognition model, and a current workload level corresponding to the target pilot is output.

[0101] Specifically, the preset workload recognition model includes a first feature recognition network and a second feature recognition network, and the step S202 can include the following steps.

[0102] In step S2021, the physiological time series data and the electroencephalogram time series data are input into the first feature extraction network to generate time series modal features.

[0103] Specifically, as shown in Figure 4 The first feature extraction network includes a first sub-feature extraction branch, a second sub-feature extraction branch, and a first sub-feature fusion network, and the step S2021 can include the following steps.

[0104] In step a1, the physiological time series data is input into the first sub-feature extraction branch, and the electroencephalogram time series data is input into the second sub-feature extraction branch.

[0105] Specifically, the electronic device can input the physiological time series data into the first sub-feature extraction branch, and input the electroencephalogram time series data into the second sub-feature extraction branch.

[0106] In step a2, the first sub-feature extraction branch extracts features from the physiological time series data to generate a physiological signal sequence.

[0107] The physiological signal sequence includes at least one of respiratory data features, electrocardiogram data time domain features, electrocardiogram data frequency domain features, electrocardiogram data nonlinear features, and electromyogram data features.

[0108] Specifically, before feature extraction is performed on the physiological time series data, data preprocessing can be performed on the physiological time series data first.

[0109] For example, the electrocardiogram data and the respiratory data preprocessing steps include low-pass filtering, eliminating power frequency interference, and eliminating baseline drift. The low-pass filtering mainly filters out high-frequency data interference such as muscle electrical signals, then a 50Hz notch filter is used to eliminate power frequency interference, and finally the baseline drift is eliminated. The skin electricity data preprocessing only includes low-pass filtering and eliminating power frequency interference.

[0110] Then, the first sub-feature extraction branch extracts features from the physiological time series data, and the specific extraction process can be as follows:

[0111] The electrocardiogram data and the respiration data are periodic data. The electrocardiogram data preprocessing further includes determination of R-wave peaks, and the time for each heartbeat can be determined through adjacent R-wave peaks, and the heart rate can be determined by taking the inverse of the heartbeat time. The R-wave peak detection method is a threshold method, and the principle of the threshold method is that, first, the first sub-feature extraction branch determines all peak values of the electrocardiogram data based on a peak function, and then peak values greater than a certain threshold value are determined as R-wave peak values. After the R-wave peak values are determined, the heartbeat duration can be calculated, and then the instantaneous heart rate is equal to the inverse of the heartbeat duration, and the average heart rate is equal to the average of all instantaneous heart rates.

[0112] The respiration data preprocessing further includes determination of respiration peak values and respiration valley values. The respiration peak value refers to the turning point at which the end of inhalation starts exhalation, and the respiration valley value refers to the turning point at which the end of exhalation starts inhalation, and the determination of the respiration peak value and the respiration valley value is based on the determination of the exhalation amplitude and the inhalation amplitude. The principle of the respiration peak value and the respiration valley value determination method is that, first, the first sub-feature extraction branch determines all peak values and valley values of the respiration data by using a peak function. Then, the exhalation amplitude and the inhalation amplitude are obtained, and it is judged whether each exhalation amplitude and inhalation amplitude is less than a threshold value, and the corresponding peak value and valley value should be deleted for the exhalation amplitude and inhalation amplitude less than the threshold value. Finally, the respiration peak value and the valley value are obtained.

[0113] The electrocardiogram data feature extraction process calculates the time domain features, the frequency domain features and the nonlinear features of the electrocardiogram data according to the RR interval data. The time domain features include SDNN, RMSSD, SDSD, pNN50, pNN20, the frequency domain features include TP, ULF, VLF, LF, HF, pLF, pHF, LF / HF, and the nonlinear features include the SD1 value and the SD2 value of the Poincare scatter plot of the RR interval data, and the features are shown in Table 1.

[0114] Table 1: Electrocardiogram data features and their meanings

[0115]

[0116]

[0117] The electromyogram data feature extraction process, in order to fully extract the electromyogram data features, the diff function in the first sub-feature extraction branch obtains the first-order difference and the second-order difference of the EDA data, and finally the data mean, the median, the standard deviation, the minimum value and the maximum value of the EDA data, the first-order difference of the EDA data and the second-order difference of the EDA data are calculated, and there are 15 kinds of features, as shown in Table 2.

[0118] Table 2: Electromyogram data features

[0119]

[0120] The respiratory data feature extraction process, in order to fully extract the skin electricity data features, the first sub-feature extraction branch first acquires the respiratory peak value, the respiratory valley value, the expiration amplitude, the expiration time, the inspiration amplitude, the inspiration time, the respiratory time (determined by the respiratory peak interval), and the respiratory time (determined by the respiratory valley interval). Then, the diff function in the first sub-feature extraction branch acquires the first-order difference and the second-order difference of the above respiratory data, and finally the data mean, the median, the standard deviation, the minimum value and the maximum value of the respiratory data, the respiratory data first-order difference and the respiratory second-order difference are calculated, respectively, and a total of 120 features are obtained.

[0121] Table 3 Respiratory data features

[0122]

[0123]

[0124] The statistical analysis process first uses the S-W test (Shapiro-Wilk test) for normality test. If the normality test result h is "0", it means that the data conforms to the normal distribution, and if the normality test result h is "1", it means that the data does not conform to the normal distribution. For electrocardiogram data, if both groups of data belong to the normal distribution, the independent sample t test method can be used to further judge whether there is a difference between the two groups of data; for electromyography data and respiratory data, if both groups of data belong to the normal distribution, the paired sample t test or independent sample t test method can be used to further judge whether there is a difference between the two groups of data; if one of the two groups of data does not conform to the normal distribution or neither of the two groups of data conforms to the normal distribution, the independent sample t test method or the paired t test method cannot be used, and the rank sum test method needs to be used to judge whether there is a difference between the two groups of data features.

[0125] Finally, the first sub-feature extraction branch combines the obtained various features to generate a physiological signal sequence.

[0126] Step a3, based on the sparse attention score method, feature extraction is performed on the physiological signal sequence to generate a physiological signal matrix.

[0127] Specifically, the above step a3 can include the following steps:

[0128] Step a31, linearly changing the physiological signal sequence to generate an initial query matrix, an initial key matrix and an initial value matrix.

[0129] Specifically, the first sub-feature extraction branch can convert the physiological signal sequence into a vector representation that can be processed by the model. Assuming that the length of the physiological signal sequence is T, and the physiological signal feature dimension of each time step is d, the physiological signal sequence can be represented as a matrix X ∈ R T×d. Then, the first sub-feature extraction branch can use linear transformation to map the input matrix X into an initial query matrix Q, an initial key matrix K and an initial value matrix V, respectively. Specifically, linear transformation is performed by three learnable weight matrices Wq∈R d×dk , Wk∈R d×dk and Wv∈R d×dv , and the following results are obtained:

[0130] Q=XW q ∈R T×dk ;

[0131] K=XW k ∈R T×dk ;

[0132] V=XW v ∈R T×dv .

[0133] where dkand dvare the dimensions of the initial query matrix, the initial key matrix and the initial value matrix, respectively.

[0134] Step a32, the similarity score between each query vector in the initial query matrix and all key vectors in the initial key matrix is calculated.

[0135] Specifically, the first sub-feature extraction branch can calculate the similarity score S(q i ,k j ) between each query vector q i in the initial query matrix Q and all key vectors k j in the initial key matrix K. Dot product similarity is usually used, i.e. S(q i ,k j )=q i T k j .

[0136] Then, the similarity score is subjected to softmax operation to obtain the standard attention probability distribution. Assuming the formula is as follows:

[0137] where, is used to scale the dot product similarity to avoid the problem of gradient vanishing or explosion.

[0138] Step a33, physiological signal-specific prior is introduced, and the sparsity measure value corresponding to each query vector in the initial query matrix is calculated based on the similarity score.

[0139] Specifically, for the i-th query vector q i in the initial query matrix Q, its sparsity measure value M(q i ) is calculated according to the formula. Assuming the formula is as follows:

[0140]

[0141] Where, log∑ j exp(S(q i k j )) refers to the query vector q i With all key vectors k j Similarity score S(q) between i ,k j The logarithmic form of the softmax operation, which measures the query vector q. i Global correlation with the entire sequence. It is to calculate the query vector q i With all key vectors k j The average similarity score represents the query vector q. i The average correlation with the sequence. λ·HRV_mask(q) i ), where HRV_mask(qi) is the attention enhancement mask for the heart rate variability related channels. If the query vector q i Related to heart rate variability, the mask value is a positive number, and vice versa. λ is an adaptive weighting coefficient used to adjust the influence of this part.

[0142] Step a34: Sort the query vectors according to the sparsity metric and select the Top-u target query vectors with the highest sparsity metric.

[0143] Specifically, the first sub-feature extraction branch sorts the query vectors according to the calculated sparsity metric values ​​corresponding to each query vector, and selects the Top-u target query vectors with the highest sparsity metric values. Where u = c·log(T) (c is an adjustable constant).

[0144] Step a35: Form a sparse query matrix based on the Top-u target query vectors.

[0145] Specifically, the first sub-feature extraction branch is based on a sparse query matrix composed of the Top-u target query vectors.

[0146] Step a36: Generate the physiological signal matrix based on the sparse query matrix, the initial key matrix, and the initial value matrix.

[0147] Specifically, the electronic device uses the filtered sparse query matrix to calculate the modified attention. The specific formula is shown below:

[0148] in, This is a sparse query matrix. In this way, only the sparse query matrix is ​​calculated. The attention is between the initial key matrix K, thereby reducing the computational complexity from O(T 2 ) to O(uT).

[0149] Then, based on the modified attention, a physiological signal matrix is generated.

[0150] Step a4, feature extraction is performed on the physiological signal matrix based on multi-period convolution kernels, and physiological modality features are output.

[0151] Specifically, assuming that the physiological signal matrix is X, its dimension is [N, T], where N represents the number of channels of the signal, and T represents the number of sampling points, i.e., the length of the time series.

[0152] According to the characteristics of the physiological signal and the analysis purpose, a set of convolution kernels with different periods are designed. The size, shape and weight of these convolution kernels are usually obtained through training learning. Assuming that there are K convolution kernels in total, the dimension of each convolution kernel is [N, L k ], where L k represents the length of the kth convolution kernel, corresponding to different time periods. For example, a shorter convolution kernel can be used to capture high-frequency details in the physiological signal, while a longer convolution kernel can be used to extract low-frequency trend and periodic features.

[0153] Each convolution kernel is respectively convolved with the physiological signal matrix X. For the kth convolution kernel, the calculation method of the convolution result Yk of X is as follows: Y k =X*W k +b k . Where * represents convolution operation, W k is the weight matrix of the kth convolution kernel, and b k is the bias term. The result of convolution operation Y k is a feature matrix with a dimension of [N, T-L k +1], which represents the feature response of the physiological signal after being filtered by the kth convolution kernel.

[0154] The output feature matrices Y1, Y2, …, Y K of all convolution kernels are fused. Common fusion methods include concatenation, summation, etc. For example, through concatenation operation, all feature matrices can be spliced in the channel dimension to obtain a fused feature matrix Y with a dimension of [N, (T-L1+1)+(T-L2+1)+…+(T-L K +1)], i.e., physiological modality features.

[0155] To better extract physiological modal features, some subsequent processing can also be performed on the basis of the fused feature matrix Y, such as a pooling operation, application of a nonlinear activation function, etc. The pooling operation can reduce the dimensionality of the features while retaining important feature information; the nonlinear activation function can increase the nonlinear expression capability of the model, making the extracted features more representative.

[0156] After the above steps, the final feature matrix obtained is the physiological modal feature. It contains local and global features of the physiological signal at different time scales, and can reflect the inherent patterns and rules of the physiological signal.

[0157] In step a5, the second sub-feature extraction branch calculates the energy spectral density and the power spectral density corresponding to the electroencephalogram time series data.

[0158] Specifically, the electronic device can select the power spectral density (PSD) and the energy spectral density (ESD) corresponding to the electroencephalogram time series data as the electroencephalogram analysis index. For the analysis of dynamic signals, since there are many frequency bands in the electroencephalogram time series data, it is difficult to effectively divide in the time domain, so the frequency domain analysis is more commonly used, and the method of analyzing the frequency spectrum is usually adopted. The method extracts 5 frequency bands: [delta (1-4 Hz), theta (4-8 Hz), alpha (8-14 Hz), beta (14-30 Hz), and gamma (30-75 Hz)]. By arranging the harmonics decomposed from the original signal in order of energy from high to low, an energy spectrum is formed, which represents the energy distribution of different frequency components in the signal.

[0159] Specifically, energy spectral density analysis is a method of electroencephalogram frequency domain analysis. In the process of analyzing dynamic signals, since the electroencephalogram time series data itself often contains multiple frequency bands, it is very difficult to divide these frequency bands in the time domain, so frequency domain analysis has become a commonly used means. If the harmonics decomposed from the original signal are arranged in order of energy from high to low, an energy spectrum is formed, which represents the energy distribution of different frequency components in the signal, and the formula is as follows:

[0160]

[0161] where φ(w) is the energy spectral density, f n is each frequency component in the signal, F(ω) is the Fourier transform of the signal, and F*(ω) is the complex conjugate of F(ω).

[0162] PSD (Power Spectral Density) is a measure that calculates the average power distribution proportion of random variables per unit frequency, i.e. a form of mean square value. Its advantage is that it can convert the amplitude of the original time-varying EEG time series signal into the power spectrum of the frequency-varying EEG time series signal, so as to visualize the distribution and conversion of the EEG time series rhythm. The second sub-feature extraction branch calculates the power spectral density of the EEG signal based on the pwelch function, which is based on the Welch method and estimates the PSD of the signal by the segmented average periodogram method. This method selects the pwelch function and uses a 512-sample Hamming window, a 256-sample overlap, and a fast Fourier transform (FFT) of 1024 data points to estimate the frequency spectrum. Finally, the power spectrum is obtained by the modulus square of the frequency spectrum, as shown in the formula:

[0163]

[0164] where P x (f) is the power spectral density, N is the number of samples, X N (f) is the Fourier transform result after windowing. E(h) is the energy in the frequency interval, f h is the upper limit of the frequency interval, f i is the lower limit of the frequency interval.

[0165] Step a6, based on the energy spectral density and the power spectral density, the EEG time series data is extracted, and the EEG modal features are output.

[0166] Specifically, the above step a6 can include the following steps:

[0167] Step a61, map each frequency point in the EEG time series data to the first initial frequency band, the second initial frequency band and the third initial frequency band.

[0168] Specifically, the electronic device can map each frequency point in the EEG time series data to the first initial frequency band, the second initial frequency band and the third initial frequency band. For example, the first initial frequency band is alpha (8-12 Hz), the second initial frequency band is beta (13-30 Hz), and the third initial frequency band is gamma (30-100 Hz).

[0169] Step a62, based on the energy spectral density and the power spectral density, the energy mean and the power mean corresponding to the first initial frequency band, the second initial frequency band and the third initial frequency band are calculated respectively.

[0170] Specifically, based on the energy spectral density and the power spectral density, the energy mean and the power mean corresponding to the first initial frequency band, the second initial frequency band and the third initial frequency band are calculated respectively, to obtain the frequency domain feature X eeg ∈R T×B×Deeg (B = 3 is the number of frequency bands).

[0171] For example, taking the power spectral density as an example, for the alpha frequency band, the power mean value F α is the number of frequency points in the alpha frequency band. These frequency band features can reflect the intensity of brain electrical activity in different frequency ranges, and in cognitive load research, the power changes of different frequency bands are closely related to the cognitive state, such as the increase of beta wave power, which may be related to high cognitive load.

[0172] Step a63, based on the energy mean value and the power mean value corresponding to the first initial frequency band, the second initial frequency band and the third initial frequency band respectively, the weight information corresponding to the first initial frequency band, the second initial frequency band and the third initial frequency band is calculated respectively.

[0173] Specifically, based on the full connection layer, the energy mean value and the power mean value corresponding to the first initial frequency band, the second initial frequency band and the third initial frequency band are processed to obtain the weight information corresponding to the first initial frequency band, the second initial frequency band and the third initial frequency band respectively. f b , S'(q, k b ) = S(q, k b ) · a f b .

[0174] For example, the electronic device can convert the energy mean value or the power mean value of each frequency band into weight through the softmax function. Assuming that the energy mean value is used to calculate the weight, first calculate S = E1 + E2 + E3, then the weight of the first initial frequency band w1 = E1 / S, the weight of the second initial frequency band w2 = E2 / S, and the weight of the third initial frequency band w3 = E3 / S, and w1 + w2 + w3 = 1.

[0175] For example, if the power of the beta frequency band continues to rise in a certain period of time, after processing by the full connection layer and the Softmax function, the weight of the beta frequency band a f β will increase, so that the model pays more attention to the information of the beta frequency band when calculating self-attention, thereby focusing on the frequency band activated during high load. The time-frequency cross-attention uses a two-dimensional attention matrix to combine the distribution of energy spectral density or power spectral density at different time steps and frequency bands to capture the dominant frequency band at a specific time point. For example, during the peak period of the task, the energy or power of the gamma frequency band increases explosively, and the time-frequency cross-attention can more accurately capture such changes.

[0176] Step a64, multiplying the first initial frequency band, the second initial frequency band and the third initial frequency band by the corresponding weight information respectively to generate the first target frequency band, the second target frequency band and the third target frequency band.

[0177] Specifically, the first initial frequency band, the second initial frequency band, and the third initial frequency band are multiplied by corresponding weight information respectively to generate a first target frequency band, a second target frequency band, and a third target frequency band.

[0178] In step a65, phase information corresponding to the electroencephalogram time series data is extracted by Hilbert transform to generate a phase matrix.

[0179] Specifically, the electronic device convolves the Hilbert kernel with the electroencephalogram time series data x(t) to obtain the Hilbert transformed signal y(t). From the frequency domain perspective, the Hilbert transform shifts the positive frequency component of the signal right by 90 and the negative frequency component left by 90. Based on this, the original electroencephalogram time series data x(t) and its Hilbert transformed signal y(t) can constitute an analytic signal z(t) = x(t) + jy(t). Through the analytic signal, the phase information can be calculated In actual operation, the Hilbert transform is applied to the electroencephalogram time series data after power frequency filtering and short-time Fourier transform, and the phase information of each frequency point at different time points can be obtained, providing data in the phase dimension for constructing a three-dimensional feature.

[0180] In step a66, based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, the information in the time dimension, the frequency dimension, and the phase dimension corresponding to the electroencephalogram time series data is integrated to construct a time-frequency-phase three-dimensional feature and generate an electroencephalogram modality feature.

[0181] Specifically, based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, the information in the time dimension, the frequency dimension, and the phase dimension corresponding to the electroencephalogram time series data is integrated to construct a time-frequency-phase three-dimensional feature.

[0182] Assuming that the time step is T, the frequency dimension has three target frequency bands, i.e., the first target frequency band, the second target frequency band, and the third target frequency band, and the dimension of the phase matrix is TxF (F is the number of frequency points). A three-dimensional tensor Feature ∈ R T ×3×2+T×F where the first dimension represents time, the first three elements of the second dimension correspond to the energy spectral density and power spectral density information of the three target frequency bands, and the remaining elements correspond to the elements of the phase matrix. The specific representation is as follows:

[0183]

[0184] where E targeti (t) is the corresponding energy spectral density, P targeti (t) is the corresponding power spectral density, and Φ(t, j) is the phase matrix.

[0185] Step a7, the first sub-feature fusion network fuses the physiological modality features and the electroencephalogram modality features to generate the time-series modality features.

[0186] Specifically, the first sub-feature fusion network splices or performs weighted fusion on the physiological modality features and the electroencephalogram modality features to generate the time-series modality features.

[0187] Step S2022, input the facial image data and the eye movement image data into the second feature extraction network to generate the visual modality features.

[0188] Specifically, the second feature extraction network includes a third sub-feature extraction branch, a fourth sub-feature extraction branch, and a second sub-feature fusion network, and the step S2022 can include the following steps.

[0189] Step b1, input the facial image data into the third sub-feature extraction branch and input the eye movement image data into the fourth sub-feature extraction branch.

[0190] Specifically, the electronic device inputs the facial image data into the third sub-feature extraction branch and inputs the eye movement image data into the fourth sub-feature extraction branch.

[0191] Step b2, the dynamic feature extraction branch in the third sub-feature extraction branch performs optical flow feature extraction on the facial image data to output facial optical flow features corresponding to the facial image data.

[0192] Specifically, the facial image data is at least three frames, and the step b2 can include the following steps.

[0193] Step b21, perform target detection on each frame of facial image data to determine a region of interest in each frame of facial image data.

[0194] The region of interest includes orbicularis oculi, orbicularis oris, and frontalis.

[0195] Specifically, the third sub-feature extraction branch can perform target detection on each frame of facial image data based on a target detection algorithm to determine a region of interest in each frame of facial image data.

[0196] Step b22, based on a preset optical flow algorithm, calculate the motion vector corresponding to each frame of facial image data according to the position information of each region of interest in the corresponding facial image data.

[0197] The motion vector includes horizontal displacement and vertical displacement.

[0198] The preset optical flow algorithm can be a Lucas-Kanade (LK) optical flow algorithm, a Farneback optical flow algorithm, or other optical flow algorithms, and embodiments of the present application do not make specific limitations on the preset optical flow algorithm.

[0199] Specifically, the third sub-feature extraction branch can convert the continuous two frames of face images It, It+1∈R H×W×3 (RGB three channels) into single-channel images Gt, Gt+1∈R H×W , eliminating color interference. Then, a 3x3 Gaussian kernel is used for denoising to reduce the influence of high-frequency noise on optical flow calculation.

[0200] Next, the third sub-feature extraction branch can calculate the motion vectors corresponding to each face image data based on the position information of each region of interest in the corresponding face image data according to the Farneback optical flow algorithm.

[0201] Then, the motion vectors are normalized to [-1, 1] and aligned with the original frame timestamp to form an optical flow feature sequence: Flowt = [Ut, Vt] ∈ R H×W×2 .

[0202] Step b23, calculating the motion amplitude, motion direction, and motion acceleration corresponding to the motion vector.

[0203] Specifically, the third sub-feature extraction branch calculates the motion amplitude and the motion direction θ = arctan2(v, u) of the motion vector as additional feature channels (a total of 4 channels: U, V, M, θ), enriching the representation dimension of the motion feature.

[0204] In addition, the third sub-feature extraction branch calculates the second-order difference ΔFlow t = Flow t - Flow t-1 of the continuous three frames of motion vectors, capturing the motion acceleration corresponding to the motion vector, and enhancing the sensitivity to sudden expression changes (such as rapid opening and closing of eyelids when surprised).

[0205] Step b24, fusing the motion vector, motion amplitude, motion direction, and motion acceleration to generate a face optical flow feature.

[0206] Specifically, the motion vector, motion amplitude, motion direction, and motion acceleration are spliced to generate a face optical flow feature.

[0207] Step b3, the static feature extraction branch in the third sub-feature extraction branch performs static feature extraction on the face image data, and outputs the face static feature corresponding to the face image data.

[0208] Specifically, the static feature extraction branch includes multiple parallel branches; the step b3 can include the following steps.

[0209] Step b31, the static feature extraction branch extracts features from the face image data to obtain initial static features.

[0210] Specifically, the static feature extraction branch extracts features from the face image data based on the convolution kernels in the convolution layer to obtain the initial static features.

[0211] For example, the size of the convolution kernel in the convolution layer is 3x3, and the convolution kernel is slid on the face image data to perform convolution operation on each local region, that is, the corresponding elements are multiplied and summed to obtain a new feature value. Through a plurality of different convolution kernels, different features of the image such as edges and textures can be extracted to obtain the initial static features.

[0212] The initial feature extraction is performed on the input face image data to convert the original face image pixel information into more representative feature maps. This process can be regarded as a simple feature abstraction of the image, which prepares for subsequent multi-scale feature extraction.

[0213] Step b32, the initial static features are input to each first parallel branch.

[0214] Then, the initial static features are input to each first parallel branch.

[0215] Step b33, the first parallel branch weights the initial static features based on the channel and spatial dimensions to obtain weighted static features.

[0216] Specifically, the first parallel branch can compress the feature map of each channel into a numerical value through a global average pooling operation to obtain the global statistical information of the channel. Then, the statistical information is input into a multi-layer perception (MLP), and the MLP learns how to generate a suitable weight for each channel according to the statistical information. This weight can be regarded as a quantitative representation of the importance of the channel.

[0217] In addition, the first parallel branch can use convolution kernels to perform sliding convolution operation on the feature map to generate spatial weights by learning the parameters of the convolution kernels. These convolution kernels can capture the feature correlation in the local spatial region to generate a suitable weight value for each spatial position. In addition, some attention mechanisms such as spatial attention modules can also be combined to automatically learn the importance distribution of the spatial position.

[0218] Then, channel dimension weighting and spatial dimension weighting are respectively performed to obtain channel weights and spatial weights. Then, the two weights are fused, for example, they can be combined by simple multiplication or addition operation to obtain a comprehensive weight matrix. Finally, multiplying the weight matrix with the initial static feature, the weighted static feature can be obtained.

[0219] Step b34, local feature extraction is performed on the weighted static feature based on a preset number of multi-size local feature extraction modules to obtain local static features at various scales.

[0220] Specifically, multi-scale parallel convolution is used for each multi-size local feature extraction module, the receptive field is expanded to a 5x5 feature region by connecting two groups of 1x3 and 3x1 asymmetric convolution layers in series, and local feature extraction is performed on the weighted static feature to obtain local static features at various scales.

[0221] Step b35, each local static feature is spliced to generate multi-scale local features.

[0222] Specifically, the local static features at various scales are spliced to generate multi-scale local features.

[0223] For example, the extracted multi-scale initial feature M(x) can be represented by the formula:

[0224] M(x)=Concat{f (1×1) (x),f (3×3) (x),f (5×5) (x),f avg (x)}

[0225] Where f 1×1 (·), f 3×3 (·), f 5×5 (·) represent convolution operations of different receptive field sizes, and f avg (·) represents an average pooling layer for extracting features of different scales of the face. Concat{·,·,·,·} is an operation for connecting these multi-scale features along the channel dimension. In addition, a 1x1 convolution is used before and after M(x) operation to construct a bottleneck structure to reduce the calculation parameters of multi-scale feature extraction, and finally generate multi-scale local features x' as follows, i.e.: x'=f (1×1) (f (1×1) (M(x)))。

[0226] Step b36, the multi-scale local features are contracted based on a global feature contraction attention module to generate target local features.

[0227] Specifically, the global feature shrinkage attention module first takes the absolute value of the multi-scale local feature x', and then obtains a new feature map F through global pooling to summarize and compress the global information of the facial features gap Finally, the feature map is simplified into a one-dimensional vector as shown in the following formula:

[0228] Where |·| represents the absolute value operation, W and H represent the width and height of the feature map respectively, and F gap A feature map with C channels will be generated, and the value in each channel represents the average value of all pixels in the corresponding channel dimension feature map.

[0229] Then F gap After two 1x1 convolutions and sigmoid functions, a normalized scaling parameter a is obtained as shown below, and the value range of the parameter is (0, 1) so that the obtained threshold value is not too large and always positive, that is:

[0230] Where a is the scaling parameter, and z is the output of the two 1x1 convolution layers. In order to ensure that the threshold value of the shrinkage function is positive, and the threshold value is not too large so as to set a large number of features to zero, the threshold value t can be represented as: t = a x A(x'). In this way, different samples can obtain different threshold values, so that the learned high-level features can become more discriminative.

[0231] After obtaining the threshold value, the features with absolute values lower than the threshold value are removed from the multi-scale local feature x', and the features with absolute values greater than the threshold value are shrunk towards 0. The processed feature S(x', t) can be represented as follows:

[0232]

[0233] After shrinking, the current irrelevant features can be set to zero, thereby alleviating the redundancy problem brought by multi-scale information and suppressing noise irrelevant to the task of facial emotion recognition, while strengthening the relationship between local features and global features.

[0234] Finally, a cross-layer data path is used to connect the original feature before multi-scale feature extraction and the output of the global feature shrinkage attention module, helping the network to retain and reuse more rich global features, while preventing the problem of gradient disappearance and network degradation in deep neural networks. The method is shown in the following formula:

[0235] H(x) = x + S(M(x))

[0236] Wherein, H(·) is the expression feature map fused with the local and global features of the face, that is, the target local feature x is the original face feature map output by the last layer, M(·) represents the local feature extracted by the multi-scale local feature extraction module, and S(·) represents the face feature processed by the global feature contraction attention module.

[0237] Step b37, input the target local feature into the next parallel branch, and repeat the process until the last parallel branch is processed.

[0238] Specifically, the target local feature is input into the next parallel branch, and the process is repeated until the last parallel branch is processed. The processing process can be referred to in the foregoing, and will not be repeated here.

[0239] Step b38, globally pool the results output by each parallel branch, and output the face static feature based on the fully connected layer.

[0240] Specifically, the results output by each parallel branch are globally pooled, and the face static feature is output based on the fully connected layer.

[0241] Step b4, integrate the face optical flow feature and the face static feature to generate a face fusion feature.

[0242] Specifically, the electronic device can perform global average pooling on the motion amplitude M corresponding to the motion vector to obtain a motion saliency vector S move ∈R C , representing the motion intensity of each channel feature.

[0243] Generate channel weight w move ∈(0, 1) C through the fully connected layer, and perform weighted fusion on the face static feature and the face optical flow feature to generate a face fusion feature. The specific formula is as follows:

[0244] F final = wmove o Fstatic + (1-w move ) o Fflow.

[0245] Wherein, Fstatic is the face optical flow feature, and Fflow is the face static feature.

[0246] Step b5, the gaze point extraction branch in the fourth sub-feature extraction branch extracts features from the eye movement image data, and outputs the eye movement trajectory corresponding to the eye movement image data.

[0247] Specifically, the gaze point extraction branch identifies the eye movement image data to determine the eyeball key points in the eye movement image data. Then, the eyeball key points are tracked based on a tracking algorithm (such as a KCF algorithm, a MedianFlow algorithm, etc.). The tracking algorithm calculates the motion information of the eyeball according to the position changes of the eyeball key points between adjacent frames. During the tracking process, the position information of the eyeball key points at each time is recorded. Then, according to the recorded position information of the eyeball key points, the points are connected to obtain the motion trajectory of the eyeball.

[0248] In step b6, the feature of the eyeball motion trajectory is extracted to determine the gaze point distribution feature corresponding to the eye movement image data.

[0249] Specifically, the feature of the eyeball motion trajectory is extracted to extract basic statistical features, spatial distribution features, time sequence features, and motion features.

[0250] Specifically, the basic statistical features include: the number of gaze points: the total number of gaze points in the eye movement image is counted, reflecting the number of visual attention points of the subject during observation. The average gaze time: the average value of the duration of all gaze points is calculated, which can reflect the attention degree of the subject to different contents and the information processing speed. Gaze point density: the image is divided into several sub-regions, and the ratio of the number of gaze points in each sub-region to the area of the region is calculated to understand the distribution density of gaze points in different regions of the image.

[0251] For the spatial distribution feature, it includes: central tendency: the mean value of the gaze point coordinates is calculated to obtain the center position of the gaze point distribution, which can reflect the overall visual focus of the subject when observing the image. The degree of dispersion: the standard deviation or variance of the gaze point coordinates is calculated to measure the dispersion of the gaze points around the center position. The larger the standard deviation or variance, the more dispersed the gaze points, and the less concentrated the visual attention of the subject. Spatial entropy: used to describe the degree of disorder of the gaze point distribution. The higher the spatial entropy value, the more uniform and random the distribution of gaze points in space; the lower the entropy value, the more concentrated the gaze points in certain specific regions.

[0252] For the time sequence feature, it includes: the time interval between gaze points: the time interval between adjacent gaze points is calculated to analyze its distribution, which can understand the frequency and rhythm of the visual attention shift of the subject. Sequence correlation: by calculating the autocorrelation function or cross-correlation function, the correlation of gaze points in the time sequence is analyzed to determine whether there is periodic or trend change.

[0253] For motion features, including: motion speed: according to the distance and time interval between adjacent fixation points, the motion speed of the fixation points is calculated, and the speed change can reflect the visual search strategy and attention allocation of the subject during observation. Motion direction: determine the motion direction between each fixation point, and count the number of times or time proportion in different directions, which helps to analyze the visual scanning mode of the subject, such as whether there is a preferred scanning direction.

[0254] Then, principal component analysis is performed on the extracted multiple features to convert the high-dimensional feature space into a low-dimensional space, find out the principal components that have the greatest impact on the fixation point distribution characteristics, realize the dimension reduction and visualization of the features, and more intuitively display the main mode of the fixation point distribution. The extracted various features are combined into a feature vector to generate the fixation point distribution characteristics. The fixation point distribution characteristics corresponding to the eye movement image data are comprehensively described, which are used for subsequent classification, identification or correlation analysis with other data.

[0255] Step b7, the eye movement extraction branch in the fourth sub-feature extraction branch extracts features from the eye movement image data, and outputs the eye movement features corresponding to the eye movement image data.

[0256] Specifically, the eye movement extraction branch calculates the motion speed of the pupil in the horizontal and vertical directions through the change of the pupil position in the continuous eye movement image. The pupil motion speed can reflect the visual search speed and the speed of attention shift of the subject. For example, when quickly browsing an image, the pupil motion speed is usually faster.

[0257] The eye movement extraction branch calculates the rotation angle of the eyeball according to the relative position change of the pupil and the iris, and the motion information of the eye feature points. The rotation angle of the eyeball can help analyze the fixation direction and the change of the visual focus of the subject, which is of great significance to understand the visual cognitive process of the subject.

[0258] The eye movement extraction branch detects the state change of eye closure in the eye movement image, and counts the number of blinks per unit time, i.e. the blink frequency. The blink frequency can reflect the fatigue degree and the degree of concentration of the subject. Generally, when tired or not concentrating, the blink frequency will increase.

[0259] Then, the pupil motion speed, the rotation angle of the eyeball and the blink frequency are fused to form a comprehensive eye movement feature vector. A simple splicing method can be used to connect different types of feature vectors together; more complex fusion algorithms such as principal component analysis-based feature fusion, neural network-based feature fusion, etc. can be used to reduce the feature dimension, remove the correlation between features, and improve the representativeness and robustness of the features.

[0260] Step b8, the fixation point distribution characteristics and the eye movement characteristics are fused to generate eye movement fusion features.

[0261] Specifically, the fourth sub-feature extraction branch fuses the gaze point distribution feature and the eye movement feature for processing to generate an eye movement fusion feature. A simple splicing method can be used to connect the gaze point distribution feature and the eye movement feature together, or a more complex fusion algorithm such as principal component analysis-based feature fusion, neural network-based feature fusion, etc. can be used to reduce the feature dimension, remove the correlation between the features, and improve the representativeness and robustness of the features.

[0262] Step b9, fusing the face fusion feature and the eye movement fusion feature for processing to generate a visual modality feature.

[0263] Specifically, the face fusion feature and the eye movement fusion feature are fused for processing to generate a visual modality feature. A simple splicing method can be used to connect the face fusion feature and the eye movement fusion feature together, or a more complex fusion algorithm such as principal component analysis-based feature fusion, neural network-based feature fusion, etc. can be used to reduce the feature dimension, remove the correlation between the features, and improve the representativeness and robustness of the features.

[0264] Step S2023, fusing the time series modality feature and the visual modality feature for processing to generate a target fusion feature.

[0265] Specifically, the above step S2023 can include the following steps:

[0266] Step c1, taking the visual modality feature as a first query matrix, and taking the time series modality feature as a first key matrix and a first value matrix.

[0267] Specifically, the electronic device takes the visual modality feature as a first query matrix, and takes the time series modality feature as a first key matrix and a first value matrix.

[0268] Step c2, calculating a first dependency weight of the visual modality feature on the time series modality feature.

[0269] Specifically, the similarity between the visual modality feature and the time series modality feature is measured by calculating the dot product of the first query matrix Q and the first key matrix K. The specific calculation formula is: sim = Q·KT. Wherein, sim i,j represents the similarity score between the i-th visual modality feature and the j-th time series modality feature.

[0270] In order to avoid the instability of the gradient caused by the too large dot product result, the similarity score is usually scaled.

[0271] The scaling factor is That is:

[0272] Then, the scaled similarity scores are applied with a Softmax function to convert them into a probability distribution, thereby obtaining a first dependency weight matrix W. The calculation formula of the Softmax function is as follows:

[0273] Step c3, taking the time-series modality feature as a second query matrix and the visual modality feature as a second key matrix and a second value matrix.

[0274] Specifically, the time-series modality feature is taken as a second query matrix, and the visual modality feature is taken as a second key matrix and a second value matrix.

[0275] Step c4, calculating a second dependency weight of the time-series modality feature on the visual modality feature.

[0276] Specifically, referring to the calculation method of the first dependency weight calculated above, the second dependency weight is calculated, which will not be described here.

[0277] Step c5, multiplying the first dependency weight by the time-series modality feature to obtain a target time-series feature.

[0278] Specifically, the electronic device multiplies the first dependency weight by the time-series modality feature to obtain a target time-series feature.

[0279] Step c6, multiplying the second dependency weight by the visual modality feature to obtain a target visual feature.

[0280] Specifically, the electronic device multiplies the second dependency weight by the visual modality feature to obtain a target visual feature.

[0281] Step c7, merging the target time-series feature and the target visual feature to generate a target fusion feature.

[0282] Specifically, the electronic device merges the target time-series feature and the target visual feature to generate a target fusion feature.

[0283] Step S2024, outputting a current workload level corresponding to the target pilot based on the target fusion feature.

[0284] Specifically, the preset workload recognition model outputs a current workload level corresponding to the target pilot based on the target fusion feature.

[0285] Step S203, outputting a warning information based on the current workload level.

[0286] For this step, please refer to the description of step S103 above, which will not be described here.

[0287] The pilot workload identification method provided by the embodiments of the present application inputs the physiological time series data into a first sub-feature extraction branch and inputs the electroencephalogram time series data into a second sub-feature extraction branch. The first sub-feature extraction branch extracts features from the physiological time series data to generate a physiological signal sequence, thereby ensuring the accuracy of the generated physiological signal sequence. The physiological signal sequence is subjected to linear change to generate an initial query matrix, an initial key matrix and an initial value matrix. The original physiological signal sequence is mapped into a new feature space. In this way, the data can be preliminarily extracted and converted, thereby preparing for subsequent attention mechanism calculation and helping the model to better capture the potential patterns and features in the physiological signal. The similarity scores between each query vector in the initial query matrix and all key vectors in the initial key matrix are calculated, which can help the model to discover the potential relationship between the elements in the physiological signal sequence and thus determine which part is more important for feature extraction. The physiological signal specificity prior is introduced, and the sparsity measure value corresponding to each query vector in the initial query matrix is calculated based on the similarity score. In this way, the model can be more adapted to the characteristics of the physiological data and better mine valuable information related to the physiological signal. The sparsity measure value is calculated based on the similarity score, and thus the importance degree of each query vector can be determined according to the correlation degree between the signals, thereby providing a basis for subsequent screening. According to the sparsity measure value, the query vectors are sorted, and the top-u target query vectors with the highest sparsity measure value are selected. Thus, only the vectors with the most representative and important features can be retained, and the noise and redundant information can be removed, thereby reducing the data volume and improving the calculation efficiency and generalization ability of the model. At the same time, these key target query vectors can better reflect the core features of the physiological signal, which helps to improve the accuracy of subsequent feature extraction and model prediction. The sparse query matrix is formed based on the top-u target query vectors; and the physiological signal matrix is generated based on the sparse query matrix, the initial key matrix and the initial value matrix. The key information after screening and other related information in the original data can be comprehensively considered. The physiological signal matrix is subjected to feature extraction based on a multi-period convolution kernel to output physiological modal features. Through the synergistic effect of different period convolution kernels, the physiological signal features are comprehensively extracted from multiple scales, thereby enhancing the expression ability of the features and making the extracted physiological modal features more rich and accurate in describing the internal relationship between the physiological signal and the workload.

[0288] The second sub-feature extraction branch calculates the energy spectral density and the power spectral density corresponding to the electroencephalogram time series data. Thus, the electroencephalogram time series data can be analyzed in depth from the frequency domain, and the energy distribution in different frequency bands can be mined. The energy of the electroencephalogram signal in each frequency band will change under different cognitive states of the brain, and the extraction of the electroencephalogram modal features based on these indicators can accurately capture the changes in the electroencephalogram signal related to the workload, thereby providing an important basis for workload identification.

[0289] The frequency points in the electroencephalogram time series data are mapped to the first initial frequency band, the second initial frequency band, and the third initial frequency band, which can preliminarily divide the electroencephalogram signal into frequency bands, making the subsequent analysis more organized. Based on the energy spectral density and the power spectral density, the energy mean and the power mean corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band are calculated. The energy and power levels of each frequency band can be quantified. Energy and power are important characteristics of electroencephalogram signals, and their changes can reflect the intensity and activity of brain neural activity. By calculating the mean, a relatively stable energy and power index of each frequency band can be obtained, which provides an objective data basis for subsequent analysis and helps to judge the activity changes of each frequency band under different workloads. Based on the energy mean and the power mean corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band, the weight information corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band is calculated. The relative importance of different frequency bands in reflecting the workload can be highlighted. Different frequency bands may have different sensitivities to workload, and by assigning weights, the model can pay more attention to frequency bands with higher correlation with workload.

[0290] The first initial frequency band, the second initial frequency band, and the third initial frequency band are multiplied by the corresponding weight information to generate the first target frequency band, the second target frequency band, and the third target frequency band. After weighting, the first target frequency band, the second target frequency band, and the third target frequency band can more accurately reflect the contribution of different frequency bands in workload identification. This can enhance the frequency band features related to workload and suppress irrelevant or interfering frequency band information, making the subsequent constructed electroencephalogram modal features more targeted and effective, and helping to improve the model's ability to identify workload. The phase information corresponding to the electroencephalogram time series data is extracted by Hilbert transform to generate a phase matrix. Important phase information in the electroencephalogram signal can be obtained. Based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, the information of the time dimension, the frequency dimension, and the phase dimension corresponding to the electroencephalogram time series data is integrated to construct a time-frequency-phase three-dimensional feature and generate an electroencephalogram modal feature. The feature can comprehensively describe the characteristics of the electroencephalogram time series data.

[0291] The first sub-feature fusion network fuses the physiological modal feature and the electroencephalogram modal feature to generate a time series modal feature. Information complementation is achieved, and the fused time series modal feature integrates the information of the body and the brain, which can more comprehensively and accurately reflect the overall state of the pilot at work. For example, under high workload, physiological indicators may be abnormal and electroencephalogram signal frequency band energy may change, and the fusion feature can integrate these information to avoid the one-sidedness of single modal feature analysis.

[0292] The facial image data is input to the third sub-feature extraction branch, and the eye movement image data is input to the fourth sub-feature extraction branch. Thus, the feature extraction can be performed based on the differences in information representation and feature extraction methods of the facial image data and the eye movement image data. Target detection is performed on each frame of facial image data to determine the region of interest in each frame of facial image data, thereby ensuring the accuracy of the determined region of interest. Based on a preset optical flow algorithm, the motion vector corresponding to each facial image data is calculated according to the position information of each region of interest in the corresponding facial image data. Thus, the motion of the facial muscles between adjacent frames can be accurately captured, and the computational complexity is reduced. The motion amplitude, motion direction and motion acceleration corresponding to the motion vector are calculated. The feature description of the facial muscle movement is enriched. The motion amplitude reflects the intensity of the facial muscle movement, the motion direction indicates the direction of the muscle movement, and the motion acceleration can reflect the change speed of the muscle movement. These information can comprehensively describe the dynamic changes of facial expressions. The motion vector, motion amplitude, motion direction and motion acceleration are fused to generate facial optical flow features. A comprehensive facial motion feature representation can be obtained. The fused features contain information such as the position, intensity, direction and change speed of the facial muscle movement, and can more accurately reflect the dynamic changes of facial expressions.

[0293] The static feature extraction branch extracts features from the face image data to obtain initial static features. This step focuses on relatively stable information in the face image, such as the outline of the face, the shape and proportion of the features, etc. These static features are not affected by dynamic facial expression changes and can reflect the essential features of the pilot's face, providing basic data for subsequent feature processing and analysis. The initial static features are input into each first parallel branch, and the first parallel branch weights the initial static features based on the channel and spatial dimensions to obtain weighted static features. In the channel dimension, weighting can highlight the importance of different feature channels in expressing facial features, such as some channels that may correspond to key texture information of the face. By weighting, the expression of these information can be enhanced. In the spatial dimension, weighting can focus on the importance differences of different regions of the face, thereby enhancing the expression ability of the features. Based on a pre-set number of multi-size local feature extraction modules, local feature extraction is performed on the weighted static features to obtain local static features at various scales. Different scale local feature extraction modules can capture feature information of different size regions of the face, improving the richness and diversity of the features. Each local static feature is spliced to generate multi-scale local features; this can fully utilize the advantages of different scale features, avoid the limitations of single scale features, and enable subsequent feature analysis to consider the details and overall structure of the face. Based on the global feature contraction attention module, the multi-scale local features are contracted to generate target local features. The important parts of the multi-scale local features are automatically identified and enhanced, while the unimportant features are suppressed. This helps to highlight the key facial features related to the pilot's workload, reduces the interference of redundant information, and improves the quality and effectiveness of the features. The target local features are input into the next parallel branch, and the cycle continues until the last parallel branch is processed. The cyclic processing of multiple parallel branches can iteratively optimize the features. Each cycle can further excavate and enhance the expression ability of the facial features, and through continuous weighting, extracting local features, and contracting attention, the quality of the features and their representation ability for the pilot's workload are gradually improved. The results output by each parallel branch are globally pooled, and the face static features are output based on the fully connected layer. The global pooling operation can reduce the dimensionality of the high-dimensional feature vector, reducing the number of features while preserving the main information. The fully connected layer can map the reduced features to a fixed-dimensional space to form the final face static features. This processing method makes the features more compact and easy to process, and also facilitates subsequent identification and analysis of the pilot's workload.

[0294] The facial light flow feature and the facial static feature are fused to generate a facial fusion feature. The state of the face can be more comprehensively described, richer information can be provided, and the understanding and analysis capability of the pilot's facial expression can be improved. The gaze point extraction branch in the fourth sub-feature extraction branch extracts features from the eye movement image data, and outputs an eye movement trajectory corresponding to the eye movement image data. The eye movement trajectory can accurately reflect the transfer path of the pilot's visual attention when observing the external environment. The eye movement trajectory is extracted to determine the gaze point distribution feature corresponding to the eye movement image data; the gaze point distribution feature can mine the area of visual focus of the pilot during observation. For example, in a complex flight scene, by analyzing the gaze point distribution, it can be found that the pilot is more inclined to focus on which key information or instrument when facing an emergency, which has important reference value for optimizing the layout of flight instruments and improving the effectiveness of information display. The eye movement extraction branch in the fourth sub-feature extraction branch extracts features from the eye movement image data, and outputs an eye movement feature corresponding to the eye movement image data. The eye movement feature also covers other important information such as pupil size change, blink frequency, etc. These comprehensive eye movement features can comprehensively capture the eye movement behavior of the pilot and more accurately reflect their physiological and psychological state. For example, the size change of the pupil can reflect the emotional state and attention concentration of the pilot, and the increase of the blink frequency may indicate fatigue or tension. The gaze point distribution feature and the eye movement feature are fused to generate an eye movement fusion feature, ensuring the comprehensiveness and accuracy of the generated eye movement fusion feature. The facial fusion feature and the eye movement fusion feature are fused to generate a visual modality feature. Thus, the information conveyed by the pilot's facial expression and eye movement behavior can be integrated to form a more complete and rich visual modality feature representation. This visual modality feature can more comprehensively reflect the visual state and emotional state of the pilot, providing stronger support for subsequent tasks such as pilot workload recognition based on the feature.

[0295] The visual modality feature is taken as a first query matrix, the time sequence modality feature is taken as a first key matrix and a first value matrix; the first dependency weight of the visual modality feature on the time sequence modality feature is calculated, thereby ensuring the accuracy of the calculated first dependency weight. The time sequence modality feature is taken as a second query matrix, the visual modality feature is taken as a second key matrix and a second value matrix; the second dependency weight of the time sequence modality feature on the visual modality feature is calculated, thereby ensuring the accuracy of the calculated second dependency weight. The target time sequence feature is obtained by multiplying the first dependency weight by the time sequence modality feature, and the target visual feature is obtained by multiplying the second dependency weight by the visual modality feature. Such a weighting operation can strengthen the part with a higher correlation with the feature of another modality, so that the target feature can better reflect the correlation between different modalities. For example, in the target time sequence feature, the physiological signal feature closely related to the visual information is enhanced, so that it is more easily paid attention to and utilized in subsequent analysis. In addition, through the weighting operation, the part with a lower correlation with the feature of another modality is weakened, the redundant information in the feature is reduced, and the quality and efficiency of the feature are improved. The target time sequence feature and the target visual feature are combined to generate a target fusion feature. The deep fusion of the visual modality and the time sequence modality is completed. The fused feature integrates the advantage information of the two modalities and can more comprehensively and accurately reflect the working state of the pilot.

[0296] Based on the target fusion feature, a current workload level corresponding to the target pilot is output. The target fusion feature contains multiple aspects of information and can more accurately reflect the actual workload of the pilot. Compared with the evaluation based on only a single modality feature, this way can reduce the probability of misjudgment and omission, and provide more reliable workload evaluation results for the pilot, the flight management personnel and the related system.

[0297] In the embodiment, a pilot workload identification device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and contemplated.

[0298] The embodiment provides a pilot workload identification device, as shown in Figure 5 The device includes:

[0299] The acquisition module 301 is configured to acquire physiological time sequence data, electroencephalogram time sequence data, eye movement image data and face image data corresponding to a target pilot; the physiological time sequence data includes at least one of electrocardiogram data, electromyogram data and respiration data;

[0300] The first output module 302 is configured to input the physiological time series data, the electroencephalogram time series data, the eye movement image data and the face image data into a preset workload identification model, and output a current workload level corresponding to the target pilot.

[0301] The second output module 303 is configured to output early warning information based on the current workload level.

[0302] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for identifying pilot workload, characterized in that, The method includes: Acquire physiological time-series data, electroencephalogram (EEG) time-series data, eye-tracking image data, and facial image data corresponding to the target pilot; the physiological time-series data includes at least one of electrocardiogram (ECG), electromyogram (EMG), and respiratory data. The physiological time series data, the electroencephalogram time series data, the eye movement image data, and the facial image data are input into a preset workload recognition model, and the current workload level corresponding to the target pilot is output. Based on the current workload level, output early warning information; The preset workload recognition model includes a first feature extraction network and a second feature extraction network. The step of inputting the physiological time-series data, the electroencephalogram (EEG) time-series data, the eye-tracking image data, and the facial image data into the preset workload recognition model, and outputting the current workload level corresponding to the target pilot, includes: The physiological time-series data and the EEG time-series data are input into the first feature extraction network to generate time-series modal features; The facial image data and the eye movement image data are input into the second feature extraction network to generate visual modality features; The temporal modality features and the visual modality features are fused to generate target fusion features; Based on the target fusion features, the current workload level corresponding to the target pilot is output; The first feature extraction network includes a first sub-feature extraction branch, a second sub-feature extraction branch, and a first sub-feature fusion network. The step of inputting the physiological time-series data and the EEG time-series data into the first feature extraction network to generate the time-series modal features includes: The physiological time series data is input into the first sub-feature extraction branch, and the EEG time series data is input into the second sub-feature extraction branch; The first sub-feature extraction branch performs feature extraction on the physiological time-series data to generate a physiological signal sequence; the physiological signal sequence includes at least one of the following: respiratory data features, electrocardiogram (ECG) data time-domain features, ECG data frequency-domain features, ECG data nonlinear features, and electromyography (EMG) data features. Based on the sparse attention scoring method, feature extraction is performed on the physiological signal sequence to generate a physiological signal matrix; The physiological signal matrix is ​​used to extract features based on multi-period convolution kernels, and physiological modality features are output. The second sub-feature extraction branch calculates the energy spectral density and power spectral density corresponding to the EEG time-series data; Based on the energy spectral density and the power spectral density, feature extraction is performed on the EEG time-series data to output EEG modal features; The first sub-feature fusion network fuses the physiological modality features and the EEG modality features to generate the temporal modality features.

2. The method according to claim 1, characterized in that, The sparse attention-based scoring method extracts features from the physiological signal sequence to generate a physiological signal matrix, including: The physiological signal sequence is linearly transformed to generate an initial query matrix, an initial key matrix, and an initial value matrix; Calculate the similarity score between each query vector in the initial query matrix and all key vectors in the initial key matrix; Introducing physiological signal-specific priors, the sparsity metric value corresponding to each query vector in the initial query matrix is ​​calculated based on the similarity score; Based on the sparsity metric, sort the query vectors and select the Top -u target query vectors with the highest sparsity metric. A sparse query matrix is ​​formed based on the top - u target query vectors; The physiological signal matrix is ​​generated based on the sparse query matrix, the initial key matrix, and the initial value matrix.

3. The method according to claim 1, characterized in that, The step of extracting features from the EEG time-series data based on the energy spectral density and the power spectral density, and outputting the EEG modal features, includes: Each frequency point in the EEG time series data is mapped to a first initial frequency band, a second initial frequency band, and a third initial frequency band. Based on the energy spectral density and the power spectral density, the average energy and average power of the first initial frequency band, the second initial frequency band, and the third initial frequency band are calculated respectively. Based on the average energy and average power corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band, respectively, calculate the weight information corresponding to the first initial frequency band, the second initial frequency band, and the third initial frequency band; The first initial frequency band, the second initial frequency band, and the third initial frequency band are multiplied by the corresponding weight information to generate the first target frequency band, the second target frequency band, and the third target frequency band. Phase information corresponding to the EEG time-series data is extracted using Hilbert transform to generate a phase matrix; Based on the first target frequency band, the second target frequency band, the third target frequency band, and the phase matrix, the information of the time dimension, frequency dimension, and phase dimension corresponding to the EEG time series data is integrated to construct a time-frequency-phase three-dimensional feature and generate the EEG modal feature.

4. The method according to claim 1, characterized in that, The second feature extraction network includes a third sub-feature extraction branch, a fourth sub-feature extraction branch, and a second sub-feature fusion network. The step of inputting the facial image data and the eye-tracking image data into the second feature extraction network to generate visual modal features includes: The facial image data is input into the third sub-feature extraction branch, and the eye movement image data is input into the fourth sub-feature extraction branch; The dynamic feature extraction branch in the third sub-feature extraction branch performs optical flow feature extraction on the facial image data and outputs the facial optical flow features corresponding to the facial image data. The static feature extraction branch in the third sub-feature extraction branch performs static feature extraction on the facial image data and outputs the facial static features corresponding to the facial image data. The facial optical flow features and the facial static features are integrated to generate facial fusion features; The fixation point extraction branch in the fourth sub-feature extraction branch performs feature extraction on the eye movement image data and outputs the eye movement trajectory corresponding to the eye movement image data; Feature extraction is performed on the eye movement trajectory to determine the gaze point distribution features corresponding to the eye movement image data; The eye movement extraction branch in the fourth sub-feature extraction branch performs feature extraction on the eye movement image data and outputs the eye movement features corresponding to the eye movement image data; The fixation point distribution features and the eye movement features are integrated to generate eye movement fusion features; The facial fusion features and the eye-tracking fusion features are fused together to generate the visual modality features.

5. The method according to claim 4, characterized in that, The facial image data consists of at least three frames. The dynamic feature extraction branch in the third sub-feature extraction branch performs optical flow feature extraction on the facial image data and outputs the facial optical flow features corresponding to the facial image data, including: Target detection is performed on each frame of the facial image data to determine the region of interest in each frame of the facial image data; the region of interest includes the orbicularis oculi muscle, the orbicularis oris muscle, and the frontalis muscle. Based on a preset streamer algorithm, a motion vector corresponding to each facial image data is calculated according to the position information of each region of interest in the corresponding facial image data; the motion vector includes horizontal displacement and vertical displacement. Calculate the motion amplitude, motion direction, and motion acceleration corresponding to the motion vector; The motion vector, motion amplitude, motion direction, and motion acceleration are fused to generate the facial optical flow feature.

6. The method according to claim 4, characterized in that, The static feature extraction branch includes multiple parallel branches; the static feature extraction branch in the third sub-feature extraction branch performs static feature extraction on the facial image data and outputs the facial static features corresponding to the facial image data, including: The static feature extraction branch extracts features from the facial image data to obtain initial static features; The initial static features are input into each of the first parallel branches; The first parallel branch weights the initial static features based on two dimensions: channel and space, to obtain weighted static features; Based on a preset number of multi-size local feature extraction modules, local feature extraction is performed on the weighted static features to obtain local static features at various scales. The local static features described above are spliced ​​together to generate multi-scale local features; The multi-scale local features are shrunk using a global feature shrinkage attention module to generate target local features. The target local features are input into the next parallel branch, and this process is repeated until the last parallel branch is completed. The results output from each of the parallel branches are globally pooled, and the facial static features are output based on the fully connected layer.

7. The method according to claim 1, characterized in that, The process of fusing the temporal modality features and the visual modality features to generate target fusion features includes: The visual modal features are used as the first query matrix, and the temporal modal features are used as the first key matrix and the first value matrix; Calculate the first dependency weight of the visual modality feature on the temporal modality feature; The temporal modality features are used as the second query matrix, and the visual modality features are used as the second key matrix and the second value matrix. Calculate the second dependency weight of the temporal modality feature on the visual modality feature; The target temporal feature is obtained by multiplying the first dependency weight by the temporal modality feature; The target visual features are obtained by multiplying the second dependency weight by the visual modality features; The target temporal features and the target visual features are merged to generate the target fusion features.

8. A pilot workload identification device, characterized in that, The device includes: The acquisition module is used to acquire physiological time-series data, electroencephalogram (EEG) time-series data, eye-tracking image data, and facial image data corresponding to the target pilot; the physiological time-series data includes at least one of electrocardiogram (ECG) data, electromyogram (EMG) data, and respiratory data. The first output module is used to input the physiological time-series data, the EEG time-series data, the eye-tracking image data, and the facial image data into a preset workload recognition model, and output the current workload level corresponding to the target pilot. The preset workload recognition model includes a first feature extraction network and a second feature extraction network. The step of inputting the physiological time-series data, the EEG time-series data, the eye-tracking image data, and the facial image data into the preset workload recognition model and outputting the current workload level corresponding to the target pilot includes: inputting the physiological time-series data and the EEG time-series data into the first feature extraction network to generate temporal modal features; inputting the facial image data and the eye-tracking image data into the second feature extraction network to generate visual modal features; fusing the temporal modal features and the visual modal features to generate target fusion features; and outputting the current workload level corresponding to the target pilot based on the target fusion features. The first feature extraction network includes a first sub-feature extraction branch, a second sub-feature extraction branch, and a first sub-feature fusion network. The method for inputting the physiological time-series data and the EEG time-series data into the first feature extraction network to generate the time-series modal features includes: inputting the physiological time-series data into the first sub-feature extraction branch and inputting the EEG time-series data into the second sub-feature extraction branch; the first sub-feature extraction branch extracts features from the physiological time-series data to generate a physiological signal sequence; the physiological signal sequence includes at least one of respiratory data features, ECG data time-domain features, ECG data frequency-domain features, ECG data nonlinear features, and electromyographic data features; extracting features from the physiological signal sequence based on a sparse attention scoring method to generate a physiological signal matrix; extracting features from the physiological signal matrix based on a multi-period convolution kernel to output physiological modal features; the second sub-feature extraction branch calculates the energy spectral density and power spectral density corresponding to the EEG time-series data; extracting features from the EEG time-series data based on the energy spectral density and the power spectral density to output EEG modal features; and the first sub-feature fusion network fuses the physiological modal features and the EEG modal features to generate the time-series modal features. The second output module is used to output early warning information based on the current workload level.

Images