Controller cognitive load quantitative evaluation method, device, equipment, medium and product
By collecting and fusing multimodal physiological data from controllers, a dynamic regression network model for cognitive load is constructed, which solves the problems of subjectivity in assessing controller decision-making effectiveness and data fusion in existing technologies, and achieves high-precision quantification and real-time assessment of cognitive load.
Patent Information
- Application Number
- CN202511247620.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-03
- Publication Date
- 2025-12-02
AI Technical Summary
Existing technologies rely on subjective observation and post-hoc interviews when assessing controller decision-making effectiveness. These methods are highly subjective and make it difficult to effectively integrate heterogeneous physiological data, which affects the accurate analysis of cognitive states and decision-making processes.
By collecting multimodal physiological data (eye movement, skin conductance, and electroencephalography) from controllers, feature extraction and cross-modal feature fusion are performed to construct a cognitive load dynamic regression network model. The model is then trained using slice datasets to achieve quantitative assessment of cognitive load.
It enables full-cycle quantitative analysis of controllers' cognitive load, improves the scientificity and accuracy of assessment results, and can assess controllers' cognitive state and decision-making process in real time, overcoming the subjectivity of traditional methods and the challenges of heterogeneous data fusion.
Smart Images
Figure CN121040883A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of aviation safety management and artificial intelligence technology, and in particular to a method, device, equipment, medium and product for quantitatively assessing the cognitive load of air traffic controllers. Background Technology
[0002] In the civil aviation sector, air traffic control is a crucial link in ensuring aviation safety and efficient operation. With the continuous growth of global air transport volume, the complexity of air traffic has increased significantly, making the scientific evaluation of controller decision-making effectiveness an important issue in the civil aviation field.
[0003] The development of biosensing technology has provided new avenues for assessing controller decision-making effectiveness. Multimodal physiological data acquisition technologies, such as eye tracking, electrodermal response monitoring, and electroencephalography (EEG), can achieve synchronous acquisition with millisecond-level precision, laying the foundation for analyzing the correlation between controller cognitive states and decision-making processes. However, current controller effectiveness assessments primarily rely on subjective observation and post-event interviews, which are highly subjective. Furthermore, in air traffic control scenarios, the physiological data collected by eye trackers, electrodermal sensors, and EEG devices are heterogeneous, with varying sampling rates, leading to challenges in spatiotemporal alignment during data fusion. Traditional methods struggle to effectively process this heterogeneous data, affecting the subsequent accurate analysis of controller cognitive states and decision-making processes. Summary of the Invention
[0004] The purpose of this application is to provide a method, device, equipment, medium, and product for quantitative assessment of controller cognitive load. By integrating multimodal physiological data and artificial intelligence technology, it can achieve full-cycle quantitative analysis of controller cognitive load and improve the scientificity and accuracy of the assessment results.
[0005] To achieve the above objectives, this application provides the following solution.
[0006] Firstly, this application provides a method for quantitatively assessing the cognitive load of controllers, including: In air traffic control scenarios, multimodal physiological data of controllers are collected, including eye movement data, raw electrodermal signals, and raw electroencephalogram (EEG) signals. Feature extraction and cross-modal feature fusion processing are performed on multimodal physiological data to obtain a multimodal feature tensor; A cognitive load dynamic regression network model is constructed; the input of the cognitive load dynamic regression network model is a multimodal feature tensor, and the output is the corresponding cognitive load index. Construct a slice dataset for model training; the slice dataset includes time slices of multimodal feature tensors and corresponding slice labels; A cognitive load dynamic regression network model was trained using a slice dataset. The loss function was calculated and the parameters were optimized. The trained cognitive load dynamic regression network model was then used as a dynamic cognitive load calculation model for subsequent use. During the experimental phase, the multimodal feature tensor corresponding to the controller's current multimodal physiological data is input into the dynamic cognitive load calculation model to obtain the corresponding cognitive load index.
[0007] Optionally, the collection of controllers' multimodal physiological data specifically includes: Eye trackers were used to collect controllers' eye movement data, including fixation coordinates, pupil diameter, and blink interval. Raw skin conductance signals from controllers are collected using skin conductance sensors. Raw electroencephalogram (EEG) signals of controllers were collected using EEG equipment.
[0008] Optionally, the step of performing feature extraction and cross-modal feature fusion processing on multimodal physiological data to obtain a multimodal feature tensor specifically includes: For eye-tracking data, cubic spline interpolation is used to resample the sampling rate from the eye tracker to the EEG device sampling rate, forming... Dimensional eye movement temporal features ;in For time points; The original electrodermal signal is first subjected to a Butterworth low-pass filter to remove high-frequency noise, resulting in... The standardized electrodermal signal sequence was then extracted using a 3-layer LSTM encoder. Characteristics of the skin conductance signal ; Wavelet transform is used to extract from the raw EEG signal. Band energy is formed after energy calculation and dimensionality reduction. EEG signal characteristics of VI ; Using a cross-modal attention module with a learnable weight matrix to... , , The three features are weighted and fused to output the result. Multimodal feature tensors of 1D.
[0009] Optionally, the constructed cognitive load dynamic regression network model specifically includes: a time slice layer, a spatiotemporal convolutional layer, an attention gating layer, and a dynamic regression layer; The time-slicing layer is used to process the input multimodal feature tensor according to... The time interval of seconds is divided into The first time slice, the second Each time slice is denoted as ;in , It is a positive integer; The spatiotemporal convolutional layer uses a 3-layer one-dimensional convolutional neural network to extract each time slice. Local patterns and features, to obtain Local eigenvectors of dimension ; The attention gating layer is used to calculate the scene relevance weights for air traffic control scenarios. ,use right Weighting is performed based on specific scene features to obtain each time slice. Scene weighted features ; The dynamic regression layer is used to process each time slice Scene weighted features Mapped to cognitive load index .
[0010] Optionally, constructing the slice dataset for model training specifically includes: Obtain the controller's scale scores and normalize them to the [0,1] interval as global labels. ; Based on time slices within a unit of time instruction density Determine each time slice weight And match global tags using a weighted average. The weighted average formula is as follows: ; Each time slice is obtained by solving the weighted average formula. slice tags ; Slice the time and slice labels The slice dataset is constructed by storing the slices in a one-to-one correspondence.
[0011] Optionally, the loss function of the cognitive load dynamic regression network model is: ;in The number of time slices in a batch of training samples; It is the cognitive load index predicted by the model. ; This is the calculated loss value.
[0012] Secondly, this application provides a device for quantitatively assessing the cognitive load of controllers, comprising: The data acquisition module is used to collect multimodal physiological data of air traffic controllers in air traffic control scenarios, including eye movement data, raw skin conductance signals, and raw electroencephalogram (EEG) signals. The feature extraction and fusion module is used to extract features from multimodal physiological data and perform cross-modal feature fusion processing to obtain multimodal feature tensors; The network model construction module is used to construct a cognitive load dynamic regression network model; the input of the cognitive load dynamic regression network model is a multimodal feature tensor, and the output is the corresponding cognitive load index. A slice dataset construction module is used to construct a slice dataset for model training; the slice dataset includes time slices of multimodal feature tensors and corresponding slice labels; The model training module is used to train the cognitive load dynamic regression network model using the sliced dataset, calculate the loss function and optimize the parameters, and use the trained cognitive load dynamic regression network model as the dynamic cognitive load calculation model for subsequent use. The experimental module is used to input the multimodal feature tensor corresponding to the controller's current multimodal physiological data into the dynamic cognitive load calculation model during the experimental phase, so as to obtain the corresponding cognitive load index.
[0013] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the controller cognitive load quantification assessment method.
[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the controller cognitive load quantification assessment method.
[0015] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the controller cognitive load quantification assessment method.
[0016] Based on the specific embodiments provided in this application, the following technical effects are disclosed.
[0017] This application provides a method, device, equipment, medium, and product for quantitatively assessing controller cognitive load. By extracting features from multimodal physiological data and fusing cross-modal features, it provides a unified data foundation with high temporal consistency and multidimensional integration for subsequent analysis of controller cognitive states and decision-making processes. This overcomes the shortcomings of traditional methods in handling heterogeneous data, allowing physiological data from different modalities to synergistically contribute to cognitive state analysis. By constructing a dynamic regression network model for cognitive load, it achieves millisecond-level cognitive load quantification, accurately capturing changes in controller cognitive load at different times. By constructing a slice dataset for model training, each time slice has a corresponding precise label, building a reliable bridge between input and output in model training, ensuring the model learns the true mapping relationship between cognitive load and multi-source features. In the experimental stage, the multimodal feature tensors extracted from controller eye movement, electrodermal transfer (EDT), and electroencephalogram (EEG) data are input into the trained dynamic cognitive load calculation model, which quickly outputs the corresponding cognitive load index, enabling real-time assessment of controller cognitive states at different task stages and in different scenarios. This application organically combines data fusion, model design, and label construction to form a complete assessment loop. This helps overcome the shortcomings of traditional assessment methods, such as strong subjectivity and difficulty in quantification, and provides an operable and highly accurate technical means for scientifically assessing controllers' cognitive load. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating a method for quantitatively assessing the cognitive load of controllers according to this application. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0021] This application proposes a method, device, equipment, medium, and product for quantitative assessment of controller cognitive load, aiming to achieve full-cycle quantitative analysis of controller cognitive load by integrating multimodal physiological data and artificial intelligence technology, thereby improving the scientificity and accuracy of the assessment results.
[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0023] In one exemplary embodiment, such as Figure 1 As shown, a method for quantitatively assessing the cognitive load of controllers is provided, including the following steps 1 to 6.
[0024] Step 1: In an air traffic control scenario, collect multimodal physiological data from air traffic controllers, including eye movement data, raw electrodermal signals, and raw electroencephalogram (EEG) signals.
[0025] The development of biosensing technology has provided new avenues for assessing controller decision-making effectiveness. Multimodal physiological data acquisition technologies, such as eye tracking, skin conductance monitoring, and electroencephalography (EEG), can achieve synchronous acquisition with millisecond-level precision, laying the foundation for analyzing the correlation between controller cognitive states and decision-making processes. In the data acquisition phase, this application uses an eye tracker to collect controller eye movement data, typically at a sampling rate of 250Hz. The collected eye movement data includes fixation coordinates. Pupil diameter and blink interval Raw electrodermal signals from controllers are acquired using a skin conductance sensor, typically at a sampling rate of 100 Hz. Raw electroencephalogram (EEG) signals from controllers are acquired using an EEG device, typically at a sampling rate of 500 Hz.
[0026] Step 2: Perform feature extraction and cross-modal feature fusion processing on the multimodal physiological data to obtain the multimodal feature tensor.
[0027] In air traffic control scenarios, physiological data such as eye movements, conductance of the skin, and electroencephalogram (EEG) activity of air traffic controllers are closely related to their operational behaviors. Furthermore, data collected by different sensors differ in sampling rates and other aspects, necessitating the fusion of this heterogeneous data. To address this, this application designs a multi-source sensor data fusion network to achieve spatiotemporal alignment between physiological indicators and operational behaviors, providing a unified data foundation for subsequent analysis of air traffic controllers' cognitive states and decision-making processes. The input to the multi-source sensor data fusion network consists of raw signals collected by an eye tracker (common sampling rate 250Hz), a conductance of the skin sensor (common sampling rate 100Hz), and an EEG device (common sampling rate 500Hz). The output is a time-synchronized multimodal feature tensor.
[0028] To address the issue of sampling rate differences among heterogeneous sensors, this application proposes a temporal resampling-attention alignment mechanism. Specifically, the processing procedure of the multi-source sensor data fusion network includes steps 2.1 to 2.4.
[0029] Step 2.1: For eye-tracking data, the sampling rate is resampled from the eye tracker sampling rate to the EEG device sampling rate using cubic spline interpolation, forming... Dimensional eye movement temporal features ;in This refers to a point in time.
[0030] The input eye-tracking data includes gaze coordinates. Pupil diameter and blink interval These data can reflect the controller's visual attention allocation and eye activity status. Because the eye tracker's sampling rate (250Hz) differs from the EEG device's sampling rate (500Hz), cubic spline interpolation is used to resample the eye tracker data to 500Hz to unify the sampling frequency, thereby obtaining the dimensions... Dimensional eye movement temporal features ,in The number of time points, where 4 represents four eye-tracking related feature values, i.e. , and .
[0031] Step 2.2: For the original electrodermal signal, high-frequency noise is first removed by Butterworth low-pass filtering to obtain... The standardized electrodermal signal sequence was then extracted using a 3-layer LSTM encoder. Characteristics of the skin conductance signal .
[0032] The raw electrodermal signal reflects the body's physiological stress response, and its sampling rate is 100Hz. First, the raw electrodermal signal is subjected to a Butterworth low-pass filter to remove high-frequency noise interference, obtaining the received signal with a specific dimension. The standardized Galvanic Skin Response (GSR) sequences were obtained, with one feature per time point. Features were then extracted using a three-layer LSTM (Long Short-Term Memory) encoder, with 8, 16, and 8 nodes per layer, respectively. LSTM excels at processing time-series data and can capture long-term dependencies in Galvanic Skin Response signals, ultimately yielding... 3D feature map as a feature of skin conductance signal The 8-dimensional features are dimensions that are obtained through network model training and can effectively characterize the features of the skin conductance signal.
[0033] Step 2.3: Extract the original EEG signal using wavelet transform. Band energy is formed after energy calculation and dimensionality reduction. EEG signal characteristics of VI .
[0034] The raw EEG spectrum contains rich information on brain electrical activity, which can be extracted using wavelet transform. Band (4-8Hz) energy This wavelength is closely related to cognitive activity in the brain, and has been observed to... dimensional vectors as features of EEG signals Specifically, the calculation is performed according to steps 2.3.1 to 2.3.3 below.
[0035] Step 2.3.1: First, apply a Butterworth filter to the raw EEG signal (sampling rate 500Hz) for a 1-40Hz bandpass filter to remove high-frequency noise (such as electromyographic interference) and low-frequency drift (such as respiratory artifacts). Then, segment the continuous EEG signal into... There are 2 time slices (each slice = 2ms), and each time slice corresponds to one data sample.
[0036] Step 2.3.2: Perform wavelet transform decomposition.
[0037] The complex Morlet wavelet is the complex form of the Morlet wavelet. It exhibits good time-frequency focusing properties, making it suitable for non-stationary EEG analysis. Its formula is: (1); in, It is a complex Morlet wavelet function, which is a basis function of wavelet transform and represents the characteristics of the signal at different time and frequency positions. This is the bandwidth parameter, set to 2 here, used to control the time-frequency resolution balance of the wavelet. This is the center frequency, which is 6Hz in this application (corresponding to...). The wavelet band center determines the main frequency range that the wavelet function is sensitive to, allowing the wavelet to analyze the signal near a specific frequency. It is a time variable, representing the independent variable of the wavelet function as it changes over time. It describes the shape of the wavelet on the time axis and corresponds to the time dimension of the original signal. It is obtained by sliding... It enables the analysis of signals at different times. It is in complex exponential form, with superscript. The imaginary unit, Used to represent the oscillatory components of wavelets, it facilitates the simultaneous analysis of the amplitude and phase information of signals, enabling wavelet transform to obtain the time-frequency and phase characteristics of signals.
[0038] According to the target frequency With center frequency Relationship, calculus scale : (2); in Hz is the sampling rate. Calculations at 4Hz show that... =750, =8Hz =375. This represents the key frequency to be analyzed. This value is specified as a priori parameter based on expert experience, where 4-8Hz... Wave frequency is related to cognitive load, relaxation state, etc., and can be adjusted by setting... =[4,8]Hz can be used to extract the time-frequency characteristics of this frequency band, and then analyze the state of people related to cognition.
[0039] Then perform a continuous wavelet transform (CWT) for each time point. EEG signal implement: (3); in, Represents complex conjugation, that is The sign of the imaginary part is inverted. It is the time-dependent variable of the signal, used to traverse the entire time axis of the signal to describe the change of the signal over time for integration. These are the wavelet coefficients used for calculation.
[0040] Step 2.3.3: Perform Band energy extraction.
[0041] First, energy calculations are performed, focusing on the scale. The time-frequency energy is obtained by squared the modulus of the wavelet coefficients (i.e., 4-8Hz). : (4); in and respectively scale The minimum and maximum values.
[0042] Then, dimensionality reduction is performed: summation is performed along the scale dimension to obtain the output for each time point. of Total energy value of the band, final output EEG signal characteristics of VI : (5).
[0043] Step 2.4: Use a cross-modal attention module with a learnable weight matrix to... , , The three features are weighted and fused to output the result. Multimodal feature tensors of 1D.
[0044] Finally, using a cross-modal attention module, the three features mentioned above are weighted and fused using formula (6): (6); in, , and These represent the processed eye movement timing features, skin conductance signal features, and EEG signal features, respectively. It is a learnable weight matrix used to adjust the importance of features of different modalities. The function normalizes the weighted result into a probability distribution to obtain the attention weights. Through attention weights The three features are weighted and summed to obtain the final output. 3D feature tensor as multimodal feature tensor Its time alignment error is less than 2ms, ensuring a high degree of temporal consistency of the fused data.
[0045] Step 3: Construct a cognitive load dynamic regression network model; the input of the cognitive load dynamic regression network model is a multimodal feature tensor, and the output is the corresponding cognitive load index.
[0046] A cognitive load dynamic regression network model is designed to achieve millisecond-level cognitive load quantification, accurately assessing the cognitive load level of air traffic controllers at different times, and characterizing it using a cognitive load index. Since the task complexity and pressure faced by controllers vary in different air traffic control scenarios, and their cognitive load characteristics also differ, a network model that can adapt to different scenarios needs to be designed. The input of this cognitive load dynamic regression network model is the multimodal feature tensor output from step 2. The output is the cognitive load index. To adapt to the differences in load characteristics across various air traffic control scenarios, this application proposes a scenario-adaptive weight learning mechanism.
[0047] Specifically, the cognitive load dynamic regression network model constructed in this application includes: 3.1) a time slice layer, 3.2) a spatiotemporal convolutional layer, 3.3) an attention gating layer, and 3.4) a dynamic regression layer. These will be described in detail below.
[0048] 3.1) Time slice layer The time-slicing layer is used to process the input multimodal feature tensor. according to The time interval of seconds is divided into The first time slice, the second Each time slice is denoted as ;in , It is a positive integer.
[0049] Specifically, input features First, it is divided into time slice layers. Seconds clip, here =2, this division is because a 2-second time window can capture the trend of cognitive load changes while avoiding the loss of details due to an excessively large window or the introduction of too much noise due to an excessively small window. For each time slice Perform the following operations at each level.
[0050] 3.2) Spatiotemporal Convolutional Layer The spatiotemporal convolutional layer uses a 3-layer 1D-CNN (One-Dimensional Convolutional Neural Network) to extract each time slice. Local patterns and features, to obtain Local eigenvectors of dimension .
[0051] Each 1D-CNN layer used in the spatiotemporal convolutional layers has a kernel size of 5 and a stride of 2. The number of channels in the three 1D-CNN layers are 16, 32, and 32, respectively. 1D-CNN can effectively extract local patterns and features from time-series data. A kernel size of 5 can capture feature information from five adjacent time points; a stride of 2 controls the convolutional stride, reducing computation while ensuring the completeness of feature extraction; and 32 channels can learn multiple different feature representations. The final result is... Local eigenvectors of dimension .
[0052] 3.3) Attention Gating Layer The attention gating layer is used to calculate the scene relevance weights for air traffic control scenarios. ,use right Weighting is performed based on specific scene features to obtain each time slice. Scene weighted features .
[0053] After passing through the attention gating layer, the scene relevance weights are first calculated. : (7); in, The scene encoding vector is obtained by encoding prior knowledge of different air traffic control scenarios. It represents the features of a specific scenario and is measured according to the following aspects: = [Traffic density, instruction complexity, emergency signage, weather conditions] (8); Traffic density is the number of aircraft per square kilometer; command complexity is the number of procedural commands + emergency commands per unit time. 2) Emergency situations are identified by binary features of 0 and 1; meteorological conditions are identified by three-dimensional category features of (sunny, cloudy, rainy / foggy). It is a learnable weight matrix used to adjust the correlation between the scene encoding vector and the features; For activation functions (such as the sigmoid function), the calculation result is mapped to the range of 0-1 to obtain the scene relevance weights. This weight is used to highlight features relevant to the current scene.
[0054] use right Weighting is performed based on specific scene features to obtain each time slice. Scene weighted features : (9).
[0055] 3.4) Dynamic Regression Layer The dynamic regression layer is used to process each time slice Scene weighted features Mapped to cognitive load index .
[0056] Design a dynamic regression layer, which is a fully connected neural network regressor, to incorporate scene-weighted features. Mapped to cognitive load index The formula is as follows: (10); in, The regression weight matrix is a learnable matrix with dimensions (32×1); As a bias term, it can be optimized during training. Use the Sigmoid activation function to ensure that the output is between [0, 1].
[0057] Step 4: Construct a slice dataset for model training; the slice dataset includes time slices of multimodal feature tensors and corresponding slice labels.
[0058] This application designs a novel label allocation method based on the NASA-TLX (NASA Task Load Index) scale, which combines instruction density to map global labels to time slice labels, providing high-quality regression labels for model training. Step 4 specifically includes steps 4.1 to 4.4.
[0059] Step 4.1: Obtain the controller's scale scores and normalize them to the [0,1] interval as global labels. .
[0060] Organizational controllers used the NASA-TLX scale to test and score the cognitive load of individuals for each individual task. This scale includes six core dimensions, such as psychological needs, time needs, and performance levels, with scores ranging from 0 to 100. The scores for the six core dimensions were then averaged and normalized to the [0,1] interval, serving as the regression label (i.e., the global label). This leads to a regression problem.
[0061] Step 4.2: Based on the time slices within a unit of time. instruction density Determine each time slice weight And match global tags using a weighted average. .
[0062] The instruction density per unit time is used to determine the weight of each time slice (hereinafter referred to as slice), and a weighted average is used to match the global label of the scale. This process uses "instruction density" as an indicator of the contribution of each time slice to the global cognitive load; that is, the higher the instruction density, the more likely the cognitive load of the corresponding slice is to be high, and the greater its weight. A weighted average formula is used to make the weighted result of all slices equal to the global label, realizing the mapping from global label to slice label. The specific process is as follows.
[0063] ① Time slice division: according to =1 piece of multimodal feature tensor corresponding to the collected multimodal physiological data every 2 seconds Segmentation is performed to obtain A slice, denoted as .
[0064] ② Instruction density extraction: For each slice The number of instructions issued by controllers is counted and categorized into control instructions and emergency instructions. Each control instruction is assigned a weight. =0.8, Special Incident Command Weight =1.2. Multiply the number of instructions by the corresponding instruction weight and then divide by the slice duration (in seconds) to obtain the slice. instruction density (Unit: items / second), reflecting the workload of the period.
[0065] ③ Weighting based on instruction density Weight normalization: normalize the weights of each slice instruction density Normalization to weights To make the total weights equal to 1, the formula is: (11).
[0066] ④ Associate slice tags with global tags Slice The corresponding slice tag is That is, the slice to be solved. The corresponding cognitive load index is Let the global tag be... , which is the cognitive load index of the entire task measured by the scale, has been normalized to [0,1].
[0067] Establish a weighted average formula: the sum of "label × weight" for all slices equals the global label, that is: (12).
[0068] Step 4.3: Obtain each time slice by solving the weighted average formula. slice tags .
[0069] Considering that cognitive load is positively correlated with task complexity, it is known that slice labels With its own instruction density Positive correlation can make ( (where is the proportionality coefficient), substituting it into the weighted average formula, we get: (13).
[0070] because Simplifying, we get ,Right now Therefore, the formula for calculating slice labels is as follows: (14); in The average instruction density for all slices, used for normalization, ensures Within the range [0,1].
[0071] Step 4.4: Slice the time. and slice labels The slice dataset is constructed by storing the slices in a one-to-one correspondence.
[0072] Each time slice and corresponding slice tags As a training sample, a slice dataset is constructed.
[0073] Step 5: Train the cognitive load dynamic regression network model using the slice dataset, calculate the loss function and optimize the parameters, and use the trained cognitive load dynamic regression network model as the dynamic cognitive load calculation model for subsequent use.
[0074] Based on the regression problem constructed in step 4, the Mean Squared Error (MSE) loss function is designed, as shown in the following formula: (15); in The number of time slices in a batch of training samples; These are slice labels calculated based on the scale, which serve as the true values during the model training process. These are the slice labels predicted by the cognitive load dynamic regression network model, which are the cognitive load indices output by the model. , as the predicted value during the model training process; This is the calculated loss value.
[0075] The slice label calculated in step 4 As slices of time Cognitive load labels (2 seconds per slice) are generated, and then a mapping between input and label pairs is constructed. Finally, each acquisition... The model is trained by slicing the model, the loss is calculated and the parameters are optimized, and finally a trained cognitive load dynamic regression network model is obtained. Its parameters are encapsulated as a dynamic cognitive load calculation model for subsequent use.
[0076] Step 6: During the experimental phase, the multimodal feature tensor corresponding to the controller's current multimodal physiological data is input into the dynamic cognitive load calculation model to obtain the corresponding cognitive load index.
[0077] During the experimental phase, the multimodal physiological data collected by controllers will be processed using the multi-source sensor data fusion network from step 2 to generate corresponding multimodal feature tensors. By inputting this information into the dynamic cognitive load calculation model, the corresponding cognitive load index can be obtained. Cognitive Load Index Based on Calculation This allows for a quantitative assessment of controllers' decision-making effectiveness.
[0078] This application achieves full-cycle quantitative analysis of controllers' cognitive state, decision-making process, and effectiveness by integrating multi-source sensor data and artificial intelligence technology. Through the above steps, this application constructs an automated method capable of real-time assessment of controllers' working status. This method can accurately capture features related to cognitive load based on multimodal physiological data such as eye movement, electrodermal conductance, and electroencephalography, establishing a personalized cognitive baseline for controllers. By quantitatively assessing cognitive stress and concentration under different training subjects, the cognitive state of trainee controllers can be analyzed and training programs optimized, leading to targeted training and improved skill mastery efficiency. By monitoring relevant signals, it is also possible to dynamically warn of controllers' cognitive overload or fatigue, assisting in the rational scheduling of work tasks, mitigating the risk of human error, and providing scientific and accurate data support for the safe and efficient conduct of air traffic control work.
[0079] In one exemplary embodiment, this application also provides a device for quantitatively assessing controller cognitive load, comprising: The data acquisition module is used to collect multimodal physiological data of air traffic controllers in air traffic control scenarios, including eye movement data, raw skin conductance signals, and raw electroencephalogram (EEG) signals. The feature extraction and fusion module is used to extract features from multimodal physiological data and perform cross-modal feature fusion processing to obtain multimodal feature tensors; The network model construction module is used to construct a cognitive load dynamic regression network model; the input of the cognitive load dynamic regression network model is a multimodal feature tensor, and the output is the corresponding cognitive load index. A slice dataset construction module is used to construct a slice dataset for model training; the slice dataset includes time slices of multimodal feature tensors and corresponding slice labels; The model training module is used to train the cognitive load dynamic regression network model using the sliced dataset, calculate the loss function and optimize the parameters, and use the trained cognitive load dynamic regression network model as the dynamic cognitive load calculation model for subsequent use. The experimental module is used to input the multimodal feature tensor corresponding to the controller's current multimodal physiological data into the dynamic cognitive load calculation model during the experimental phase, so as to obtain the corresponding cognitive load index.
[0080] In the feature extraction and fusion stage, this application designs a multi-source sensor data fusion network. It employs a temporal resampling-attention alignment mechanism, effectively addressing the spatiotemporal alignment challenge of heterogeneous data from eye trackers, electrodermal sensors, and EEG devices in air traffic control scenarios. Specifically, for eye-tracking data, cubic spline interpolation is used to resample the sampling rate from 250Hz to 500Hz, preserving key features reflecting visual attention allocation, such as gaze coordinates and pupil diameter, to form... Dimensional eye movement temporal features This ensures matching with the EEG data in terms of sampling frequency. For the ductal skin signal, high-frequency noise is first removed by a Butterworth low-pass filter to obtain a standardized ductal skin sequence, which is then extracted by a 3-layer LSTM encoder. Characteristics of the skin conductance signal This fully captures the long-term dependence of skin conductance signals on physiological stress responses. EEG signals are extracted using complex Morlet wavelet transform. The energy of the 4-8Hz band, which is closely related to brain cognitive activity, was formed after energy calculation and dimensionality reduction. EEG signal characteristics of VI Finally, the cross-modal attention module weights and fuses the three features using a learnable weight matrix, outputting... Multimodal feature tensor Furthermore, the time alignment error is less than 2ms, providing a unified data foundation with high temporal consistency and multi-dimensional integration for subsequent analysis of controllers' cognitive state and decision-making process. This overcomes the shortcomings of traditional methods in handling heterogeneous data and allows physiological data from different modalities to work synergistically in cognitive state analysis.
[0081] In the network model construction phase, a cognitive load dynamic regression network model was designed. By introducing a scenario-adaptive weight learning mechanism, it can achieve millisecond-level cognitive load quantification and accurately capture the cognitive load changes of controllers at different times. This model will... Multimodal feature tensor The time-slicing layer divides the data into 2-second segments. This division avoids the loss of detail caused by excessively large windows, while preventing excessive noise introduced by excessively small windows, thus laying a reasonable temporal granularity foundation for subsequent feature extraction. Next, the spatiotemporal convolutional layer extracts local features through a 3-layer 1D-CNN. The settings for the convolutional kernel size and stride effectively capture local patterns in the time series. The attention gating layer calculates scene relevance weights based on scene encoding vectors such as traffic density and instruction complexity. This highlights key features relevant to the current scenario, enabling the model to adapt to the differences in load characteristics across different air traffic control scenarios.
[0082] In the segmented dataset construction phase, a new label allocation method was designed. Based on the NASA-TLX scale, and combined with instruction density, a mapping from global labels to time segment labels was achieved, providing high-quality regression labels for model training. The NASA-TLX scale obtains controllers' cognitive load scores from six core dimensions, including psychological and time requirements. The average value is taken and normalized to [0,1] as the global label, ensuring a combination of comprehensiveness, subjectivity, and objectivity in the labels. Based on this, the data is segmented into 2-second intervals to obtain... Each slice is analyzed by counting the number of control and emergency commands in each slice, assigning higher weight to emergency commands to calculate command density. This density serves as an indicator of each slice's contribution to the global cognitive load, reflecting both the differences in the importance of different commands and the objective level of task busyness during a given time period. Through weight normalization and a weighted average formula, global labels are mapped to each slice, ensuring that slice labels are positively correlated with command density and fall within the range of [0,1]. This solves the matching problem between global labels and local time slices, providing each slice with a corresponding accurate label. This establishes a reliable bridge between input and output during model training, ensuring the model learns the true mapping relationship between cognitive load and multi-source features.
[0083] Building upon this foundation, a model for assessing controller cognitive load was designed. Multimodal feature tensors extracted from controllers' multimodal physiological data, such as eye movements, electrodermal conductance, and electroencephalography (EEG), are input into a pre-trained dynamic cognitive load calculation model. This model rapidly outputs the corresponding cognitive load index, enabling real-time assessment of controllers' cognitive states at different task stages and in different scenarios. The method and apparatus in this application organically combine data fusion, model design, and label construction, forming a complete assessment loop. This helps overcome the shortcomings of traditional assessment methods, such as strong subjectivity and difficulty in quantification. It provides an operable and highly accurate technical means for scientifically assessing controller decision-making effectiveness and has broad application prospects.
[0084] In one exemplary embodiment, this application also provides a computer device, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces, and a communication interface. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface is connected to the system bus via the input / output interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the aforementioned controller cognitive load quantification assessment method.
[0085] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the aforementioned controller cognitive load quantification assessment method.
[0086] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned controller cognitive load quantification assessment method.
[0087] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to computer program instructions. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any reference to memory or other media in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0088] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0089] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0090] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for quantitatively assessing the cognitive load of air traffic controllers, characterized in that, include: In air traffic control scenarios, multimodal physiological data of controllers are collected, including eye movement data, raw electrodermal signals, and raw electroencephalogram (EEG) signals. Feature extraction and cross-modal feature fusion processing are performed on multimodal physiological data to obtain a multimodal feature tensor; A cognitive load dynamic regression network model is constructed; the input of the cognitive load dynamic regression network model is a multimodal feature tensor, and the output is the corresponding cognitive load index. Construct a slice dataset for model training; the slice dataset includes time slices of multimodal feature tensors and corresponding slice labels; A cognitive load dynamic regression network model was trained using a slice dataset. The loss function was calculated and the parameters were optimized. The trained cognitive load dynamic regression network model was then used as a dynamic cognitive load calculation model for subsequent use. During the experimental phase, the multimodal feature tensor corresponding to the controller's current multimodal physiological data is input into the dynamic cognitive load calculation model to obtain the corresponding cognitive load index.
2. The method for quantitatively assessing controller cognitive load according to claim 1, characterized in that, The collection of multimodal physiological data from controllers specifically includes: Eye trackers were used to collect controllers' eye movement data, including fixation coordinates, pupil diameter, and blink interval. Raw skin conductance signals from controllers are collected using skin conductance sensors. Raw electroencephalogram (EEG) signals of controllers were collected using EEG equipment.
3. The method for quantitatively assessing controller cognitive load according to claim 2, characterized in that, The process of feature extraction and cross-modal feature fusion of multimodal physiological data to obtain a multimodal feature tensor specifically includes: For eye-tracking data, cubic spline interpolation is used to resample the sampling rate from the eye tracker to the EEG device sampling rate, forming... Dimensional eye movement temporal features ;in For time points; The original electrodermal signal is first subjected to a Butterworth low-pass filter to remove high-frequency noise, resulting in... The standardized electrodermal signal sequence was then extracted using a 3-layer LSTM encoder. Characteristics of the skin conductance signal ; Wavelet transform is used to extract from the raw EEG signal. Band energy is formed after energy calculation and dimensionality reduction. EEG signal characteristics of VI ; Using a cross-modal attention module with a learnable weight matrix to... , , The three features are weighted and fused to output the result. Multimodal feature tensors of 1D.
4. The method for quantitatively assessing controller cognitive load according to claim 3, characterized in that, The constructed cognitive load dynamic regression network model specifically includes: a time slice layer, a spatiotemporal convolutional layer, an attention gating layer, and a dynamic regression layer; The time-slicing layer is used to process the input multimodal feature tensor according to... The time interval of seconds is divided into The first time slice, the second Each time slice is denoted as ;in , It is a positive integer; The spatiotemporal convolutional layer uses a 3-layer one-dimensional convolutional neural network to extract each time slice. Local patterns and features, to obtain Local eigenvectors of dimension ; The attention gating layer is used to calculate the scene relevance weights for air traffic control scenarios. ,use right Weighting is performed based on specific scene features to obtain each time slice. Scene weighted features ; The dynamic regression layer is used to process each time slice Scene weighted features Mapped to cognitive load index .
5. The method for quantitatively assessing controller cognitive load according to claim 4, characterized in that, The construction of the slice dataset for model training specifically includes: Obtain the controller's scale scores and normalize them to the [0,1] interval as global labels. ; Based on time slices within a unit of time instruction density Determine each time slice weight And match global tags using a weighted average. The weighted average formula is as follows: ; Each time slice is obtained by solving the weighted average formula. slice tags ; Slice the time and slice labels The slice dataset is constructed by storing the slices in a one-to-one correspondence.
6. The method for quantitatively assessing controller cognitive load according to claim 5, characterized in that, The loss function of the cognitive load dynamic regression network model is: ;in The number of time slices in a batch of training samples; It is the cognitive load index predicted by the model. ; This is the calculated loss value.
7. A device for quantitatively assessing the cognitive load of air traffic controllers, characterized in that, include: The data acquisition module is used to collect multimodal physiological data of air traffic controllers in air traffic control scenarios, including eye movement data, raw skin conductance signals, and raw electroencephalogram (EEG) signals. The feature extraction and fusion module is used to extract features from multimodal physiological data and perform cross-modal feature fusion processing to obtain multimodal feature tensors; The network model construction module is used to construct a cognitive load dynamic regression network model; the input of the cognitive load dynamic regression network model is a multimodal feature tensor, and the output is the corresponding cognitive load index. A slice dataset construction module is used to construct a slice dataset for model training; the slice dataset includes time slices of multimodal feature tensors and corresponding slice labels; The model training module is used to train the cognitive load dynamic regression network model using the sliced dataset, calculate the loss function and optimize the parameters, and use the trained cognitive load dynamic regression network model as the dynamic cognitive load calculation model for subsequent use. The experimental module is used to input the multimodal feature tensor corresponding to the controller's current multimodal physiological data into the dynamic cognitive load calculation model during the experimental phase, so as to obtain the corresponding cognitive load index.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the controller cognitive load quantification assessment method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the controller cognitive load quantification assessment method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the controller cognitive load quantification assessment method as described in any one of claims 1 to 6.