Pilot situational awareness judgment method and system based on decision-making layer fusion

Abstract

The present invention provides a pilot situational awareness assessment system based on decision-making layer fusion, comprising: a data acquisition module for collecting pilot personal information, eye movement device data, and electrocardiogram device data; a data interface module for receiving data transmitted by the data acquisition module and calculating conversion indicators; a determination module for performing real-time evaluation and comprehensive determination of the pilot's physiological characteristics; and an output report module for comprehensively outputting the information in the above modules and the online assessment results, including data such as the input personal information data, the selected situational awareness determination model, the pilot's situational awareness determination level, and the pilot's situational awareness determination results after decision-making fusion optimization. The present invention implements online assessment of a pilot's situational awareness status, capable of real-time determination of a pilot's situational awareness level in various mission environments, providing a basis for ensuring flight safety and reducing human error.

Description

Technical Field

[0001] The present invention relates to a pilot situation awareness judgment method and system based on decision-making layer fusion. Background Art

[0002] A pilot's situational awareness (SA) refers to their perception of various elements in the flight environment, their understanding of their significance, and their ability to predict their subsequent states during flight time and space. It is a crucial factor influencing pilot decision-making and operations. Real-time assessment of a pilot's comprehensive SA over time, and within specific time units, is crucial for reflecting and understanding their understanding of the various factors and conditions that influence their operations within a specific time period and context. This is crucial for reducing flight accidents caused by lapses in SA.

[0003] Currently, subjective evaluation, memory probe measurement, task performance measurement, and physiological measurement are the typical methods commonly used to measure pilot situational awareness. Compared with other measurement methods, physiological measurement has the advantages of objectivity, real-time performance, and limited task intrusion. Related research has shown that physiological parameters (such as eye movements and electrocardiogram) are closely correlated with pilot situational awareness, making physiological measurement an effective means of assessing pilot situational awareness during operations. However, such existing research has not achieved real-time identification and comprehensive evaluation of pilot situational awareness within a specific time unit or cumulative time by measuring pilot physiological parameters, nor has it achieved information fusion of multiple pilot situational awareness identification results based on physiological data. Summary of the Invention

[0004] In response to the above-mentioned problems in the prior art, the inventors have proposed a method and system for determining pilot situational awareness based on decision-making layer fusion. The decision-making layer fusion strategy is applied to a pilot situational awareness determination model based on a variety of physiological data during flight, thereby improving the real-time evaluation of the pilot's situational awareness status and the stability and generalization ability of the comprehensive evaluation model.

[0005] The present invention utilizes a pilot situational awareness assessment system based on a decision-level fusion strategy. This system collects physiological characteristic indicators during flight to conduct real-time dynamic and comprehensive assessments of the pilot's situational awareness status. The system includes a cumulative time-based situational awareness assessment model for comprehensive pilot status assessment and a unit-time situational awareness assessment model for dynamic time slices. Based on the application scenario requirements and characteristics, the system first collects and calculates eye movement characteristics, electrocardiogram characteristics, or a combination thereof, during the pilot's operation in real time. These characteristic data are then fed into a matching model in the system. Finally, based on the decision-level fusion strategy, the combined assessment results are subjected to a final fusion assessment to determine the pilot's real-time situational awareness status level. This system provides a dynamic assessment of the pilot's situational awareness level at the current time unit point and a comprehensive dynamic assessment over the cumulative time period.

[0006] The beneficial effects of the present invention are:

[0007] This invention overcomes the shortcomings of current real-time measurement and assessment technologies for pilots' situational awareness levels. By connecting an eye tracker and electrocardiogram (ECG) device via an interface module to read the pilot's physiological data in real time during flight, the system assesses the pilot's situational awareness level using a time-based discriminant SVM model based on eye movement features, ECG features, or all features, as well as a time-based discriminant SVM model. Furthermore, the system optimizes the situational awareness discriminant model using a decision-layer fusion strategy, improving model stability and generalization capabilities. This invention enables online assessment of a pilot's situational awareness level and can determine their situational awareness in real time across a variety of mission environments, providing a basis for ensuring flight safety and reducing human error.

[0008] Advantages of the present invention include:

[0009] The pilot situational awareness assessment system and method based on decision-level fusion according to the present invention does not rely on tedious questionnaires or post-analysis and evaluation. Instead, it can perform real-time measurement and comprehensive evaluation of pilot situational awareness levels during flight tests simply by collecting pilot eye movement data and electrocardiogram data during flight. The system is simple and feasible to use, and provides an optimized user interface and usage process.

[0010] The present invention uses a decision-making layer fusion strategy based on the discrimination of multiple physiological characteristic data, which makes the pilot situational awareness discrimination system have the advantages of multiple application scenarios, good stability, strong generalization ability, and high accuracy;

[0011] It is suitable for online monitoring and comprehensive status control of pilots' situational awareness during flight missions in various scenarios such as civil airliners, trainer aircraft, and simulation training, thereby helping to improve aviation safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 2 is a schematic diagram of a pilot situational awareness determination method based on decision-layer fusion according to an embodiment of the present invention.

[0013] Figure 2 1 is an architectural diagram of a pilot situational awareness determination system based on decision-layer fusion according to an embodiment of the present invention.

[0014] Figure 3 The present invention is a pilot information collection interface of a pilot situational awareness judgment system based on decision-making layer fusion according to an embodiment of the present invention.

[0015] FIG4 is an online evaluation interface of a pilot situational awareness determination system based on decision-layer fusion according to an embodiment of the present invention.

[0016] Figure 5 This is an output evaluation report result of a pilot situational awareness judgment system based on decision-layer fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] 1. Overall situation

[0018] In the pilot situational awareness judgment system and method based on decision-level fusion according to the present invention, a situational awareness decision fusion model based on the pilot's eye movement characteristics and electrocardiogram characteristics in a variety of feature combination judgment modes is established and adopted for real-time collection of applications and different usage requirements of different scenes and / or different equipment, which includes a situational awareness cumulative time judgment model for pilot comprehensive status evaluation and a situational awareness unit time judgment model for pilot dynamic time slicing. According to one embodiment of the present invention, with a 30-second time unit as the calculation interval, 30 seconds of eye movement characteristics and electrocardiogram characteristics data are collected and calculated in real time and input into the judgment system. Every 30 seconds, the calculation, judgment and output of the pilot's situational awareness status level in the current 30-second time unit and the comprehensive situational awareness status level from the beginning to the current cumulative time period are simultaneously updated and output, thereby realizing a dynamic evaluation of the pilot's situational awareness level at the current time unit point during the flight and a comprehensive dynamic evaluation within the cumulative time period. Figure 1 shown.

[0019] 2 Architecture design of pilot situational awareness judgment system based on decision-making layer fusion

[0020] like Figure 2As shown, according to one embodiment of the present invention, a pilot situational awareness assessment system and method based on decision-making layer fusion completes the real-time reading of pilot eye movement and electrocardiogram data and other physiological signals, personnel information input and setting, parameter setting, etc., and performs multi-level pilot situational awareness identification and assessment. The pilot situational awareness assessment system based on decision-making layer fusion includes four parts: a data interface module, a data acquisition module, a determination module, and an output report module. Figure 2 shown. Specifically:

[0021] (1) Data acquisition module

[0022] This module collects three types of pilot data: personal information, eye movement equipment, and electrocardiogram equipment.

[0023] Personal information collected includes pilot age, height, flight time, test site, number of experiments, and other recorded information, which is input through the user interface;

[0024] Eye movement device acquisition: Wear an eye movement device to read the original eye movement data in real time, such as the original spatial position coordinates of the eye movement, blink marks, fixation marks, saccade marks, etc.

[0025] ECG device acquisition reads raw ECG data in real time by wearing an ECG device, such as heart rate, raw ECG waveform data, etc.

[0026] (2)Data interface module

[0027] This module is used to receive data transmitted by the data acquisition module and convert the data into indicator calculations. It contains three sub-modules: data transceiver interface sub-module, data storage sub-module, and data conversion sub-module.

[0028] in:

[0029] Data transceiver interface submodule: First, set the UDP port number and host address in the eye tracking device and ECG device. Then, based on the C++ platform, establish UDP data communication between the data interface module and the eye tracking device and ECG device through the data transceiver interface. Read the raw data collected by the eye tracking device at a frequency of 60Hz, and read the raw ECG data at a frequency of 1Hz.

[0030] Data storage submodule: stores the real-time collected eye movement and ECG raw data of different frequencies in the memory space;

[0031] Data conversion module: The calculation of eye movement characteristics is somewhat non-real-time. With a calculation interval of 30 seconds, the raw eye movement and ECG data are converted into indicators such as fixation rate, saccade rate, pupil diameter, and average fixation time, as well as ECG characteristic indicators such as average heart rate, average RR interval, standardized low-frequency power, and standardized high-frequency power.

[0032] (3) Discrimination module

[0033] This module is used to perform real-time evaluation and comprehensive discrimination of the pilot's physiological characteristics, and includes an eye movement feature preprocessing submodule, an electrocardiogram feature preprocessing submodule, a feature selection submodule, and a discrimination classifier submodule.

[0034] The eye movement feature preprocessing submodule reads the eye movement feature indicators (fixation rate, saccade rate, pupil diameter, average fixation time, etc.) sent by the data interface module and performs normalization processing such as removing null values ​​or outliers, as well as preprocessing operations such as standardization.

[0035] The ECG feature preprocessing submodule reads the ECG feature indicators (average heart rate, average RR interval, normalized low-frequency power, normalized high-frequency power, etc.) sent by the data interface module, and performs normalization processing such as removing null values ​​or abnormal values, as well as preprocessing operations such as normalization processing;

[0036] Feature selection submodule, which selects calculation based on full feature (ECG and eye movement) model / eye movement feature model / ECG feature model;

[0037] The discriminant classifier submodule includes a time-unit discriminant model for situational awareness and a time-accumulated discriminant model for situational awareness. These two models are trained using two different data methods. This submodule reads the pilot's physiological characteristic data in real time and calls the corresponding discriminant model to perform time-unit situational awareness discrimination and time-accumulated situational awareness discrimination, respectively. The two discriminant models share the same logic, differing in that the time-unit discriminant model uses eye movement and electrocardiogram data at 30-second intervals during dynamic flight as input, focusing on evaluating changes in situational awareness within a specific time period during dynamic flight. The time-accumulated discriminant model uses eye movement and electrocardiogram data within a cumulative time period as input, focusing on evaluating the comprehensive state of situational awareness changes due to the current cumulative effect during flight. The model discrimination logic is as follows: you can choose to establish a discrimination model with full features (eye movement and ECG) / eye movement features / ECG features. Based on the discrimination results of different models, you can choose one of seven different model discrimination result combinations: eye movement feature model, ECG feature model, full feature model, eye movement feature + ECG feature model, eye movement feature + full feature model, ECG feature + full feature model, eye movement feature + ECG feature + full feature model, and eye movement feature + ECG feature + full feature model. Based on the different model discrimination result combinations, logistic regression decision fusion optimization is performed to obtain the pilot situational awareness discrimination result based on decision-level fusion. The logic of the seven different models is illustrated as follows: if a single model is selected, the discrimination result is the independent discrimination result of the model. For example, after selecting the eye movement feature model, the final discrimination result is the decision based on the independent discrimination result of the eye movement feature model; if a combined model is selected, the discrimination result is the result of decision fusion optimization of the discrimination results of multiple different models. For example, if the eye movement feature + electrocardiogram feature model is selected, the two independent discrimination results of the eye movement feature result and the electrocardiogram feature result are decision fused and optimized to obtain the final result.

[0038] (4) “Output Report Module”

[0039] Comprehensively output pilot (subject) information and online assessment results from the above modules, including basic personnel information, selected assessment models, pilot situational awareness status assessment levels, and optimized results after decision fusion. Output results support exporting data lists and status diagrams in various formats, such as PNG and txt, by selecting a specified path.

[0040] 3 User interaction interface of pilot situational awareness judgment system based on decision-making layer fusion

[0041] The user interface of the pilot situational awareness judgment system based on decision-making layer fusion includes three application modules: pilot information collection, online evaluation interface, and output evaluation report interface. (1) Pilot information collection interface of the pilot situational awareness judgment system based on decision-making layer fusion

[0042] This interface supports inputting pilot's personal flight test information, which can be input by selecting drop-down menu or text box style, including: name, age, height, place of origin, flight time, number of experiments, experimental site and other information, such as Figure 3 shown.

[0043] (2) Online evaluation interface of pilot situational awareness judgment system based on decision-making layer fusion

[0044] This interface is the main interface for online assessment and observation, as shown in Figure 4. First, click the "Parameter Settings" button on this interface to enter the system parameter setting window, as shown in Figure 4(a). Then, enter the device address (IP address) of the computer where the system is located in the "Device Address" field in the parameter setting window. Then, enter the port numbers of the eye tracking device and ECG device in the "Port Selection" field to establish a connection with the eye tracking device and ECG device. Then, select the full feature / eye tracking feature / ECG feature model used for discrimination (multiple selections are allowed, with a total of seven combinations). Select the number of features collected by the ECG and eye tracking devices required for the online assessment of the pilot situational awareness discrimination system as needed. Then, select the situational awareness time unit discrimination model or the situational awareness time cumulative discrimination model in the "Discrimination Model Selection" field. Finally, click "Confirm and Submit" to complete the parameter setting.

[0045] After completing the parameter settings, click the "Run" button to begin the online evaluation. The result display interface is shown in Figure 4(b). The interface can display some key ECG indicators, eye movement indicators, and a graphical area showing the changes in the pilot's situational awareness status over time based on different feature models in real time. The real-time situational awareness status graph updates the real-time evaluation results of the situational awareness status time unit every 30 seconds, with green indicating "high", yellow indicating "medium", and red indicating "low". After the evaluation is completed, the system can display the cumulative judgment results of the pilot's situational awareness time during the operation and the decision fusion optimization results.

[0046] (3) Output evaluation report result interface of pilot situational awareness judgment system based on decision-making layer fusion

[0047] This interface is used to display the output evaluation results of the pilot situational awareness online evaluation system, including the input pilot personal information, physiological characteristics monitoring results, situational awareness real-time evaluation results and situational awareness judgment results after decision fusion optimization. And through the "output report" function, the evaluation result data and visual graphics can be output according to the specified path, such as Figure 5 shown.

Claims

1. Pilot situational awareness judgment system based on decision-making layer fusion, characterized by include: The data acquisition module is used to collect pilots' personal information data, eye movement device data, and electrocardiogram device data, including: Personal information data including pilot age, height, flight time, test site, and number of experiments are input through the user interface; Eye tracking device data includes raw eye tracking data read in real time by wearing an eye tracking device; ECG device data includes raw ECG data read in real time by wearing an ECG device; the data interface module is used to receive data transmitted by the data acquisition module and calculate conversion indicators. It includes a data transceiver interface submodule, a data storage submodule, and a data conversion submodule, wherein: The data transceiver interface submodule is used to first set the UDP port number and host address in the eye tracking device and ECG device. Then, based on the C++ platform, it establishes UDP data communication between the data interface module and the eye tracking device and ECG device through the data transceiver interface. It reads the raw data collected by the eye tracking device at a frequency of 60Hz and the raw ECG data at a frequency of 1Hz. The data storage submodule is used to store the real-time collected eye movement and ECG raw data of different frequencies in the memory space; The data conversion submodule is used to calculate the eye movement characteristics, which is somewhat non-real-time. It uses a 30-second calculation interval to convert the raw eye movement and ECG data into indicators such as fixation rate, saccade rate, pupil diameter, and average fixation time, as well as ECG characteristic indicators such as average heart rate, average RR interval, normalized low-frequency power, and normalized high-frequency power. The discrimination module is used to perform real-time evaluation and comprehensive discrimination of the pilot's physiological characteristics. It includes: eye movement feature preprocessing submodule, electrocardiogram feature preprocessing submodule, feature selection submodule, and discrimination classifier submodule, among which: The eye movement feature preprocessing submodule is used to read the eye movement feature indicators including fixation rate, saccade rate, pupil diameter, and average fixation time sent by the data interface module, and perform normalization processing including removal of null values ​​or abnormal values ​​and preprocessing operations including standardization processing; The ECG feature preprocessing submodule is used to read the ECG feature indicators including average heart rate, average RR interval, normalized low-frequency power, and normalized high-frequency power sent from the data interface module, and perform normalization processing including removal of null values ​​and / or abnormal values ​​and preprocessing operations including normalization processing; A feature selection submodule, for selecting calculations based on a full feature model and / or an eye movement feature model and / or an electrocardiogram feature model; The discriminant classifier submodule includes a situational awareness time unit discrimination model and a situational awareness time accumulation discrimination model, and performs data training based on the two models, including: reading the pilot's physiological characteristic data in real time, calling the corresponding models in the situational awareness time unit discrimination model and the situational awareness time accumulation discrimination model to perform time unit situational awareness discrimination and / or time accumulation situational awareness discrimination, wherein: The discrimination logic of the situational awareness time unit discrimination model and the situational awareness time accumulation discrimination model is consistent. The difference is that the time unit discrimination model uses the eye movement and electrocardiogram data of 30 seconds intervals during dynamic flight as input, and focuses on evaluating the situational awareness changes within a certain time period during dynamic flight, while the time accumulation discrimination model uses the eye movement and electrocardiogram data within the cumulative time period as input, and focuses on evaluating the comprehensive state changes of the situational awareness of the current cumulative effect during flight. The model discrimination logic of the situational awareness time unit discrimination model and the situational awareness time accumulation discrimination model is as follows: select to establish a discrimination model based on full features and / or eye movement features and / or ECG features, and based on different model selections, select one of seven different model discrimination result combinations: eye movement feature model, ECG feature model, full feature model, eye movement feature + ECG feature model, eye movement feature + full feature model, ECG feature + full feature model, eye movement feature + ECG feature + full feature model, and Based on the combination of different model discrimination results, logistic regression decision fusion optimization is performed to obtain the pilot situational awareness discrimination results based on decision-level fusion; The output report module is used to comprehensively output the information in the above modules and the online evaluation results, including the entered personal information data, the selected situational awareness discrimination model, the pilot's situational awareness discrimination level, and the pilot's situational awareness discrimination result data after logistic regression decision fusion optimization.

2. The pilot situational awareness determination system based on decision-making layer fusion according to claim 1, characterized in that: The raw eye movement data includes the raw eye movement spatial position coordinates, blink marks, fixation marks, and saccade marks; The original ECG data includes heart rate and ECG raw waveform data.

3. The pilot situational awareness assessment system based on decision-making layer fusion according to claim 1, characterized in that: The output report module supports selecting a specified path to export data lists and / or status diagrams in multiple formats including png and txt.

4. The pilot situational awareness identification method based on decision-making layer fusion is characterized by include: Data collection steps include collecting pilots' personal information data, eye tracking device data, and electrocardiogram device data, including: Personal information data including pilot age, height, flight time, test site, and number of experiments are input through the user interface; Eye tracking device data includes raw eye tracking data read in real time by wearing an eye tracking device; ECG device data includes raw ECG data read in real time by wearing an ECG device; The data interface step is used to receive data transmitted by the data acquisition module and perform conversion index calculation, which includes: First, set the UDP port number and host address in the eye tracking device and ECG device. Then, based on the C++ platform, establish UDP data communication between the data interface module and the eye tracking device and ECG device through the data transceiver interface. Read the raw data collected by the eye tracking device at a frequency of 60Hz, and read the raw ECG data at a frequency of 1Hz. The real-time collected eye movement and ECG raw data of different frequencies are stored in the memory space; The calculation of eye movement characteristics is somewhat non-real-time. With a calculation interval of 30 seconds, the raw eye movement and ECG data are converted into indicators, including eye movement characteristic indicators such as fixation rate, saccade rate, pupil diameter, and average fixation time, and ECG characteristic indicators such as average heart rate, average RR interval, standardized low-frequency power, and standardized high-frequency power. The discrimination steps are to conduct real-time evaluation and comprehensive discrimination of the pilot's physiological characteristics, which include: Reading the eye movement characteristic indicators including fixation rate, saccade rate, pupil diameter, and average fixation time sent by the data interface module, and performing normalization processing including removal of null values ​​or abnormal values ​​and preprocessing operations including standardization processing; Reading the ECG characteristic indicators sent by the data interface module, including the average heart rate, average RR interval, normalized low-frequency power, and normalized high-frequency power, and performing normalization processing including removal of null values ​​and / or abnormal values ​​and preprocessing operations including normalization processing; Choose to perform calculation based on the full feature model and / or the eye movement feature model and / or the electrocardiogram feature model; Using a discriminant classifier including a situational awareness time unit discrimination model and a situational awareness time accumulation discrimination model, data training based on the two models is performed, including: reading the pilot's physiological characteristic data in real time, calling the corresponding models in the situational awareness time unit discrimination model and the situational awareness time accumulation discrimination model to perform time unit situational awareness discrimination and / or time accumulation situational awareness discrimination, wherein: The discrimination logic of the situational awareness time unit discrimination model and the situational awareness time accumulation discrimination model is consistent. The difference is that the time unit discrimination model uses the eye movement and electrocardiogram data of 30 seconds intervals during dynamic flight as input, and focuses on evaluating the situational awareness changes within a certain time period during dynamic flight, while the time accumulation discrimination model uses the eye movement and electrocardiogram data within the cumulative time period as input, and focuses on evaluating the comprehensive state changes of the situational awareness of the current cumulative effect during flight. The model discrimination logic of the situational awareness time unit discrimination model and the situational awareness time accumulation discrimination model is as follows: select to establish a discrimination model based on full features and / or eye movement features and / or ECG features, and based on different model selections, select one of the following seven different model discrimination result combinations: eye movement feature model, ECG feature model, full feature model, eye movement feature + ECG feature model, eye movement feature + full feature model, ECG feature + full feature model, eye movement feature + ECG feature + full feature model, and Based on the combination of different model discrimination results, logistic regression decision fusion optimization is performed to obtain the pilot situational awareness discrimination results based on decision-making layer fusion. The output report step comprehensively outputs the information in the above modules and the online evaluation results, including the entered personal information data, the selected situational awareness discrimination model, the pilot's situational awareness discrimination level, and the pilot's situational awareness discrimination result data after logistic regression decision fusion optimization.

5. The pilot situational awareness determination method based on decision-making layer fusion according to claim 4 is characterized by: The raw eye movement data includes the raw eye movement spatial position coordinates, blink marks, fixation marks, and saccade marks; The original ECG data includes heart rate and ECG raw waveform data.

6. The pilot situational awareness determination method based on decision-making layer fusion according to claim 4 is characterized by: The output report step supports selecting a specified path to export data lists and / or status diagrams in multiple formats including png and txt.

7. A computer-readable storage medium storing a computer program, the computer program enabling a processor to execute the method according to any one of claims 4 to 6.

Images