Cockpit integrated warning system based on decision fusion

Abstract

A comprehensive cockpit alarm system based on decision fusion includes: a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module, and an alarm module. The data acquisition module collects pilot operation data, attitude data, and physiological data. The data synchronization and preprocessing module performs time synchronization based on the pilot operation data, attitude data, and physiological data, and performs noise reduction and feature extraction. The model training and prediction module trains and predicts the input data features using a Support Vector Machine (SVM) model and a Long Short-Term Memory (LSTM) model, and performs decision-level fusion of the prediction results from these two models. The alarm module, based on the prediction results and the current flight phase, issues corresponding alarms according to the alarm threshold and alarm priority. This invention, by integrating pilot operation data, attitude data, and physiological data, utilizes a decision fusion algorithm and a multi-layer alarm mechanism to achieve real-time monitoring of pilot behavior and promptly issue alarms when abnormal behavior occurs.

Description

Technical Field

[0001] This invention relates to a technology in the field of civil aviation, specifically a cockpit integrated warning system based on decision fusion. Background Technology

[0002] Currently, pilot behavior monitoring mainly relies on single monitoring methods (such as physiological signals or attitude monitoring), which cannot comprehensively capture the pilot's overall state, thus limiting the effectiveness of identification and monitoring. Meanwhile, existing alarm systems often issue alerts after abnormal behavior occurs. When pilots are fatigued, distracted, or overloaded, traditional systems cannot promptly identify and respond to changes in pilot behavior, resulting in insufficient accuracy and timeliness of alarms, and failing to provide effective support for pilots and crew. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies that fail to fully integrate comprehensive information from pilot operational data, attitude data, and physiological data, and lack an alarm mechanism. It proposes a cockpit integrated alarm system based on decision fusion. By integrating pilot operational data, attitude data, and physiological data, and utilizing decision fusion algorithms and multi-layer alarm mechanisms, the system achieves real-time monitoring of pilot behavior and issues timely alarms when abnormal behavior occurs.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a comprehensive cockpit warning system based on decision fusion, comprising: a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module, and an alarm module, wherein: the data acquisition module acquires pilot operation data, attitude data, and physiological data; the data synchronization and preprocessing module performs time synchronization based on the pilot operation data, attitude data, and physiological data, and performs noise cancellation and feature extraction; the model training and prediction module trains and predicts the input data features using a support vector machine (SVM) model and a long short-term memory network (LSTM) model, and performs decision-level fusion of the prediction results of these two types of models; the alarm module, based on the prediction results and the current flight phase, issues corresponding alarms according to the alarm threshold judgment results and alarm priorities.

[0006] Technical effect

[0007] This invention, based on the comprehensive integration of pilot operational data, attitude data, and physiological data to reflect pilot status and combined with a multi-layered alarm mechanism, can dynamically adjust alarm strategies according to the different degrees and priorities of different pilot behavioral states, combined with flight phases. It overcomes the limitations of a single data source, comprehensively captures the multi-dimensional behavioral states of pilots, and provides more comprehensive and accurate information on pilot behavioral states. At the same time, through a dynamic alarm mechanism, it automatically adjusts alarm levels and response strategies according to the different degrees of pilot behavioral states, thereby providing timely, accurate, and personalized alarm information, avoiding excessive or insufficient alarms, and effectively improving the effectiveness of pilot flight safety management. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the system of the present invention;

[0009] Figure 2 This is a data processing flowchart for an example.

[0010] Figure 3 The following is a flowchart of the alarm process for an example. Detailed Implementation

[0011] like Figure 1 As shown, this embodiment illustrates a cockpit integrated warning system based on decision fusion, comprising: a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module, and an alarm module. Specifically: the data acquisition module collects pilot operation data, attitude data, and physiological data; the data synchronization and preprocessing module performs time synchronization based on the pilot operation data, attitude data, and physiological data, and performs noise cancellation and feature extraction; the model training and prediction module trains and predicts the input data features using a Support Vector Machine (SVM) model and a Long Short-Term Memory (LSTM) network model, and performs decision-level fusion of the prediction results from these two models; the alarm module, based on the prediction results and the current flight phase, issues corresponding alarms according to the alarm threshold judgment result and alarm priority.

[0012] The pilot operation data includes: response time of critical commands and alarms, operation time intervals, and compliance with standard operating procedures.

[0013] The pilot attitude data includes the pilot's sitting posture, center of gravity changes, and contact pressure distribution.

[0014] The pilot's attitude data is collected through a pressure sensor array, which is arranged in multiple parts of the pilot's seat, including the back, seat cushion, and lumbar support area.

[0015] The pilot's physiological data includes electroencephalogram (EGG), electrocardiogram (ECG), and electrical conductance of the skin (EDA). Specifically, the EGG signal is collected by an EGG sensor that is in direct contact with the scalp; the ECG signal is collected by a chest strap patch-type heart rate sensor; and the EDA signal is collected by a skin conductance sensor worn on the wrist.

[0016] like Figure 2 As shown, the data synchronization and preprocessing module includes: a data synchronization unit, a noise cancellation unit, and a feature extraction unit. Specifically: the data synchronization unit aligns all data from different sensors to a unified time axis using linear interpolation; the noise cancellation unit employs Kalman filters and wavelet transforms for denoising pilot attitude data and physiological data, respectively; and the feature extraction unit extracts features x from the pilot's operational data. operation Attitude data features x posture and physiological data characteristics x physiology .

[0017] The operational data features include: response time of key instructions and alarms, operation time interval, and compliance with standard operating procedures. The resulting feature vector is: x operation =[Tr mean ,Tr max ,Tr min ,Td mean ,Td std SOP rate ], where: Tr mean Tr max and Tr min These are the average response time, maximum response time, and minimum response time, respectively; Td mean and Td std These represent the mean difference and standard deviation between the operation time interval and the actual operation time interval, respectively; SOP rate This indicates the degree to which the executed operation matches the standard procedure.

[0018] The posture data features include: pressure value, pressure change rate, posture stability, center of gravity position, center of gravity change rate, and contact area of ​​each sensor in the pressure sensor array. The resulting feature vector is: x posture =[P,ΔP,P var ,X,Y,Z,ΔX,ΔY,ΔZ,A], where: P(t)=[p1,p2,…,p n [p] represents the pressure value from the sensor; i This represents the pressure value of the i-th sensor; ΔP = [Δp1, Δp2, ..., Δp n ] represents the pressure change rate of the sensor; Δp iThe pressure change rate of the i-th sensor is represented; the stability of the pilot's posture is determined by the variance P of the pressure distribution. var Measurement; X, Y, and Z are the coordinates of the center of gravity; ΔX, ΔY, and ΔZ represent the rates of change of the center of gravity at time t; A is the contact area.

[0019] The physiological data features mentioned include: EEG signal features X ECG ECG signal characteristics X ECG and skin conductance signal characteristics X EDA The resulting eigenvector is x physiology =[X EGG ,X ECG ,X EDA ].

[0020] The aforementioned EEG signal feature X EGG =[P delta ,P theta ,P alpha ,P beta ,P gamma ,PR,H,μ,T], where: P delta P theta P alpha P beta and P gamma , , represent the power spectral density of the EGG signal in different frequency bands; PR is the relative proportion of power in the Alpha and Beta bands; H represents the entropy of the EEG signal; μ is the mean of the EEG signal; T represents the time-domain characteristics of the EGG signal, including but not limited to mean, standard deviation, variance, skewness, kurtosis, and root mean square.

[0021] The electrocardiogram signal feature X ECG = [HR,SNDD,RMSSD,LF / HF], where: HR represents heart rate; SNDD represents the standard deviation of the RR interval (the time interval between adjacent heartbeats); RMSSD is the root mean square value of the difference between adjacent RR intervals; LF / HF is the low-frequency / high-frequency power ratio of the heart rate signal.

[0022] The described skin conductance signal feature X EDA =[SCL mean SCL std SCR N SCR mean SCR N_total ], where: SCCL mean and SCL std These represent the mean and standard deviation of basal skin conductance level (SCL); SCR N Indicates the number of peak values ​​in the skin conductance response (SCR); SCR mean Indicates the average peak amplitude of SCR; SCRN_total This represents the total skin conductance response count.

[0023] The model training and prediction module includes: a Support Vector Machine (SVM) model, a Long Short-Term Memory (LSTM) network model, and a decision fusion model. Specifically: the SVM model is trained based on pilot operation data to predict the pilot's behavioral state; the LSTM model is trained based on fused attitude data and physiological data to predict the pilot's behavioral state; and the decision fusion model uses the Stacking algorithm to perform decision-level fusion of the prediction results of the SVM model and the LSTM model.

[0024] The pilot's behavioral states include fatigue, distraction, and excessive workload. Among them, fatigue is divided into mild fatigue and severe fatigue; excessive workload is divided into three states: slightly high workload, excessive workload, and unbearable workload.

[0025] The SVM model described uses a supervised learning-based support vector regression (SVR) algorithm, with an RBF kernel function: Where: K(x) i ,x j ) represents the sample point x i and x j The kernel function value between; ||x i -x j || 2 σ represents the square of the Euclidean distance between sample points; σ is the kernel width parameter, the value of which is determined through cross-validation. In this embodiment, σ is set to 1.

[0026] The LSTM model described is an LSTM incorporating an attention mechanism, and it uses a sliding time window to input pose and physiological data. The time window size is set to W. At each time step t, data from the previous W time steps is used as input. In this embodiment, the step size is 1 second, and W is set to 60.

[0027] The Stacking algorithm employs a weighted average fusion method to combine the prediction results of the SVM and LSTM models, and uses the fused prediction result as the final pilot behavior state identification result (fatigue, distraction, or excessive workload). Specifically: Where: w SVM and w LSTM represent the weights of the SVM and LSTM models, respectively, and their values ​​are determined through cross-validation. In this embodiment, w SVM and w LSTM The values ​​are 0.6 and 0.4 respectively; This represents the prediction result of the SVM model; This represents the prediction result of the LSTM model; This is the final fusion prediction result.

[0028] The alarm module uses the Softmax function to convert the fused prediction results into class probabilities, specifically: Where: p i This represents the predicted probability of category i (fatigue, distraction, and excessive workload); This is the value corresponding to category i in the fused output.

[0029] like Figure 3 As shown, the alarm module determines the alarm threshold based on the monitored pilot behavior and flight phase, and generates an alarm reminder to the pilot based on the alarm priority.

[0030] Considering that pilots are fatigued, distracted, or under excessive workload, their vision is either blocked (fatigue, distraction) or overloaded (excessive workload), making it difficult for them to effectively receive information from warning lights or displays. Therefore, pilot alarms will primarily rely on auditory (voice) or tactile (vibration) cues, supplemented by visual cues (warning lights), with alarm information simultaneously displayed on the screen. Specific alarm rules are as follows:

[0031]

[0032] Where: θ fatigue1 and θ fatigue2 These are the alarm thresholds for mild fatigue and severe fatigue, respectively. In this embodiment, θ fatigue1 and θ fatigue2 They are 0.5 and 0.8 respectively; θ distraction In this embodiment, θ is the distraction alarm threshold. distraction Set to 0.5; θ workload1 θ workload2 and θ workload3 These are the alarm thresholds for slightly high workload, slightly high workload, and unbearable workload, respectively. In this embodiment, θ workload1 θ workload2 and θ workload3 Set them to 0.5, 0.7 and 0.9 respectively.

[0033] When two or more alarms exist simultaneously, the alarm priority logic rules are as follows:

[0034] 1) If fatigue and distraction coexist: If distraction is detected and fatigue exists, fatigue alarm will be output first, i.e., the alarm priority logic is: severe fatigue > mild fatigue > distraction;

[0035] 2) If fatigue and workload coexist: In this case, fatigue is caused by excessive workload. In this case, the workload alarm will be output first. That is, the alarm priority logic is: workload cannot be borne > workload is too high > workload is slightly high > fatigue.

[0036] The alarm thresholds are adjusted according to different flight phases, and the specific rules are as follows:

[0037] 1) During takeoff and landing, increase the sensitivity of workload alarms, i.e., lower the workload alarm threshold;

[0038] 2) During the cruise phase, increase the sensitivity of distraction and fatigue alarms, i.e., lower the threshold for distraction and fatigue alarms.

[0039] In this embodiment, the reduction in the alarm threshold for the above states is 0.1.

[0040] This system integrates pilot operational, attitude, and physiological data, fully leveraging the complementarity of different data types. Employing a decision fusion-based algorithm, it improves the accuracy and comprehensiveness of pilot behavior monitoring and alerts. Compared to traditional "post-event" alerts for abnormal events, this system analyzes pilot behavior in real time, quickly identifying abnormal behaviors and issuing alerts. This ensures pilots receive timely information and take action, reducing flight risks. Furthermore, by introducing multi-level alerts that consider the varying degrees and priorities of pilot fatigue, distraction, and excessive workload, the system achieves more precise alerts. In addition, the system can adjust alert thresholds and sensitivity based on flight phases, avoiding over- or under-alertness and enhancing its applicability and usability.

[0041] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A cockpit integrated alarm system based on decision fusion, characterized in that, include: The system comprises a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module, and an alarm module. Specifically: the data acquisition module collects pilot operation data, attitude data, and physiological data; the data synchronization and preprocessing module performs time synchronization based on the pilot operation data, attitude data, and physiological data, and performs noise reduction and feature extraction; the model training and prediction module trains and predicts the input data features using a support vector machine model and a long short-term memory network model, and performs decision-level fusion of the prediction results from these two models; the alarm module, based on the prediction results and the current flight phase, determines the alarm threshold and alarm priority, and issues corresponding alarms. The pilot operation data includes: response time of critical commands and alarms, operation time intervals, and compliance with standard operating procedures; The pilot attitude data includes the pilot's sitting posture, center of gravity changes, and contact pressure distribution; The pilot attitude data is collected through a pressure sensor array, which is arranged in the back, seat cushion, and lumbar support area of ​​the pilot's seat. The pilot's physiological data includes electroencephalogram (EGG), electrocardiogram (ECG), and electrical conductance analysis (EDA).

2. The cockpit integrated alarm system based on decision fusion according to claim 1, characterized in that, The data synchronization and preprocessing module includes a data synchronization unit, a noise cancellation unit, and a feature extraction unit. Specifically: the data synchronization unit aligns all data from different sensors to a unified time axis using linear interpolation; the noise cancellation unit employs Kalman filters and wavelet transforms for denoising pilot attitude and physiological data, respectively; and the feature extraction unit extracts features from the pilot's operational data. Attitude data features and physiological data characteristics .

3. The cockpit integrated alarm system based on decision fusion according to claim 2, characterized in that, The operational data features include: response time of key instructions and alarms, operation time interval, and compliance with standard operating procedures. The resulting feature vector is: ,in: , and These are the average response time, maximum response time, and minimum response time, respectively. and These represent the mean difference and standard deviation between the operation time interval and the actual operation time interval, respectively. This indicates the degree to which the executed operation matches the standard procedure.

4. The cockpit integrated alarm system based on decision fusion according to claim 2, characterized in that, The posture data features include: pressure value, pressure change rate, sitting posture stability, center of gravity position, center of gravity change rate, and contact area of ​​each sensor in the pressure sensor array. The resulting feature vector is: ,in: The pressure value of the sensor; Indicates the first Pressure values ​​from each sensor; The pressure change rate of the sensor; Indicates the first The pressure change rate of each sensor; the stability of the pilot's posture is determined by the variance of the pressure distribution. measure; , and These are the coordinates of the center of gravity; , and They represent in The rate of change of the center of gravity at any given time; Contact area; The physiological data characteristics mentioned include: electroencephalogram (EEG) signal characteristics. ECG signal characteristics and skin conductance signal characteristics The resulting feature vector is ; The aforementioned EEG signal characteristics ,in: , , , and These represent the power spectral density of the EGG signal in different frequency bands; This represents the relative proportion of power in the Alpha and Beta bands. This represents the entropy of the EEG signal; The mean value of the EEG signal; The time-domain characteristics of EGG signals include: mean, standard deviation, variance, skewness, kurtosis, or root mean square. The electrocardiogram signal characteristics ,in: Indicates heart rate; This represents the standard deviation of the time interval between adjacent heartbeats; The root mean square value of the difference between adjacent RR intervals; This represents the low-frequency / high-frequency power ratio of the heart rate signal. The aforementioned skin conductance signal characteristics ,in: and The mean and standard deviation of basal skin conductance level (SCL) are represented, respectively. Indicates the number of peak values ​​in the skin conductance response (SCR); Indicates the average peak amplitude of SCR; This represents the total skin conductance response count.

5. The cockpit integrated alarm system based on decision fusion according to claim 1, characterized in that, The model training and prediction module includes: a Support Vector Machine (SVM) model, a Long Short-Term Memory (LSTM) network model, and a decision fusion model. Specifically: the SVM model is trained based on pilot operation data to predict pilot behavior; the LSTM model is trained based on fused attitude and physiological data to predict pilot behavior; and the decision fusion model uses a Stacking algorithm to perform decision-level fusion of the prediction results from the SVM and LSTM models. The pilot's behavioral states include fatigue, distraction, and excessive workload. Fatigue is divided into mild fatigue and severe fatigue. Excessive workload is divided into three states: slightly high workload, excessive workload, and unbearable workload.

6. The cockpit integrated alarm system based on decision fusion according to claim 5, characterized in that, The SVM model described uses a supervised learning-based support vector regression (SVR) algorithm, with the RBF kernel function as follows: ,in: Represents sample points and Kernel function values ​​between; This represents the square of the Euclidean distance between sample points; For kernel width parameters; The LSTM model described is an LSTM incorporating an attention mechanism, and it inputs pose and physiological data through a sliding time window, with the time window size set to [value missing]. At every moment Before use Data at each time step is used as input.

7. The cockpit integrated alarm system based on decision fusion according to claim 5, characterized in that, The Stacking algorithm employs a weighted average fusion method to combine the prediction results of the SVM and LSTM models, and uses the fused prediction result as the final pilot behavior state recognition result. Specifically: ,in: and These represent the weights of the SVM and LSTM models, respectively, and their values ​​are determined through cross-validation. This represents the prediction result of the SVM model; This represents the prediction result of the LSTM model; This is the final fusion prediction result.

8. The cockpit integrated alarm system based on decision fusion according to claim 1, characterized in that, The alarm module uses the Softmax function to convert the fused prediction results into class probabilities, specifically: ,in: Indicates category The predicted probability; For the fused output results and categories The corresponding value.

9. The cockpit integrated alarm system based on decision fusion according to claim 8, characterized in that, The alarm module determines the alarm threshold based on the monitored pilot behavior and flight phase, and generates an alarm reminder to the pilot based on the alarm priority. The alarm threshold determination mentioned above refers to: When the state is mild fatigue and the criteria are met. When this occurs, the alarm channel is activated as a tactile channel, and the interface displays a message indicating that the pilot is in a state of mild fatigue. When the state is severe fatigue and the criteria are met. When this occurs, the alarm channels are activated as tactile and auditory channels, and the interface displays a message indicating severe pilot fatigue. When the state is distracted and the criterion is met. When this occurs, the alarm channel is activated as a tactile channel, and the interface displays a pilot distraction status alert. When the state is slightly high workload and the criteria are met. When this occurs, the alarm channel is activated as a tactile channel, and the interface displays a message indicating that the pilot's workload is slightly high. When the status is "workload too high" and the criteria are met. When this occurs, the alarm channels are activated as tactile and auditory channels, and the interface displays a message indicating that the pilot's workload is too high. When the state is that the workload is unbearable and the criteria are met. When this occurs, the alarm channels are activated as tactile, auditory, and visual, and the interface displays a status message indicating that the pilot's workload is unbearable. Specifically: and These are the alarm thresholds for mild fatigue and severe fatigue, respectively. Distraction alarm threshold; , and These are the alarm thresholds for slightly high workload, slightly high workload, and unbearable workload, respectively.

10. The cockpit integrated alarm system based on decision fusion according to claim 9, characterized in that, When two or more alarms exist simultaneously, the alarm priority logic rules include: 1) If fatigue and distraction coexist: If distraction is detected and fatigue exists, fatigue alarm will be output first, i.e., the alarm priority logic is: severe fatigue > mild fatigue > distraction; 2) If fatigue and workload coexist: In this case, fatigue is caused by excessive workload, so the workload alarm will be output first. That is, the alarm priority logic is: workload cannot be handled > workload is too high > workload is slightly high > fatigue. The alarm thresholds are adjusted according to different flight phases, specifically including: 1) During takeoff and landing, increase the sensitivity of workload alarms, i.e., lower the workload alarm threshold; 2) During the cruise phase, increase the sensitivity of distraction and fatigue alarms, i.e., lower the threshold for distraction and fatigue alarms.

Images