Cockpit comprehensive alarm system based on decision fusion

Abstract

A cockpit comprehensive alarm system based on decision fusion comprises a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module and an alarm module, and the data acquisition module acquires pilot operation data, attitude data and physiological data; the data synchronization and preprocessing module carries out time synchronization according to the pilot operation data, the attitude data and the physiological data, and carries out noise elimination and feature extraction; the model training and predicting module is used for training and predicting input data features through a support vector machine (SVM) model and a long-short term memory (LSTM) network model, and performing decision-level fusion on prediction results of the two models; and the alarm module gives out a corresponding alarm according to an alarm threshold judgment result and an alarm priority based on a predicted result in combination with the current flight stage. According to the invention, by integrating the operation data, the attitude data and the physiological data of the pilot and utilizing the decision fusion algorithm and the multi-layer alarm mechanism, the behavior state of the pilot is monitored in real time, and an alarm is given out in time when an abnormal behavior occurs.

Description

Technical Field

[0001] The present invention relates to a technology in the field of civil aviation, in particular to a cockpit integrated warning system based on decision fusion. Background Art

[0002] Currently, pilot behavior status monitoring mainly relies on a single monitoring method (such as physiological signals or posture monitoring), which cannot fully capture the pilot's comprehensive status, limiting the recognition and monitoring effect. At the same time, the existing warning system often issues warnings based on abnormal behavior operations. When the pilot is in a state of fatigue, distraction or workload overload, the traditional system cannot promptly identify and respond to changes in pilot behavior, resulting in insufficient accuracy and timeliness of the warning, and unable to provide effective support for pilots and crews. Summary of the invention

[0003] In view of the shortcomings of the prior art that the pilot's operation data, posture data and physiological data cannot be fully integrated and there is no warning mechanism, the present invention proposes a cockpit integrated warning system based on decision fusion. By integrating the pilot's operation data, posture data and physiological data, using a decision fusion algorithm and a multi-layer warning mechanism, real-time monitoring of the pilot's behavior status can be achieved, and timely warnings can be issued when abnormal behavior occurs.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a cockpit integrated warning system based on decision fusion, comprising: a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module and a warning module, wherein: the data acquisition module collects pilot operation data, posture data and physiological data; the data synchronization and preprocessing module performs time synchronization according to the pilot operation data, posture data and physiological data, and performs noise elimination and feature extraction; the model training and prediction module trains and predicts input data features through a support vector machine (SVM) model and a long short-term memory network (LSTM) model, and performs decision-level fusion on the prediction results of the two types of models; the warning module gives corresponding warnings based on the prediction results, combined with the current flight stage, according to the warning threshold judgment result and the warning priority. Technical Effects

[0006] The present invention is based on the comprehensive integration of the pilot's operation data, posture data and physiological data to reflect the pilot's status and combines it with a multi-layer warning mechanism. It can dynamically adjust the warning strategy according to the different degrees and priorities of the pilot's different behavioral states and in combination with the flight phase; it overcomes the limitations of a single data source, can comprehensively capture the pilot's multi-dimensional behavioral state, and provide more comprehensive and accurate pilot behavioral state information; through a dynamic warning mechanism, the warning level and response strategy are automatically adjusted according to the pilot's current different degrees of behavioral state, thereby providing timely, accurate and personalized warning information, avoiding excessive or insufficient warnings, and effectively improving the effectiveness of pilot flight safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the system of the present invention;

[0008] Figure 2 is a data processing flow chart of an embodiment;

[0009] Figure 3 The following is an alarm flow chart of an embodiment. DETAILED DESCRIPTION

[0010] like Figure 1 As shown, a cockpit integrated warning system based on decision fusion involved in this embodiment includes: a data acquisition module, a data synchronization and preprocessing module, a model training and prediction module and a warning module, wherein: the data acquisition module collects pilot operation data, posture data and physiological data; the data synchronization and preprocessing module performs time synchronization according to the pilot operation data, posture data and physiological data, and performs noise elimination and feature extraction; the model training and prediction module trains and predicts the input data features through a support vector machine (SVM) model and a long short-term memory network (LSTM) model, and performs decision-level fusion on the prediction results of these two types of models; the warning module gives corresponding warnings based on the prediction results, combined with the current flight stage, according to the warning threshold judgment results and the warning priority.

[0011] The pilot operation data includes: response time to key instructions and warnings, operation time intervals and compliance with standard operating procedures.

[0012] The pilot attitude data includes the pilot's sitting posture, center of gravity change and contact pressure distribution.

[0013] The pilot's posture data is collected through a pressure sensor array, which is arranged in multiple locations of the pilot's seat, including the back, seat cushion, and lumbar support area.

[0014] The pilot's physiological data includes electroencephalogram (EGG), electrocardiogram (ECG) and electrodermal (EDA) signals, wherein: the EEG signal is collected by an EEG sensor in direct contact with the scalp; the ECG signal is collected by a chest strap patch-type heart rate sensor; and the electrodermal signal is collected by a skin conductance sensor worn on the wrist.

[0015] like Figure 2 As shown, the data synchronization and preprocessing module includes: a data synchronization unit, a noise elimination unit and a feature extraction unit, wherein: the data synchronization unit aligns all data of different sensors to a unified time axis by linear interpolation; the noise elimination unit adopts Kalman filter and wavelet transform to remove noise of pilot posture data and physiological data respectively; the feature extraction unit extracts the feature x of pilot operation data operation , posture data features x posture and physiological data features x physiology .

[0016] The operational data features include: response time of key instructions and alarms, operation time interval and compliance with standard operating procedures. The obtained feature vector is: operation =[Tr mean ,Tr max ,Tr min ,Td mean ,Td std ,SOP rate ], where: Tr mean ,Tr max and Tr min They are the average response time, maximum response time and minimum response time respectively; Td mean and Td std are the mean difference and standard deviation between the operation time interval and the actual operation time interval; SOP rate Indicates how closely the performed operations match standard procedures.

[0017] The posture data features include: the pressure value of each sensor in the pressure sensor array, the pressure change rate, the stability of the sitting posture, the center of gravity position, the center of gravity change rate and the contact area. The obtained feature vector is: posture =[P,ΔP,P var ,X,Y,Z,ΔX,ΔY,ΔZ,A], where: P(t)=[p1,p2,…,p n ] is the pressure value of the sensor; p i represents the pressure value of the i-th sensor; ΔP = [Δp1, Δp2, …, Δp n ] is the pressure change rate of the sensor; Δp irepresents the pressure change rate of the i-th sensor; the stability of the pilot's sitting posture is determined by the variance of the pressure distribution P var Measure; X, Y and Z are the coordinate positions of the center of gravity respectively; ΔX, ΔY and ΔZ represent the rate of change of the center of gravity at time t respectively; A is the contact area.

[0018] The physiological data features include: EEG signal features X ECG 、ECG signal characteristics X ECG and skin electrical signal characteristics X EDA , the resulting eigenvector is x physiology =[X EGG ,X ECG ,X EDA ].

[0019] The EEG signal feature X EGG =[P delta ,P theta ,P alpha ,P beta ,P gamma ,PR,H,μ,T], where: P delta , P theta , P alpha , P beta and P gamma are the power spectral density of EGG signals in different frequency bands; PR is the relative ratio of the power of Alpha and Beta bands; H represents the entropy of EEG signal; μ is the mean of EEG signal; T represents the time domain characteristics of EGG signal, including but not limited to mean, standard deviation, variance, skewness, kurtosis and root mean square.

[0020] The ECG signal feature X ECG =[HR,SNDD,RMSSD,LF / HF], where: HR represents heart rate; SNDD represents the standard deviation of the RR interval (the time interval between adjacent heartbeats); RMSSD is the root mean square value of the difference between adjacent RR intervals; LF / HF is the low-frequency / high-frequency power ratio of the heart rate signal.

[0021] The electrical skin signal characteristic X EDA =[SCL mean ,SCL std ,SCR N ,SCR mean ,SCR N_total ], where: SCCL mean and SCL std Respectively represent the mean and standard deviation of basal skin conductance level (SCL); SCR N Indicates the peak value of skin conductance response (SCR); SCR mean Indicates the average peak amplitude of SCR; SCRN_total Indicates the total galvanic skin response number.

[0022] The model training and prediction module includes: a support vector machine (SVM) model, a long short-term memory network (LSTM) model and a decision fusion model, wherein: the support vector machine (SVM) model is trained based on the pilot's operation data to predict the pilot's behavior state; the long short-term memory network (LSTM) model is trained based on the fused posture data and physiological data to predict the pilot's behavior state; the decision fusion model uses the Stacking algorithm to perform decision-level fusion on the prediction results of the SVM model and the LSTM model.

[0023] The pilot behavior states include fatigue, distraction and workload overload. The fatigue state is divided into mild fatigue and severe fatigue; the workload overload state is divided into three states: slightly high workload, excessive workload and unbearable workload.

[0024] The SVM model adopts the support vector regression (SVR) algorithm based on supervised learning, and the model kernel function is the RBF kernel, which is: Where: K(x i ,x j ) represents the sample point x i and x j The kernel function value between ||x i -x j || 2 represents the square of the Euclidean distance between sample points; σ is a kernel width parameter, and its value is determined by cross-validation. In this embodiment, σ is set to 1.

[0025] The LSTM model is an LSTM combined with an attention mechanism, and inputs posture data and physiological data by sliding a time window. The time window size is set to W. At each time t, the data of the first W time steps are used as input. In this embodiment, the step size is 1s and W is set to 60.

[0026] The Stacking algorithm uses a weighted average fusion method to fuse the prediction results of the SVM model and the LSTM model, and uses the fused prediction results as the final pilot behavior state recognition results (fatigue, distraction or workload overload), specifically: Where: w SVM and w LSTM Respectively represent the weights of the SVM and LSTM models, and their values ​​are determined by cross-validation. SVM and w LSTM The values ​​of are 0.6 and 0.4 respectively; Represents the prediction results of the SVM model; Represents the prediction results of the LSTM model; is the final fusion prediction result.

[0027] The alarm module uses the Softmax function to convert the fusion prediction results into category probabilities, specifically: Where: p i represents the predicted probability of category i (fatigue, distraction, and workload excess); is the value corresponding to category i in the fused output result.

[0028] like Figure 3 As shown, the warning module determines the warning threshold according to the monitored pilot behavior status and the flight stage, and generates a warning to remind the pilot in combination with the warning priority.

[0029] Considering that when the pilot is in a state of fatigue, distraction or workload overload, the vision is blocked (fatigue, distraction) or over-occupied (workload overload), it is difficult to effectively obtain the warning light or information prompts on the display interface. Therefore, the warning for the pilot is mainly auditory prompts (voice) or tactile prompts (vibration), supplemented by visual prompts (warning lights), and the warning information is provided on the display interface simultaneously. The specific warning rules are as follows:

[0030] Where: θ fatigue1 and θ fatigue2 are respectively the warning thresholds for mild fatigue and severe fatigue. In this embodiment, θ fatigue1 and θ fatigue2 are 0.5 and 0.8 respectively; θ distraction is the distraction warning threshold. In this embodiment, θ distraction Set to 0.5; θ workload1 ,θ workload2 and θ workload3 are respectively the alarm thresholds of slightly high, slightly high and unbearable workload. In this embodiment, θ workload1 ,θ workload2 and θ workload3 Set them to 0.5, 0.7 and 0.9 respectively.

[0031] When two or more alarms exist at the same time, the alarm priority logic rules are as follows:

[0032] 1) If fatigue and distraction exist at the same time: If the distraction state is detected and the fatigue state exists, the fatigue alarm is output first, that is, the alarm priority logic is: severe fatigue > mild fatigue > distraction;

[0033] 2) If fatigue and workload exist at the same time: In this case, fatigue is caused by excessive workload, so the workload alarm is output first, that is, the alarm priority logic is: unbearable workload > excessive workload > slightly high workload > fatigue.

[0034] The warning threshold is adjusted according to different flight phases, and the specific rules are as follows:

[0035] 1) During the take-off and landing phases, the sensitivity of the workload warning is improved, that is, the workload warning threshold is lowered;

[0036] 2) During the cruise phase, the sensitivity of distraction and fatigue warnings is increased, that is, the distraction and fatigue warning thresholds are lowered.

[0037] In this embodiment, the reduction range of the above status alarm thresholds is 0.1.

[0038] This system can combine the pilot's operation data, attitude data and physiological data, make full use of the complementarity of different types of data, and adopt an algorithm based on decision fusion to improve the accuracy and comprehensiveness of pilot behavior status monitoring and warnings; compared with the traditional "after-the-fact" warnings of abnormal events, this system can analyze the pilot's behavior status in real time, quickly identify abnormal behaviors and issue warnings, ensure that pilots obtain information and take actions in a timely manner, thereby reducing flight risks. At the same time, by introducing multi-level warning levels, considering the different degrees and priorities of pilots' fatigue, distraction and workload overload, the warning is made more accurate. In addition, this system can adjust the threshold and sensitivity of the warning according to the flight stage to avoid excessive or insufficient warnings, thereby improving the applicability and practicality of the system.

[0039] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A cockpit integrated warning system based on decision fusion, characterized in that: include: Data acquisition module, data synchronization and preprocessing module, model training and prediction module and warning module, wherein: the data acquisition module collects pilot operation data, attitude data and physiological data; the data synchronization and preprocessing module performs time synchronization according to the pilot operation data, attitude data and physiological data, and performs noise elimination and feature extraction; the model training and prediction module trains and predicts the input data features through the support vector machine model and the long short-term memory network model, and performs decision-level fusion of the prediction results of these two types of models; the warning module gives corresponding warnings based on the prediction results, combined with the current flight stage, according to the warning threshold judgment results and the warning priority; The pilot operation data include: response time of key instructions and warnings, operation time interval and compliance with standard operating procedures; The pilot attitude data includes the pilot's sitting posture, center of gravity change and contact pressure distribution; The pilot attitude data is collected by a pressure sensor array, which is arranged on the back, seat cushion, and lumbar support area of ​​the pilot seat; The pilot physiological data includes electroencephalogram (EGG), electrocardiogram (ECG) and electrodermal (EDA) signals.

2. The cockpit integrated warning system based on decision fusion according to claim 1 is characterized in that: The data synchronization and preprocessing module includes: a data synchronization unit, a noise elimination unit and a feature extraction unit, wherein: the data synchronization unit aligns all data from different sensors to a unified time axis by linear interpolation; the noise elimination unit uses a Kalman filter and a wavelet transform to remove noise from the pilot's attitude data and physiological data respectively; the feature extraction unit extracts the pilot's operation data feature x operation , posture data features x posture and physiological data features x physiology .

3. The cockpit integrated warning system based on decision fusion according to claim 2 is characterized in that: The operational data features include: response time of key instructions and alarms, operation time interval and compliance with standard operating procedures. The obtained feature vector is: operation =[Tr mean ,Tr max ,Tr min ,Td mean ,Td std ,SOP rate ], where: Tr mean ,Tr max and Tr min They are the average response time, maximum response time and minimum response time respectively; Td mean and Td std are the mean difference and standard deviation between the operation time interval and the actual operation time interval; SOP rate Indicates how closely the performed operations match standard procedures.

4. The cockpit integrated warning system based on decision fusion according to claim 2 is characterized in that: The posture data features include: the pressure value of each sensor in the pressure sensor array, the pressure change rate, the stability of the sitting posture, the center of gravity position, the center of gravity change rate and the contact area. The obtained feature vector is: posture =[P,ΔP,P var ,X,Y,Z,ΔX,ΔY,ΔZ,A], where: P(t)=[p1,p2,…,p n ] is the pressure value of the sensor; p i represents the pressure value of the i-th sensor; ΔP = [Δp1, Δp2, …, Δp n ] is the pressure change rate of the sensor; Δp i represents the pressure change rate of the i-th sensor; the stability of the pilot's sitting posture is determined by the variance of the pressure distribution P var Measure; X, Y and Z are the coordinate positions of the center of gravity; ΔX, ΔY and ΔZ are the rates of change of the center of gravity at time t; A is the contact area; The physiological data features include: EEG signal features X EGG 、ECG signal characteristics X EGG and skin electrical signal characteristics X EDA , the resulting eigenvector is x physiology =[X EGG ,X ECC ,X EDA ]; The EEG signal feature X EGG =[P delta ,P theta ,P alpha ,P beta ,P gamma ,PR,H,μ,T], where: P delta , P theta , P alpha , P beta and P gamma are the power spectral density of EGG signals in different frequency bands; PR is the relative ratio of the power of Alpha and Beta bands; H represents the entropy of EEG signals; μ is the mean of EEG signals; T represents the time domain characteristics of EGG signals, including but not limited to mean, standard deviation, variance, skewness, kurtosis and root mean square; The ECG signal feature X ECG =[HR,SNDD,RMSSD,LF / HF], where: HR represents heart rate; SNDD represents the standard deviation of RR interval (the time interval between adjacent heartbeats); RMSSD is the root mean square value of the difference between adjacent RR intervals; LF / HF is the low frequency / high frequency power ratio of the heart rate signal; The electrical skin signal characteristic X EDA =[SCL mean ,SCL std ,SCR N ,SCR mean ,SCR N_total ], where: SCL mean and SCL std Respectively represent the mean and standard deviation of basal skin conductance level (SCL); SCR N Indicates the peak value of skin conductance response (SCR); SCR mean Indicates the average peak amplitude of SCR; SCR N_total Indicates the total galvanic skin response number.

5. The cockpit integrated warning system based on decision fusion according to claim 1 is characterized in that: The model training and prediction module includes: a support vector machine (SVM) model, a long short-term memory network (LSTM) model and a decision fusion model, wherein: the support vector machine (SVM) model is trained based on the pilot's operation data to predict the pilot's behavior state; the long short-term memory network (LSTM) model is trained based on the fused posture data and physiological data to predict the pilot's behavior state; the decision fusion model uses the Stacking algorithm to perform decision-level fusion on the prediction results of the SVM model and the LSTM model; The pilot behavior states include fatigue, distraction and workload excess, wherein the fatigue state is divided into mild fatigue and severe fatigue; the workload excess state is divided into three states: slightly high workload, excessive workload and unbearable workload.

6. The cockpit integrated warning system based on decision fusion according to claim 5 is characterized in that: The SVM model adopts the support vector regression (SVR) algorithm based on supervised learning, and the model kernel function is the RBF kernel, which is: Where: K(x i ,x j ) represents the sample point x i and x j The kernel function value between ||x i -x j || 2 Represents the square of the Euclidean distance between sample points; σ is the kernel width parameter; The LSTM model is an LSTM combined with an attention mechanism, and inputs posture data and physiological data in a time window sliding manner. The time window size is set to W, and at each time t, the data of the previous W time steps are used as input.

7. The cockpit integrated warning system based on decision fusion according to claim 5 is characterized in that: The Stacking algorithm uses a weighted average fusion method to fuse the prediction results of the SVM model and the LSTM model, and uses the fused prediction results as the final pilot behavior state recognition results (fatigue, distraction or workload overload), specifically: Where: w SVM and w LSTM Represent the weights of the SVM and LSTM models respectively, and their values ​​are determined by cross-validation; Represents the prediction results of the SVM model; Represents the prediction results of the LSTM model; is the final fusion prediction result.

8. The cockpit integrated warning system based on decision fusion according to claim 1 is characterized in that: The alarm module uses the Softmax function to convert the fusion prediction results into category probabilities, specifically: Where: p i represents the predicted probability of category i (fatigue, distraction, and workload excess); is the value corresponding to category i in the fused output result.

9. The cockpit integrated warning system based on decision fusion according to claim 1 or 8, characterized in that: The warning module determines the warning threshold according to the pilot's behavior status monitored and the flight stage, and generates a warning to remind the pilot according to the warning priority. The alarm threshold judgment refers to: When the state is mild fatigue and satisfies the criterion p1 ≥ θ fatigue1 When the warning channel is activated as the tactile channel, the display interface will show the pilot's mild fatigue status prompt information; When the state is severe fatigue and satisfies the criterion p1 ≥ θ fatigue2 When the warning channel is activated, the tactile channel + auditory channel is activated and the display interface is the pilot severe fatigue status prompt information; When the state is distracted and satisfies the criterion p2 ≥ θ distraction When the warning channel is enabled, the tactile channel is used and the display interface is the pilot distraction status prompt information; When the state is slightly high workload and satisfies the criterion p3 ≥ θ workload1 When the warning channel is activated, the tactile channel is used and the display interface is a prompt message of slightly high pilot workload; When the state is too high workload and satisfies the criterion p3 ≥ θ workload2 When the warning channel is activated, the tactile channel + auditory channel is activated and the display interface is the pilot workload overload status prompt information; When the state is that the workload is unbearable and satisfies the criterion p3 ≥ θ workload3 When θ is 0, the enabled warning channels are tactile channel + auditory channel + visual channel and the display interface is the pilot workload unbearable status prompt information, where: θ fatigue1 and θ fatigue2 are the warning thresholds for mild fatigue and severe fatigue respectively; θ distraction is the distraction warning threshold; θ workload1 ,θ workload2 and θ workload3 They are the warning thresholds for slightly high workload, slightly high workload, and unbearable workload.

10. The cockpit integrated warning system based on decision fusion according to claim 9 is characterized in that: When two or more alarms exist at the same time, the alarm priority logic rules include: 1) If fatigue and distraction exist at the same time: If the distraction state is detected and the fatigue state exists, the fatigue alarm is output first, that is, the alarm priority logic is: severe fatigue > mild fatigue > distraction; 2) If fatigue and workload exist at the same time: In this case, fatigue is caused by excessive workload, so the workload alarm is output first, that is, the alarm priority logic is: unbearable workload > excessive workload > slightly high workload > fatigue; The warning threshold is adjusted according to different flight phases, including: 1) During the take-off and landing phases, the sensitivity of the workload warning is improved, that is, the workload warning threshold is lowered; 2) During the cruise phase, the sensitivity of distraction and fatigue warnings is increased, that is, the distraction and fatigue warning thresholds are lowered.

Images