Pilot state monitoring and flight data real-time synchronous fusion system

Abstract

The invention relates to the technical field of pilot state monitoring, in particular to a pilot state monitoring and flight data real-time synchronous fusion system, which comprises an information acquisition module used for acquiring physiological state information of a pilot and flight data information of an airplane in real time; the intelligent control module is used for controlling the operation of the camera, the electroencephalogram acquisition device and the electrocardiograph; the information processing module is used for carrying out preprocessing operation on the collected pilot physiological state information and the collected aircraft flight data; and the data fusion module is used for fusing the physiological state information of the pilot and the flight data parameters of the aircraft. The early warning assistance module is used for immediately sending out an early warning signal when an abnormality is found; and positioning information and rescue guidance are provided for a pilot under the emergency condition that the aircraft encounters danger. According to the current attention state and the task requirement of the pilot, the system can dynamically adjust the collection frequency and the collection position of the electroencephalogram collection device so as to improve the efficiency and the pertinence of data collection.

Description

Technical Field

[0001] The present invention relates to the technical field of pilot status monitoring, and in particular to a system for real-time synchronous fusion of pilot status monitoring and flight data. Background Art

[0002] Flight safety has always been a focus of attention in the aviation field, and is directly related to the safety of life and property of passengers and crew members. With the increase in air transport volume and the complexity of the flight environment, the requirements for flight safety are becoming higher and higher. As the main executor of the flight mission, the pilot's physiological state, psychological state and operating skills have a vital impact on flight safety. At the same time, accurate recording and analysis of flight data is also an important means to ensure aviation safety. Through real-time monitoring and analysis of flight data, technical faults and abnormal conditions of the aircraft can be discovered in time, and corresponding measures can be taken to deal with them to avoid accidents.

[0003] In the prior art, fixed electrodes are usually used to collect pilot EEG information. This type of method usually cannot adjust the electrode position and has some disadvantages. It cannot accurately collect information based on the pilot's current brain activity range. During the flight, the pilot's brain activity will change with task requirements, environmental changes, and fluctuations in physiological state.

[0004] In summary, how to solve the problem that the existing technology uses fixed electrodes to collect pilot brain waves and cannot change according to the fluctuations of the pilot's physiological state has become a difficult problem that needs to be solved urgently in this field. Therefore, it is necessary to propose a real-time synchronous fusion system for pilot status monitoring and flight data. Summary of the invention

[0005] To solve the above problems, the present invention provides a pilot status monitoring and flight data real-time synchronous fusion system. According to the pilot's current attention state and task requirements, the system can dynamically adjust the collection frequency and collection position of the EEG collection device to improve the efficiency and pertinence of data collection.

[0006] In order to achieve the above-mentioned purpose, the technical solution of the present invention is as follows: a pilot status monitoring and flight data real-time synchronous fusion system comprises the following modules:

[0007] The information acquisition module is used to collect the pilot's physiological status information and the aircraft's flight data information in real time; wherein the pilot's physiological status information includes facial feature signals, EEG information and ECG data; the facial feature signals are collected using an image collector for capturing the pilot's facial features, the EEG information is collected using an EEG collection device that performs mobile collection and detection based on the range of the pilot's brain area, and the ECG data is collected using an ECG collector for collecting the pilot's electrocardiogram; the aircraft's flight data is collected using a number of onboard sensors.

[0008] The intelligent control module is used to control the operation of the image collector, EEG acquisition device and ECG acquisition device; and analyze the activity detection range of the pilot's brain area based on the facial feature signals collected by the image collector, and control the acquisition frequency and acquisition position of the EEG acquisition device in real time.

[0009] The information processing module is used to perform preprocessing operations on the collected pilot physiological status information and aircraft flight data, and extract characteristic parameters of the pilot physiological status information and aircraft flight data.

[0010] The data fusion module is used to analyze the correlation between the pilot's physiological state information and the aircraft's flight data using a neural network algorithm, and to fuse the pilot's physiological state information with the aircraft's flight data parameters based on the correlation between the two.

[0011] The early warning assistance module is used to monitor the pilot's physiological, ECG data and flight safety status in real time based on the fused data. If any abnormality is found, a warning signal will be issued immediately; and in the event of an emergency situation where the aircraft encounters danger, the pilot will be provided with positioning information and rescue guidance.

[0012] Furthermore, in the information acquisition module, the image collector includes a camera, and the electrocardiogram collector includes an electrocardiogram; the facial feature signals include blinking frequency, eye movement changes, mouth features and head posture changes; eye movement changes are used to judge the pilot's fatigue level based on the PERCLOS value.

[0013] Flight data includes speed, altitude, position, heading, fuel level and engine status.

[0014] Furthermore, in the intelligent control module, the detection range of the brain area includes frontal lobe EEG activity, parietal lobe EEG activity, occipital lobe EEG activity, temporal lobe EEG activity and insular lobe EEG activity.

[0015] The EEG information is divided into four bands according to different frequencies, namely, δ wave, θ wave, α wave and β wave; the power spectrum of each band is calculated using the Welch method, and the power ratio of the (θ+α) / β feature, the EEG fatigue feature, is calculated.

[0016] Furthermore, in the information processing module, the pilot physiological status information preprocessing includes the following steps:

[0017] S101, data cleaning: denoising the collected physiological status information through filtering algorithm; removing artifacts and noise in the data.

[0018] S102, data segmentation: for periodic data, it is segmented into multiple periods for analysis; for non-periodic data, it is segmented according to the time window and bad segments are automatically removed.

[0019] S103, feature extraction: calculate the mean, standard deviation, maximum and minimum statistics of the data; convert the data to the frequency domain, extract its frequency components and their intensities; and analyze the duration and peak occurrence time of the data; extract the approximate entropy ApEn and sample entropy SampEn of the physiological data.

[0020] S104, standardization processing: using standardization processing method to convert data to the same scale.

[0021] Furthermore, in the information processing module, aircraft flight data preprocessing includes the following steps:

[0022] S201, data synchronization: perform time synchronization processing on the flight data from different sensors and calibrate the data; and merge the data from different sensors to form a unified data set.

[0023] S202, data filtering: using a filtering algorithm to remove high-frequency noise and abnormal values ​​in the flight data; and compressing the pre-processed flight data, and storing it after compression.

[0024] S203, data conversion: converting the flight data into a unified data format and unit.

[0025] S204, Data Dimensionality Reduction: Use analytical methods to reduce data dimensions and remove redundant information.

[0026] Furthermore, in the data fusion module, the neural network algorithm analyzes the data association including the following steps:

[0027] S301, model construction: select the recurrent neural network algorithm according to the characteristics of the data; determine the number of layers of the network, the number of neurons in each layer and the activation function; use a multimodal input layer to process physiological state information and flight data information separately; and then fuse them in subsequent layers of the network.

[0028] S302, model training: divide the preprocessed data into training set, validation set and test set; determine the training parameters of learning rate, batch size and number of iterations; use the training set data to train the neural network model.

[0029] S303, Model Evaluation: Use the test set data to evaluate the performance of the model, analyze the confusion matrix of the model, and understand the performance of the model in different categories; understand the accuracy, recall rate and F1 score indicators of the model; adjust the network structure, parameters and training strategy according to the evaluation results.

[0030] S304. Result interpretation: Analyze the results of the neural network model output, explain the correlation between the pilot's physiological state and the aircraft flight data, and use the LIME method to perform explanatory modeling on the model output.

[0031] Furthermore, in the construction of the S301 model of the neural network algorithm, the multimodal input layer includes the fusion of the data layer, the feature layer and the decision layer.

[0032] Furthermore, during the S302 model training process of the neural network algorithm, validation set data is used to monitor the performance of the model and adjust parameters as needed; the training model involves forward propagation and back propagation, and back propagation updates the model parameters according to the loss function.

[0033] Furthermore, the EEG acquisition device includes a helmet, with fixing straps symmetrically fixedly connected on both sides of the bottom of the helmet, and buckles fixedly connected to the ends of the fixing straps away from the helmet, and the fixing straps are detachably connected by the buckles; a monitoring component for monitoring EEG information is provided inside the helmet.

[0034] The monitoring component includes an arc-shaped collection rod and a controller. The collection rod slides with the inner wall of the helmet. A number of collection electrodes are fixedly connected to the side of the collection rod away from the helmet. The controller is used to receive the collection electrode signals and store the electrode signals. A driving component for driving the collection rod to rotate is provided on the outside of the helmet.

[0035] Furthermore, the driving assembly includes a driving motor, which is fixedly connected to the outer wall of the helmet; the output shaft of the driving motor extends from the outside of the helmet to the inside of the helmet and is fixedly connected to the collecting rod, and the end of the collecting rod away from the driving motor rotates with the inner wall of the helmet, and the controller is used to control the operation of the driving motor.

[0036] The above scheme has the following beneficial effects:

[0037] 1. This solution uses a camera to collect facial feature signals such as blinking frequency, eye movement changes, and head posture changes in real time. These signals can reflect the pilot's fatigue level, concentration, and emotional state. Using the EEG information collected by a targeted mobile EEG acquisition device, we can gain an in-depth understanding of the pilot's brain partition activity detection, emotional response, and nervous system activity state. This is of great significance for evaluating the pilot's cognitive load, emotional stability, and potential nervous system problems.

[0038] The acquisition frequency and location of the EEG acquisition device are adjusted according to the facial feature signals of the pilot's blinking frequency and eye movement changes; according to the pilot's current attention state and task requirements, the intelligent control module can dynamically adjust the working mode of the EEG acquisition device to improve the efficiency and pertinence of data acquisition. By analyzing the EEG signals, the activity range and intensity of different brain regions such as the frontal lobe, parietal lobe, occipital lobe, temporal lobe and insular lobe can be identified; this information helps to assess the pilot's cognitive load and psychological state. According to the changes in the range of brain region activity, the parameters of the EEG acquisition device are adjusted in real time to capture more useful information.

[0039] By monitoring the pilot's physiological status and the aircraft's flight data in real time, potential safety hazards can be discovered in a timely manner, and appropriate measures can be taken to intervene and deal with them. This helps reduce the risk of flight accidents and improve flight safety. By analyzing the pilot's physiological status information and the aircraft's flight data, the system can provide pilots with more accurate flight guidance and suggestions. This helps optimize the flight operation process and improve flight efficiency and comfort.

[0040] 2. This solution uses a filtering algorithm to remove noise and artifacts in the physiological status information when preprocessing the pilot's physiological status information, which can improve the signal-to-noise ratio of the data and reduce errors in subsequent analysis. Dividing periodic data into multiple periods for analysis and dividing non-periodic data by time window can help simplify the data analysis process and improve processing efficiency. At the same time, independent analysis of data in different time periods can more carefully observe the changing trend of the pilot's physiological status.

[0041] By calculating the statistics of the data and converting it to the frequency domain to extract the frequency components and their intensity, and analyzing the duration and peak occurrence time of the data, these operations can mine the key features in the physiological state information; these characteristic parameters not only help to more accurately describe the pilot's physiological state, but also provide strong support for subsequent state assessment, prediction and warning. Converting data to the same scale through standardized processing methods can eliminate the dimensional differences and magnitude differences between different data sources and promote the fusion and comparison of multi-source data; this helps to more comprehensively evaluate the pilot's physiological state and improve the system's comprehensive judgment ability.

[0042] 3. When preprocessing the flight data of the aircraft in this scheme, time calibration is performed to ensure that the data from different sensors are consistent in time, and the data from different sources are merged into a unified data set, simplifying the data processing process so that subsequent analysis and monitoring work can be carried out on a unified platform. Compression processing not only reduces the space occupied by data storage, but also speeds up the data transmission speed; the unified format makes the data more easily compatible and parsed by different systems and software, providing convenience for cross-platform sharing and analysis of data. By reducing the data dimension and removing redundant information, the amount of calculation for subsequent data processing and analysis can be reduced, and the processing speed can be improved. Real-time synchronized flight data and pilot status monitoring can promptly detect potential safety hazards and provide strong support for preventing flight accidents.

[0043] 4. This solution integrates physiological status information and flight data through a multimodal input layer to improve the model's expressiveness and prediction accuracy. The system can simultaneously process multi-source information from the pilot's physiological status and aircraft flight data. This multi-level and multi-dimensional fusion method helps capture complex correlations, improve the comprehensiveness and accuracy of data fusion, and promptly discover potential safety hazards. In view of the temporal and continuous characteristics of flight data and physiological status data, the selection of a recurrent neural network algorithm can better capture the time dependency in the data and improve the model's ability to adapt to dynamic changes. The iterative optimization process of dynamically adjusting model parameters according to changes in real-time data ensures the stability and accuracy of the model in practical applications. By comprehensively analyzing the pilot's physiological status and flight data, it can not only provide more accurate operational advice and support, but also deeply analyze potential patterns and trends. The data fusion results based on the neural network algorithm can provide pilots with more accurate and comprehensive decision support information, which helps pilots make more informed decisions in complex flight environments.

[0044] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 4 is a structural block diagram of a system for real-time synchronous fusion of pilot status monitoring and flight data in an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the process of preprocessing pilot physiological status information in an embodiment of the present invention.

[0047] Figure 3 Schematic diagram of the flow of aircraft flight data preprocessing in an embodiment of the present invention.

[0048] Figure 4Schematic diagram of the process of analyzing data association using a neural network algorithm in an embodiment of the present invention.

[0049] Figure 5 It is an axonometric diagram of the EEG acquisition device in an embodiment of the present invention.

[0050] Figure 6 It is a side cross-sectional view of the EEG acquisition device in an embodiment of the present invention.

[0051] The reference numerals in the drawings of the specification include: 1. helmet; 2. collection rod; 3. collection electrode; 4. driving motor; 5. fixing belt; 6. buckle. DETAILED DESCRIPTION

[0052] The following is further described in detail through specific implementation methods:

[0053] Embodiment 1:

[0054] As attached Figure 1-Figure 4 As shown: The pilot status monitoring and flight data real-time synchronous fusion system includes an information collection module, an intelligent control module, an information processing module, a data fusion module and an early warning assistance module.

[0055] The information collection module is used to collect the pilot's physiological status information and the aircraft's flight data information in real time. The intelligent control module is used to control the operation of various electrical components. The information processing module is used to extract the characteristic parameters of the pilot's physiological status information and the aircraft's flight data. The early warning assistance module is used to provide the pilot with positioning information and rescue guidance in the event of an emergency in which the aircraft encounters danger.

[0056] The functions of each module are explained in detail below: an information acquisition module is used to collect the pilot's physiological status information and the aircraft's flight data information in real time; wherein the pilot's physiological status information includes facial feature signals, EEG information and ECG data; facial feature signals are collected using an image collector for capturing the pilot's facial features, EEG information is collected using an EEG acquisition device that performs mobile acquisition and detection based on the pilot's brain area, and ECG data is collected using an ECG collector for collecting the pilot's electrocardiogram; the aircraft's flight data is collected using a number of onboard sensors.

[0057] Specifically, the image collector includes a camera, and the electrocardiogram collector includes an electrocardiogram; the facial feature signals include blinking frequency, eye movement changes, mouth features, and head posture changes; eye movement changes are used to determine the pilot's fatigue level based on the PERCLOS value, and the PERCLOS value calculation formula is as follows:

[0058] PERCLOS=(t / T)×100%

[0059] Among them, t is the eye closure time, T is the total time period; the PERCLOS threshold is set to 0.15; when the PERCLOS value exceeds 0.15, it means that the pilot is in a fatigue state, and the higher the value, the more serious the fatigue level.

[0060] These signals can reflect the pilot's fatigue level, concentration and emotional state. For example, a decrease in blinking frequency may indicate that the pilot is in a state of fatigue, while frequent changes in eye movements may mean that the pilot is highly focused. During the flight, a sober pilot will focus his eyes forward and scan his eyes quickly as the flight path changes. A pilot who is driving under fatigue will deviate from the normal direction, move slowly and scramble irregularly.

[0061] When you are tired and sleepy, your lips will turn up and you will yawn. These actions to relieve fatigue and increase the oxygen supply in the body will change the geometry of the mouth area. Therefore, you can judge the pilot's fatigue level by observing these mouth features. During driving, if the head moves unconsciously, it indicates sleepiness.

[0062] Flight data includes speed, altitude, position, heading, fuel level and engine status.

[0063] The intelligent control module is used to control the operation of the camera, EEG acquisition device and electrocardiograph; and analyzes the activity detection range of the pilot's brain area based on the facial feature signals collected by the camera, and controls the acquisition frequency and location of the EEG acquisition device in real time; for example, when the pilot is under high cognitive load or fatigue, the system can automatically increase the monitoring frequency and breadth to conduct targeted detection of the activity range of his brain area, thereby improving detection efficiency and ensuring flight safety. This intelligent adjustment method ensures that the collection of EEG information is more accurate and efficient, and improves the accuracy and reliability of monitoring.

[0064] Specifically, the detection range of brain areas includes frontal lobe EEG activity, parietal lobe EEG activity, occipital lobe EEG activity, temporal lobe EEG activity and insular lobe EEG activity. EEG information is divided into four bands according to different frequencies, namely, δ wave, θ wave, α wave and β wave; the power spectrum of each band is calculated using the Welch method, and the power ratio of (θ+α) / β feature is calculated to obtain EEG fatigue feature.

[0065] δ waves: Healthy individuals will not have this slow wave when awake, but will only appear when they are extremely tired, have brain hypoxia, or are anesthetized. θ waves are mainly found in the frontal and parietal regions. They appear when adults are sleepy or listless, and decrease when they are excited. α waves: This band accounts for a large proportion of the EEG signals in each partition. The rhythm of this band is most obvious when people are awake, quiet, and have their eyes closed. β waves: This wave dominates when people are in a state of high mental tension, extreme excitement, and excitement.

[0066] Delta and theta waves are usually called slow waves, and beta waves are called fast waves. When a person's mental state is inactive, slow waves dominate the brain waves. When a person is in an excited and tense state, fast waves dominate.

[0067] For example, by collecting EEG information from the frontal lobe to observe frontal lobe activity to evaluate the pilot's concentration level, or by collecting EEG information from the occipital lobe to clarify the pilot's occipital lobe activity to understand his visual processing efficiency.

[0068] The information processing module is used to perform preprocessing operations on the collected pilot physiological status information and aircraft flight data, and extract characteristic parameters of the pilot physiological status information and aircraft flight data.

[0069] Among them, Figure 2 As shown, the pilot physiological status information preprocessing includes the following steps:

[0070] S101, Data cleaning: De-noise the collected physiological state information through bandpass filtering algorithm; further remove artifacts and noise in the data, which can improve the signal-to-noise ratio of the data and reduce errors in subsequent analysis. Artifacts include physiological artifacts and non-physiological artifacts; physiological artifacts include eyes, muscles and heart, etc.; non-physiological artifacts come from a variety of factors, such as equipment performance and environmental electromagnetic interference factors.

[0071] Eye movement artifacts usually have large amplitudes and wide noise frequencies, and have a significant impact on the front area of ​​the head. The EEG acquisition device used in this example is located in the occipital lobe, which contributes most to fatigue. It is far away from the eye area, so it is basically not affected by eye movement artifacts, so there is no need to use related algorithms to remove eye movement artifacts.

[0072] S102, data segmentation: For periodic data, it is segmented into multiple periods for analysis; for non-periodic data, it is segmented according to the time window, and bad segments are automatically removed. By comparing the peak-to-peak difference of the EEG, if the peak-to-peak difference exceeds the preset normal threshold, it is regarded as a bad segment and removed; in this example, every 5 minutes is set as a time window to analyze the data; with a step of 20 seconds.

[0073] Since the bad segments in EEG data are scattered and not concentrated in a period of time, and fatigue is a long-term cumulative phenomenon, it will not occur suddenly and then suddenly disappear, so the continuity of the data is not required; when there are a few seconds of bad segments in a time window, they will be directly removed. According to the normal amplitude of EEG, the normal amplitude is soft-defined as -200μV~200μV, and those that exceed this amplitude range or the peak-to-peak difference exceeds 400μV are removed.

[0074] S103, feature extraction: calculate the mean, standard deviation, maximum and minimum statistics of the data; convert the data to the frequency domain through Fourier transform FFT, extract its frequency components and their intensities, such as the energy of alpha and beta waves in brain waves; and analyze the duration of the data and the time when the peak occurs to extract the approximate entropy ApEn and sample entropy SampEn of the physiological data.

[0075] Specifically, the approximate entropy is expressed as ApEn(m,r)=φm(r)-φm+1(r); the sample entropy is expressed as SampEn(m,r)=-lnφm(r) / φm+1(r); m is the selected spatial dimension vector, and r is the given threshold.

[0076] S104. Standardization processing: Use the Z-score standardization method to convert the data to the same scale to facilitate subsequent analysis and comparison.

[0077] like Figure 3 As shown, aircraft flight data preprocessing includes the following steps:

[0078] S201, data synchronization: perform time synchronization processing on the flight data from different sensors, that is, ensure that the data points recorded by different sensors correspond to the same time point; calibrate the data; and merge the data from different sensors to form a unified data set for subsequent processing and analysis.

[0079] S202, data filtering: using a median filtering algorithm to remove high-frequency noise and outliers in the flight data; and compressing the pre-processed flight data, and storing it after compression.

[0080] S203, data conversion: converting the flight data into a unified data format and unit, such as using JSON format, to facilitate importing into a database or analysis software.

[0081] S204, Data Dimensionality Reduction: Use PCA analysis method to reduce data dimension and remove redundant information, improve data processing efficiency and reduce unnecessary computational burden.

[0082] The data fusion module is used to analyze the correlation between the pilot's physiological state information and the aircraft's flight data using a neural network algorithm, and fuse the pilot's physiological state information with the aircraft's flight data parameters based on the correlation between the two. Figure 4 As shown, the neural network algorithm analyzes data association including the following steps:

[0083] S301, model construction: select the recurrent neural network algorithm RNN according to the characteristics of the data; determine the number of network layers, the number of neurons in each layer and the activation function ReLU; for example, select the long short-term memory network LSTM as the core layer; when LSTM updates the state, it will not retain all historical information, but will selectively discard some information, and the role of the forget gate is to decide the information to be discarded. Its calculation formula is as follows:

[0084] f t =σ(W f *[h i-1 ,x t ]+b f ) Among them, h i-1 is the output of the previous moment; x t is the current input; W f is the weight matrix; b f is the bias vector, σ is the activation function; the output number is between [0,1], indicating the proportion of the previous state information to be remembered, 1 means retention, and 0 means deletion.

[0085] A multimodal input layer is used, and the multimodal input layer includes a fusion of a data layer, a feature layer, and a decision layer. In this embodiment, a multimodal feature layer is selected; physiological state information and flight data information are processed separately; and then fused in subsequent layers of the network.

[0086] Feature layer fusion is to preprocess and extract features of multimodal signals separately, combine the extracted features into a feature vector, and finally use some machine learning and deep learning models for pattern recognition. Different from data layer fusion, feature layer fusion only fuses the feature data extracted from different data sets, with small data volume and low computational complexity.

[0087] S302, model training: divide the preprocessed data into training set, validation set and test set; determine the training parameters of learning rate, batch size and number of iterations; use the training set data to train the neural network model; during the training process, use the validation set data to monitor the performance of the model and adjust the parameters as needed. Training the model involves forward propagation and back propagation, and back propagation updates the model parameters according to the loss function; the loss function includes cross entropy loss for classification tasks or mean square error for regression tasks. For RNN, forward propagation includes calculating the hidden state ht and output y t , where t represents the time step; the calculation formula is as follows:

[0088] h t =σ(W hh h t-1 +W xh x t +b h )

[0089] y t =σ(W hy h t +b y )

[0090] Among them, W hh , W xh and W hy is the weight matrix, b h and b y is the bias term and σ is the activation function.

[0091] S303, Model Evaluation: Use the test set data to evaluate the performance of the model, analyze the confusion matrix of the model, understand the performance of the model on different categories, and understand the model's accuracy, recall rate, F1 score and other indicators; adjust the network structure, parameters and training strategy according to the evaluation results. For example, increase or decrease the number of layers, change the number of neurons; adjust the learning rate or batch size; use different optimizers or regularization methods.

[0092] S304, result interpretation: Analyze the results of the neural network model output, involving visualizing the model output, calculating correlation coefficients or performing other statistical analysis. Based on the model output and data analysis results, explain the correlation between the pilot's physiological state and the aircraft flight data, and use the LIME method to perform explanatory modeling on the model output.

[0093] The early warning assistance module is used to monitor the pilot's physiological, ECG data and flight safety status in real time based on the fused data. If an abnormality is found, a warning signal will be issued immediately; and in the event of an emergency, the pilot will be provided with positioning information and rescue guidance. By monitoring the pilot's physiological status and the aircraft's flight data in real time, potential safety hazards can be discovered in a timely manner, and appropriate measures can be taken to intervene and deal with them; this helps reduce the risk of flight accidents and improve the level of flight safety.

[0094] Embodiment 2:

[0095] As attached Figure 5 and Figure 6As shown, the difference from the above embodiment is that the EEG acquisition device includes a helmet 1, and fixing straps 5 are symmetrically fixed and bonded on both sides of the bottom of the helmet 1, and buckles 6 are fixed and bonded on one end of the fixing straps 5 away from the helmet 1, and the fixing straps 5 are detachably connected by the buckles 6; a monitoring component for monitoring EEG information is provided inside the helmet 1.

[0096] The monitoring component includes an arc-shaped collection rod 2 and a controller. The collection rod 2 slides with the inner wall of the helmet 1. A plurality of collection electrodes 3 are bolted and fixedly connected to the side of the collection rod 2 away from the helmet 1. The controller is used to receive the signals of the collection electrodes 3 and store the electrode signals. A driving component for driving the collection rod 2 to rotate is provided on the outside of the helmet 1.

[0097] The driving assembly includes a driving motor 4, which is bolted and fixedly connected to the outer wall of the helmet 1; the output shaft of the driving motor 4 extends from the outside of the helmet 1 to the inside of the helmet 1 and is bolted and fixedly connected to the collecting rod 2, and the end of the collecting rod 2 away from the driving motor 4 rotates in cooperation with the inner wall of the helmet 1, and the controller is used to control the operation of the driving motor 4.

[0098] The specific implementation process is as follows: when it is necessary to collect EEG information, first apply a layer of conductive paste on the pilot's head and the collection electrode 3, wear the helmet 1 on the pilot's head, and fix the helmet 1 through the fixing belt 5 and the buckle 6. Because a drive motor 4 is bolted and fixedly connected to the outer wall of the helmet 1, the output shaft of the drive motor 4 extends from the outside of the helmet 1 to the inside of the helmet 1 and is bolted and fixedly connected to the collection rod 2, and the collection rod 2 is arc-shaped and slides with the inner wall of the helmet 1; so when the output shaft of the drive motor 4 rotates, it can drive the collection rod 2 bolted and fixed to rotate. Since the side of the collection rod 2 away from the helmet 1 is bolted and fixedly connected with a plurality of collection electrodes 3, the EEG information of the head can be received through the collection motor, and the drive motor 4 can be controlled by the controller to drive the collection rod 2 to rotate, so as to flexibly collect the EEG information of the pilot according to the activity range of the brain area.

[0099] When the camera detects frequent eye movements of the pilot, it may mean that the pilot is concentrating highly. At this time, the frequency of EEG information collection is increased to obtain more comprehensive information about the pilot's physiological state. The EEG information collected by the targeted mobile EEG collection device can provide an in-depth understanding of the pilot's brain partition activity detection, emotional response, and nervous system activity state; this is of great significance for evaluating the pilot's cognitive load, emotional stability, and potential nervous system problems.

[0100] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. Pilot status monitoring and flight data real-time synchronous fusion system, characterized by: Includes the following modules: An information collection module is used to collect the pilot's physiological status information and the aircraft's flight data information in real time; wherein the pilot's physiological status information includes facial feature signals, EEG information and ECG data; the facial feature signals are collected using an image collector for capturing the pilot's facial features, the EEG information is collected using an EEG collection device for mobile collection and detection based on the range of the pilot's brain area, and the ECG data is collected using an ECG collector for collecting the pilot's electrocardiogram; the aircraft's flight data is collected using a number of onboard sensors; Intelligent control module, used to control the operation of the image collector, EEG acquisition device and ECG acquisition device; and to analyze the activity detection range of the pilot's brain area according to the facial feature signals collected by the image collector, and to control the acquisition frequency and acquisition position of the EEG acquisition device in real time; An information processing module, used to perform preprocessing operations on the collected pilot physiological state information and aircraft flight data, and extract characteristic parameters of the pilot physiological state information and aircraft flight data; A data fusion module is used to analyze the correlation between the pilot's physiological state information and the aircraft's flight data using a neural network algorithm, and fuse the pilot's physiological state information with the aircraft's flight data parameters based on the correlation between the two; The early warning assistance module is used to monitor the pilot's physiological, ECG data and flight safety status in real time based on the fused data. If any abnormality is found, a warning signal will be issued immediately; and in the event of an emergency situation where the aircraft encounters danger, the pilot will be provided with positioning information and rescue guidance.

2. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 1 is characterized in that: In the information acquisition module, the image collector includes a camera, and the electrocardiogram collector includes an electrocardiogram; the facial feature signals include blinking frequency, eye movement changes, mouth features, and head posture changes; eye movement changes are used to determine the pilot's fatigue level based on the PERCLOS value; Flight data includes speed, altitude, position, heading, fuel level and engine status.

3. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 2 is characterized in that: In the intelligent control module, the detection range of brain areas includes frontal lobe EEG activity, parietal lobe EEG activity, occipital lobe EEG activity, temporal lobe EEG activity and insular lobe EEG activity; The EEG information is divided into four bands according to different frequencies, namely, δ wave, θ wave, α wave and β wave; the power spectrum of each band is calculated using the Welch method, and the power ratio of the (θ+α) / β feature, the EEG fatigue feature, is calculated.

4. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 3 is characterized in that: In the information processing module, the pilot physiological status information preprocessing includes the following steps: S101, data cleaning: denoising the collected physiological status information through filtering algorithms; removing artifacts and noise in the data; S102, data segmentation: for periodic data, it is segmented into multiple periods for analysis; for non-periodic data, it is segmented according to the time window; and bad segments are automatically removed; S103, feature extraction: calculate the mean, standard deviation, maximum and minimum statistics of the data; convert the data to the frequency domain, extract its frequency components and their intensities; and analyze the duration and peak occurrence time of the data; extract the approximate entropy ApEn and sample entropy SampEn of the physiological data; S104, standardization processing: using standardization processing method to convert data to the same scale.

5. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 4 is characterized in that: In the information processing module, aircraft flight data preprocessing includes the following steps: S201, data synchronization: performing time synchronization processing on the flight data from different sensors and calibrating the data; and merging the data from different sensors to form a unified data set; S202, data filtering: using a filtering algorithm to remove high-frequency noise and abnormal values ​​in the flight data; and compressing the pre-processed flight data, and storing it after compression; S203, data conversion: converting the flight data into a unified data format and unit; S204, Data Dimensionality Reduction: Use analytical methods to reduce data dimensions and remove redundant information.

6. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 5 is characterized in that: In the data fusion module, the neural network algorithm analyzes data association including the following steps: S301, model construction: select the recurrent neural network algorithm according to the characteristics of the data; determine the number of layers of the network, the number of neurons in each layer and the activation function; use a multimodal input layer to process physiological state information and flight data information respectively; and then fuse them in subsequent layers of the network; S302, model training: dividing the preprocessed data into training sets, validation sets and test sets; determining training parameters such as learning rate, batch size and number of iterations; and training the neural network model using the training set data; S303, Model Evaluation: Use the test set data to evaluate the performance of the model, analyze the confusion matrix of the model, and understand the performance of the model on different categories; understand the accuracy, recall rate and F1 score indicators of the model; adjust the network structure, parameters and training strategy according to the evaluation results; S304. Result interpretation: Analyze the results of the neural network model output, explain the correlation between the pilot's physiological state and the aircraft flight data, and use the LIME method to perform explanatory modeling on the model output.

7. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 6 is characterized in that: In the construction of the S301 model of the neural network algorithm, the multimodal input layer includes the fusion of the data layer, the feature layer and the decision layer.

8. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 7 is characterized in that: During the S302 model training process of the neural network algorithm, the validation set data is used to monitor the performance of the model and adjust the parameters as needed; the training model involves forward propagation and back propagation, and the back propagation updates the model parameters according to the loss function.

9. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 8, characterized in that: The electroencephalogram (EEG) acquisition device comprises a helmet (1), wherein fixing straps (5) are symmetrically fixedly connected to both sides of the bottom of the helmet (1), and the ends of the fixing straps (5) away from the helmet (1) are fixedly connected to buckles (6), and the fixing straps (5) are detachably connected via the buckles (6); a monitoring component for monitoring EEG information is provided inside the helmet (1); The monitoring component comprises an arc-shaped collection rod (2) and a controller, wherein the collection rod (2) is slidably matched with the inner wall of the helmet (1); a plurality of collection electrodes (3) are fixedly connected to the side of the collection rod (2) away from the helmet (1), and the controller is used to receive signals from the collection electrodes (3) and store the electrode signals; and a driving component for driving the collection rod (2) to rotate is provided on the outer side of the helmet (1).

10. The pilot status monitoring and flight data real-time synchronous fusion system according to claim 9, characterized in that: The drive assembly comprises a drive motor (4), which is fixedly connected to the outer wall of the helmet (1); an output shaft of the drive motor (4) extends from the outer side of the helmet (1) to the inside of the helmet (1) and is fixedly connected to a collection rod (2); an end of the collection rod (2) away from the drive motor (4) is rotatably matched with the inner wall of the helmet (1); and a controller is used to control the operation of the drive motor (4).

Images