Pilot dangerous driving behavior monitoring method based on multi-modal data fusion

Abstract

The invention discloses a pilot dangerous driving behavior monitoring method based on multi-modal data fusion, and belongs to the field of multi-modal data fusion monitoring, and the method comprises the steps: obtaining the physiological data of a pilot in the take-off and landing states through a multi-element sensor, judging the fatigue state of the pilot, and obtaining the dangerous driving behavior of the pilot. An eye tracker and a camera are used for conducting data collection on a pilot in the take-off and landing states, behavior data of the blinking frequency, the eyelid closure degree, the pupil change, the yawning frequency, the nodding frequency, the average moving speed of an operating lever and the moving direction of the pilot are obtained, and the fatigue state of the pilot is judged; the method comprises the following steps: constructing a multi-modal data fusion network based on Transform, and training the multi-modal data fusion network; the driving behavior of a pilot is monitored by using the trained network based on the Transform multi-modal data fusion; through a multi-channel interaction technology, the comprehensiveness and accuracy of information are improved, the influence of environment and individual differences is reduced, and the stability and robustness are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of multimodal data fusion monitoring, and in particular to a method for monitoring dangerous driving behavior of pilots based on multimodal data fusion. Background Art

[0002] With the development of my country's economy and aviation industry, air transport has become an important part of my country's transportation industry, and airplanes have become an important means of travel for humans. In recent years, the level of aircraft automation design has been continuously improved. Although the proportion of air crashes caused by aircraft engine and operating platform failures has been decreasing, the proportion of flight accidents caused by improper operation of pilots has been increasing.

[0003] Therefore, how to identify and detect dangerous driving behaviors of pilots and conduct early warning and control is of great practical significance for regulating pilot behavior and ensuring safe aviation operations.

[0004] At present, domestic and foreign research on dangerous driving behaviors of pilots (emotional driving, fatigue driving) mostly adopts single-modal analysis and detection methods, mainly including detection methods based on physiological signal characteristics and detection methods based on visual monitoring technology. Such methods are easily affected by factors such as the environment and individual differences, and have disadvantages such as low stability and low robustness.

[0005] Based on this, the present invention aims to propose a pilot dangerous driving behavior monitoring method based on multimodal data fusion to solve the above problems. Summary of the invention

[0006] In order to make up for the shortcomings of the existing technical problems, the present invention proposes a pilot dangerous driving behavior monitoring method based on multimodal data fusion. Based on the multimodal signal feature fusion method and the use of multi-channel interaction technology (including human intelligent perception, eye tracking, face detection, action recognition, etc.), it can improve the comprehensiveness and accuracy of information, reduce the impact of environment and individual differences, and enhance stability and robustness.

[0007] To achieve the above object, the present invention provides the following technical solutions: 1. A method for monitoring dangerous driving behavior of pilots based on multimodal data fusion, characterized in that it comprises the following steps: (1) Use multi-sensors to obtain the pilot's physiological data during takeoff and landing, including EEG signals, ECG signals, EMG signals, and pulse signals; use eye trackers and cameras to collect image data of the pilot during takeoff and landing; (2) Determine the pilot's fatigue status based on EEG signals, ECG signals, EMG signals, and pulse signals, mark them, and then import them into AcqKnowledge software for data processing, extraction, and storage; (3) Preprocess the image data collected by the eye tracker and the camera, and use the YOL0v5s model to detect and identify the pilot's blinking frequency, eyelid closure, pupil changes, yawning frequency, nodding frequency, average joystick movement speed, and movement direction. Based on the recognition results, further judge the pilot's fatigue state and mark it; (4) Using the data obtained in step (1) and step (2) and the public data set to establish a database, the database is divided into a training set, a test set and a validation set; (5) Construct a Transformer-based multimodal data fusion network, which includes a feature extraction module, a Transformer-based multimodal data fusion module, and a Softmax classifier; use the extraction module to obtain physiological data features and image data features respectively, input the physiological data features and image data features into the Transformer-based multimodal data fusion module respectively to obtain fusion features, and use the fusion features to complete the pilot status classification through the Softmax classifier; (6) Using the database in step (4), the Transformer-based multimodal data fusion network is trained to obtain a trained Transformer-based multimodal data fusion network; (7) Use multiple sensors, eye trackers and cameras to collect data from pilots in real time. After processing, the collected data is input into a trained Transformer-based multimodal data fusion network to obtain the pilot's status and monitor the pilot's dangerous driving behavior.

[0008] In the present invention, step (2) determines the EEG signal by spectrum analysis and ERP analysis to determine the fatigue state; determines the ECG signal by processing and analyzing the frequency domain and time domain characteristics of HRV to determine the fatigue state; processes the electromyographic signal by continuous wavelet transform to obtain the sEMG power spectrum density, and analyzes the median frequency change to determine the fatigue state; and analyzes the pulse signal by analyzing the pulse peak and amplitude change in the time domain change graph in MATLAB to determine the fatigue state.

[0009] In the present invention, the preprocessing of the image data in step (3) includes grayscale conversion, median filtering noise reduction, gamma correction, contrast enhancement and histogram equalization preprocessing of the image.

[0010] In the present invention, in step (3), the YOL0v5s model is used to detect and identify pilot actions on the image data.

[0011] In the present invention, the network classification process of Transformer-based multimodal data fusion is specifically as follows: 1) Use a convolutional neural network to extract features from the image data collected by the eye tracker and the camera to obtain image data features, including blinking frequency, eyelid closure, pupil change, yawning frequency, nodding frequency, average joystick movement speed, and joystick movement direction; Use recursive neural network to extract features from physiological data collected by multi-sensors to obtain physiological data features, including features of EEG signals, ECG signals, EMG signals and pulse signals; 2) Input the physiological data features and image data features of step 1) into the Transformer-based multimodal data fusion module, wherein the physiological data features are input into Transformer encoder 1, and the image data features are input into Transformer encoder 2; Transformer encoder 1 and Transformer encoder 2 are both composed of multiple identical layers, each of which contains a multi-head self-attention layer connected with residuals and a position-based feedforward neural network; the relationship between sequence data learning features is processed through the self-attention mechanism to capture the complementary information relationship between physiological data features and image data features, and then the attention scores between different modalities are calculated, and the information is weighted and integrated to obtain the final attention weight, thereby generating the fused fusion features; 3) Input the fused features into the Transformer decoder to generate the final fusion result, and complete the pilot status classification through the Softmax classifier.

[0012] In the present invention, in the network training process of Transformer-based multimodal data fusion, accuracy, precision, recall and F1 score are used as performance indicators.

[0013]

[0014] Among them, TP is the number of samples correctly predicted as positive examples by the model, TN is the number of samples correctly predicted as negative examples by the model, FP is the number of samples incorrectly predicted as positive examples by the model, and FN is the number of samples incorrectly predicted as negative examples by the model; The performance indicators of the training set are compared with those of the validation set. If there is a large deviation in the prediction result or a model error, the cause is analyzed. According to the evaluation results, the model is continuously optimized by increasing the number of training rounds, adjusting model parameters, and using data augmentation technology to increase the diversity of the data set and obtain the target model.

[0015] Compared with the existing technology, this innovative result has significant advantages, as follows: (1) The present invention achieves more comprehensive information integration. By fusing multimodal data such as images and physiological signals, the system can capture information from a multidimensional perspective. At the same time, the complementarity of different modal data significantly enhances the comprehensiveness and accuracy of information.

[0016] (2) The present invention significantly improves the robustness of the system. When faced with complex situations where single-modal data is interfered with or missing, the multimodal data fusion mechanism can effectively alleviate its negative impact, thereby improving the stability and robustness of the system operation.

[0017] (3) The present invention exhibits better generalization ability. Multimodal fusion technology enables the system to adapt to a variety of usage scenarios more flexibly. Even if some modal data cannot be obtained under certain conditions, the system can still maintain basic performance and exhibit good fault tolerance. In addition, the system can intelligently select the most suitable modal data for processing according to actual needs, thereby enhancing the adaptability and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0020] like Figure 1 As shown, this embodiment discloses a method for monitoring dangerous driving behavior of pilots based on multimodal data fusion, comprising the following steps: Step 1: Collect and process the pilot's physiological signal data collected by multiple sensors, and use the physiological signal analysis and processing software based on Acqknowledge to determine the judgment indicators of various signals; the specific process is as follows: Step S11, using an EEG cap, an EEG sensor, an EMG sensor, and a pulse detector to detect the EEG signal, EEG signal, EMG signal, and pulse signal of the pilot in the takeoff and landing states respectively; Step S12, in the EEG signal processing and analysis, the fatigue degree can be evaluated by analyzing the frequency and amplitude changes of the EEG signal. This project intends to use the spectrum analysis method to convert the EEG signal into a frequency distribution diagram, and analyze the changes of the delta wave and theta wave, the alpha wave and the beta wave accordingly. At the same time, the ERP analysis method is used to process the amplitude of the EEG signal to produce an amplitude distribution diagram to obtain the brain's response to a specific event, and then analyze the extension or weakening of the brain's reaction time to the event. By combining the spectrum analysis method with the ERP analysis method, a more accurate degree of pilot fatigue status can be obtained.

[0021] In terms of ECG signal status monitoring, fatigue status is judged by analyzing the heart rate variability HRV (the small variation between heartbeats in healthy people). By processing and analyzing the frequency domain and time domain features of HRV, fatigue-related information such as LF / HF, SDNN, RMSSD, PNN50, etc. is extracted to analyze the fatigue status.

[0022] In the EMG signal processing and analysis, the degree of muscle fatigue can be evaluated by analyzing the changes in EMG signal frequency and amplitude. This project intends to use continuous wavelet transform to process EMG signals to obtain sEMG power spectrum density, and analyze the changes in median frequency to determine the degree of muscle fatigue. At the same time, the time domain features are obtained by processing the time representation of the original signal. The changes in signal amplitude and sign slope are analyzed according to the root mean square value and integrated EMG value in the time domain features to further determine the muscle activation intensity and fatigue state.

[0023] In the pulse data processing and analysis, the pulse data is imported into the MATLAB workspace in the form of a data table. By accessing a column of data in the pulse signal data table and assigning it to the variable x, the data source of a certain signal is obtained. The plot command is used to use the time column in the pulse data table as the horizontal axis variable and the single pulse signal data as the vertical axis variable to display the time domain change graph of the pulse signal. The pulse peak and amplitude changes are analyzed accordingly to determine the pilot's status.

[0024] Step S13, import the raw data file obtained in step S12 into AcqKnowledge software; use the graphical display function in AcqKnowledge to check the raw data to ensure that the data quality is good and there is no obvious noise or anomaly; if noise or anomaly exists, remove high-frequency noise (such as muscle activity, electromagnetic interference), slowly changing DC offset and low-frequency noise (such as drift) and interference of specific frequencies (such as 50 / 60Hz power line noise); adjust the sampling rate as needed to reduce the amount of data or improve accuracy; identify and mark or delete data segments containing significant noise; calculate statistical indicators such as mean, standard deviation, slope index, etc.; use fast Fourier transform (FFT) and other techniques to perform spectrum analysis; for complex signals (such as EEG), nonlinear analysis methods such as entropy and fractal dimension can be used; save the processed data and extracted features in a format suitable for subsequent analysis; Step 2: Obtain the pilot's facial and motion data through an eye tracker and camera, and use the YOL0v5s model to detect and identify the pilot's movements on the image data, including blinking frequency, eyelid closure, and pupil changes; as well as the number of pilot yawns, nodding frequency, average joystick movement speed, and movement direction; further judge the pilot's fatigue state based on the recognition results and mark it to prepare for training the Transformer-based multimodal data fusion network.

[0025] Among them, the training process of the YOL0v5s model is as follows: Step S21, obtaining pilot face images, facial expression data sets, and action data sets from existing public networks; Step S22, using an eye tracker to collect image data such as blink frequency, eyelid closure and pupil changes; using video monitoring images to obtain facial and motion data such as the number of yawns, nodding frequency, average joystick movement speed, and movement direction of the pilot; Step S23: Divide the image data obtained in steps S21 and S22 into 60% training set, 20% validation set, and 20% test set; perform grayscale conversion, median filter noise reduction, gamma correction, contrast enhancement, and histogram equalization preprocessing on the image, and mainly use the weighted average method for grayscale conversion, and the formula is as follows:

[0026] In the above formula, R is the red channel, G is the green channel, and B is the blue channel.

[0027] In order to remove image noise, smooth the image, and protect the edge information of the image, the image needs to be subjected to median filtering for noise reduction after grayscale processing. For the current pixel point (x, y) to be processed, a template is selected, and all the pixels in the template area are arranged according to the grayscale value to generate a monotonic sequence, and then the median of the sequence is taken out and assigned to the current pixel point as the pixel value g (x, y) of the processed image at (x, y), that is, Respectively represent the displacement in the horizontal direction and the vertical direction, W is a two-dimensional template, and a linear template is selected for median filtering. The formula of median filtering used in this patent is as follows:

[0028] In the above formula Represents the grayscale value of each pixel in the template before processing.

[0029] In order to solve the problem of decreased image clarity caused by median filter noise reduction and enhance the key information of the image, the image should be Gamma corrected after the median filter noise reduction process to adjust the contrast of the image and reduce uneven light and local shadows in the image. The basic formula of Gamma correction used in the present invention is as follows:

[0030] In the above formula, s represents the pixel value of the output image, r represents the pixel value of the input image, is the Gamma value, usually between 0.1 and 10. When γ<1, the image becomes brighter; when γ>1, the image becomes darker.

[0031] In order to make the details in the image clearer and more prominent, and to achieve better visual effects, and to solve the problem of dimming brightness after Gamma correction, the contrast enhancement technology is used to suppress useless information and make the key information stand out. Finally, the image is subjected to histogram equalization to effectively solve the problem of uneven distribution of image grayscale values, enhance the spatial sense and clarity of the image, and make the useful information in the image more prominent; Step S24: input the training set into the YOL0v5s model for training, input the test set into the trained YOL0v5s model, and compare the output results with the verification set to verify the accuracy of the results. After repeated training, the trained YOL0v5s model is finally obtained.

[0032] Step 3: Based on the physiological and image data obtained in steps 1 and 2, and public datasets such as OpenCV face image dataset, FER2013 dataset, WIDERFACE dataset, Dlib library, CK+ dataset, etc. are obtained from the Internet, and these data are divided into training sets and test sets, and feature selection and feature dimension reduction are performed on the data samples. The physiological data is preprocessed by removing invalid symbols, extracting valid data, and removing noise through appropriate filters, and the signal is converted into a format suitable for Transformer model input. The images are preprocessed by unifying image size, inverting, cropping, coloring, normalizing, etc., and then split into blocks, and the dimensions are kept consistent with the physiological data dimensions using linear changes to ensure that all images have the same resolution and pixel value range.

[0033] Step 4: Construct a Transformer-based multimodal data fusion network, which includes a feature extraction module, a Transformer-based multimodal data fusion module, and a Softmax classifier; First, convolutional neural networks (CNNs) are used to extract visual features of the image data processed in step S31 (including facial data and motion data. Facial data: blinking frequency, eyelid closure, pupil change, yawning frequency; motion data: nodding frequency, average joystick movement speed, movement direction), and recurrent neural networks (RNNs) are used to extract features of the physiological data (electroencephalogram signals, electrocardiogram signals, electromyogram signals, and pulse signals) processed in step S31; The extracted physiological features are then input into Transformer encoder 1, and the extracted image features are input into Transformer encoder 2. The Transformer encoder is composed of multiple identical layers, each of which contains a multi-head self-attention layer connected with residuals and a position-based feedforward neural network. The Transformer model is used to process sequence data through the self-attention mechanism, learn the relationship between features, capture the complementary information between physiological features and image features, and then calculate the attention scores between different modalities, and perform weighted integration of information to obtain the final attention weight, thereby generating a fused feature representation.

[0034] Finally, the fused features are input into the Transformer decoder to generate the final fusion result, and the pilot status classification is completed through the Softmax classifier.

[0035] Step 5: Use the data in step 3 to train the Transformer-based multimodal data fusion network to obtain a trained Transformer-based multimodal data fusion network.

[0036] Step S51: Check the prediction results of the model and calculate the performance indicators of the model, such as accuracy, precision, recall and F1 score. The calculation process of the performance indicators is shown in the formula.

[0037]

[0038] Among them, TP is the number of samples correctly predicted as positive examples by the model, TN is the number of samples correctly predicted as negative examples by the model, FP is the number of samples incorrectly predicted as positive examples by the model, and FN is the number of samples incorrectly predicted as negative examples by the model.

[0039] Step S52: compare with the validation set. If the prediction result has a large deviation or the model is wrong, analyze the cause, such as whether the model can distinguish different driving states, whether there is overfitting, etc.

[0040] Step S53: According to the evaluation results, the number of training rounds is increased, model parameters are adjusted, and data enhancement technology is used to increase the diversity of the data set, thereby continuously optimizing the model to obtain the target model.

[0041] Step 6: Use multi-sensors, eye trackers and cameras to collect data from the pilot in real time. After processing, the collected data is input into the trained Transformer-based multimodal data fusion model to obtain the pilot's status and monitor the pilot's dangerous driving behavior.

[0042] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for monitoring dangerous driving behavior of pilots based on multimodal data fusion, characterized in that: The following steps are involved: (1) Use multi-sensors to obtain the pilot's physiological data during takeoff and landing, including EEG signals, ECG signals, EMG signals, and pulse signals; use eye trackers and cameras to collect image data of the pilot during takeoff and landing; (2) Preliminarily judge the pilot's fatigue state based on EEG signals, ECG signals, EMG signals, and pulse signals, mark them, and then import them into AcqKnowledge software for data processing, extraction, and storage; (3) Preprocess the image data collected by the eye tracker and the camera, and use the YOL0v5s model to detect and identify the pilot's blinking frequency, eyelid closure, pupil changes, yawning frequency, nodding frequency, average joystick movement speed, and movement direction. Based on the recognition results, further judge the pilot's fatigue state and mark it; (4) Using the data obtained in step (1) and step (2) and the public data set to establish a database, the database is divided into a training set, a test set and a validation set; (5) Construct a Transformer-based multimodal data fusion network, which includes a feature extraction module, a Transformer-based multimodal data fusion module, and a Softmax classifier; use the extraction module to obtain physiological data features and image data features respectively, input the physiological data features and image data features into the Transformer-based multimodal data fusion module respectively to obtain fusion features, and use the fusion features to complete the pilot status classification through the Softmax classifier; (6) Using the database in step (4), the Transformer-based multimodal data fusion network is trained to obtain a trained Transformer-based multimodal data fusion network; (7) Use multiple sensors, eye trackers and cameras to collect data from pilots in real time. After processing, the collected data is input into a trained Transformer-based multimodal data fusion network to obtain the pilot's status and monitor the pilot's dangerous driving behavior.

2. The method for monitoring dangerous driving behavior of pilots based on multimodal data fusion according to claim 1, characterized in that: Step (2) judge the EEG signal by spectrum analysis and ERP analysis to judge the fatigue state; judge the ECG signal by processing and analyzing the frequency domain and time domain characteristics of HRV to judge the fatigue state; use continuous wavelet transform to process the electromyography signal to obtain sEMG power spectrum density, and analyze the change of median frequency to judge the fatigue state; analyze the pulse signal by the pulse peak and amplitude change of time domain change graph in MATLAB to judge the fatigue state.

3. The method for monitoring dangerous driving behavior of pilots based on multimodal data fusion according to claim 1, characterized in that: The preprocessing of the image data in step (3) includes grayscale conversion, median filtering noise reduction, gamma correction, contrast enhancement and histogram equalization preprocessing.

4. The method for monitoring dangerous driving behavior of pilots based on multimodal data fusion according to claim 1, characterized in that: In step (3), the YOL0v5s model is used to detect and recognize pilot actions on the image data.

5. The method for monitoring dangerous driving behavior of pilots based on multimodal data fusion according to claim 1, characterized in that: The network classification process of the Transformer-based multimodal data fusion is as follows: 1) Use a convolutional neural network to extract features from the image data collected by the eye tracker and the camera to obtain image data features, including blinking frequency, eyelid closure, pupil change, yawning frequency, nodding frequency, average joystick movement speed, and joystick movement direction; Use recursive neural network to extract features from physiological data collected by multi-sensors to obtain physiological data features, including features of EEG signals, ECG signals, EMG signals and pulse signals; 2) Input the physiological data features and image data features of step 1) into the Transformer-based multimodal data fusion module, wherein the physiological data features are input into Transformer encoder 1, and the image data features are input into Transformer encoder 2; Transformer encoder 1 and Transformer encoder 2 are both composed of multiple identical layers, each of which contains a multi-head self-attention layer connected with residuals and a position-based feedforward neural network; the relationship between sequence data learning features is processed through the self-attention mechanism to capture the complementary information relationship between physiological data features and image data features, and then the attention scores between different modalities are calculated, and the information is weighted and integrated to obtain the final attention weight, thereby generating the fused fusion features; 3) Input the fused features into the Transformer decoder to generate the final fusion result, and complete the pilot status classification through the Softmax classifier.

6. The method for monitoring dangerous driving behavior of pilots based on multimodal data fusion according to claim 1, characterized in that: In the process of network training for multimodal data fusion based on Transformer, accuracy, precision, recall and F1 score are used as performance indicators. Accuracy: ; Accuracy: ; Recall: ; F1 score: ; Among them, TP is the number of samples correctly predicted as positive examples by the model, TN is the number of samples correctly predicted as negative examples by the model, FP is the number of samples incorrectly predicted as positive examples by the model, and FN is the number of samples incorrectly predicted as negative examples by the model; The performance indicators of the training set are compared with those of the validation set. If there is a large deviation in the prediction result or a model error, the cause is analyzed. According to the evaluation results, the model is continuously optimized by increasing the number of training rounds, adjusting model parameters, and using data augmentation technology to increase the diversity of the data set and obtain the target model.

Images