A pilot dangerous driving behavior monitoring method based on multi-modal data fusion

Abstract

The application discloses a kind of pilot dangerous driving behavior monitoring methods based on multi-modal data fusion, belong to multi-modal data fusion monitoring field, including using multivariate sensor obtains the physiological data of pilot under the state of take-off and landing, judge the fatigue state of pilot, utilize eye tracker and camera to the pilot under the state of take-off and landing carry out data acquisition, obtain pilot blink frequency, eyelid closure, pupil change, yawn frequency, nodding frequency, joystick average moving speed, the behavior data of moving direction, judge the fatigue state of pilot;Network based on the multi-modal data fusion of Transform is constructed, and it is trained;The driving behavior of pilot is monitored using the multi-modal data fusion network based on the Transform trained;Through multi-channel interactive technology improves the comprehensiveness and accuracy of information, reduce the influence of environment, individual difference, enhance stability and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal data fusion monitoring, and particularly relates to a pilot dangerous driving behavior monitoring method based on multi-modal data fusion. BACKGROUND

[0002] With the development of China's economy and aviation industry, the aviation transportation industry has become an important part of China's transportation industry, and the airplane has become an important means of transportation for human beings. In recent years, the level of automatic design of airplanes has been continuously improved, although the proportion of air accidents caused by engine and operation platform failures of airplanes is continuously declining, but the proportion of flight accidents caused by improper operation of pilots is continuously rising.

[0003] Therefore, how to identify and detect the dangerous driving behavior of pilots and carry out early warning control has important practical significance for regulating the behavior of pilots and ensuring the safe operation of aviation.

[0004] At present, the research on the dangerous driving behavior (emotional driving and fatigue driving) of pilots at home and abroad mainly adopts a single modal analysis and detection method, mainly including a detection method based on physiological signal characteristics and a detection method based on visual monitoring technology. Such a method is easily affected by environmental and individual differences, and has the disadvantages of poor stability and low robustness.

[0005] Based on this, the present application aims to provide a pilot dangerous driving behavior monitoring method based on multi-modal data fusion to solve the above problems. SUMMARY

[0006] In order to make up for the shortcomings of the prior art, the present application provides a pilot dangerous driving behavior monitoring method based on multi-modal data fusion, which is based on a multi-modal signal feature fusion method and uses multi-channel interaction technology (including human intelligent perception, eye tracking, face detection, motion recognition, etc.), so as to improve the comprehensiveness and accuracy of information, reduce the influence of environmental and individual differences, and enhance the stability and robustness.

[0007] To achieve the above purpose, the present application provides the following technical scheme:

[0008] 1. A pilot dangerous driving behavior monitoring method based on multi-modal data fusion, characterized by comprising the following steps:

[0009] (1) acquiring physiological data including electroencephalogram signals, electrocardiogram signals, electromyogram signals and pulse signals of a pilot in a take-off and landing state by using a multi-element sensor; and collecting image data of the pilot in the take-off and landing state by using an eye tracker and a camera;

[0010] (2) According to the electroencephalogram signal, the electrocardiogram signal, the electromyogram signal and the pulse signal, the fatigue state of the pilot is judged and marked, and then the data is imported into the AcqKnowledge software for data processing, extraction and saving;

[0011] (3) The image data collected by the eye tracker and the camera is preprocessed, the YOL0v5s model is used for image data detection and recognition of the pilot's blink frequency, eyelid closure degree, pupil change, yawning frequency, nodding frequency, joystick average moving speed, moving direction behavior, the fatigue state of the pilot is further judged according to the recognition result, and is marked;

[0012] (4) A database is established by using the data obtained in steps (1) and (2) and the public data set, and the database is divided into a training set, a test set and a verification set;

[0013] (5) A network based on the multi-modal data fusion of the Transformer is constructed, the network includes a feature extraction module, a multi-modal data fusion module based on the Transformer and a Softmax classifier; the physiological data features and the image data features are obtained by using the extraction module, the physiological data features and the image data features are input into the multi-modal data fusion module based on the Transformer respectively, the fusion features are obtained, and the pilot state classification is completed by the Softmax classifier;

[0014] (6) The network based on the multi-modal data fusion of the Transformer is trained by using the database of step (4), and the trained network based on the multi-modal data fusion of the Transformer is obtained;

[0015] (7) The pilot is real-time data acquisition by using the multi-element sensor, the eye tracker and the camera, the collected data is input into the trained network based on the multi-modal data fusion of the Transformer after processing, the pilot state is obtained, and the dangerous driving behavior of the pilot is monitored.

[0016] In the application, step (2) judges the electroencephalogram signal by using the frequency spectrum analysis method and the ERP analysis method, so as to judge the fatigue state; the electrocardiogram signal is judged by processing and analyzing the frequency domain and time domain characteristics of the HRV, so as to judge the fatigue state; the sEMG power spectral density is obtained by using the continuous wavelet transform to process the electromyogram signal, and the median frequency change is analyzed, so as to judge the fatigue state; the pulse signal is analyzed by using the time domain change graph pulse peak value and amplitude change in MATLAB, so as to judge the fatigue state.

[0017] In the present application, the pre-processing of the image data in step (3) includes grayscale, median filter noise reduction, Gamma correction, contrast enhancement and histogram equalization pre-processing.

[0018] In the present application, in step (3), the YOL0v5s model is used to detect and identify the pilot's action of the image data.

[0019] In the present application, the network classification process based on the Transformer multi-modal data fusion is as follows:

[0020] 1) using a convolutional neural network to extract features from the image data collected by the eye tracker and the camera, obtaining image data features, the image data features including blink frequency, eyelid closure, pupil change, yawn frequency, nodding frequency, joystick average moving speed and joystick moving direction features;

[0021] using a recurrent neural network to extract features from the physiological data collected by the multi-element sensor, obtaining physiological data features, the physiological data features including electroencephalogram, electrocardiogram, electromyogram and pulse signal features;

[0022] 2) input the physiological data features and image data features of step 1) into the Transformer multi-modal data fusion module, wherein the physiological data features are input into the Transformer encoder 1 and the image data features are input into the Transformer encoder 2; the Transformer encoder 1 and the Transformer encoder 2 are stacked by multiple same layers, each layer containing a multi-head self-attention layer and a position-based feedforward neural network connected by a residual connection; the self-attention mechanism is used to process sequence data to learn the relationship between features, capture the complementary information relationship between physiological data features and image data features, then calculate the attention score between different modalities, and then perform information weighted integration to obtain the final attention weight, thereby generating the fused features after fusion;

[0023] 3) input the fused features into the Transformer decoder to generate the final fusion result, and complete the pilot state classification through the Softmax classifier.

[0024] In the present application, in the network training process based on the Transformer multi-modal data fusion, the accuracy, precision, recall and F1 score are used as performance indicators

[0025]

[0026]

[0027] Where TP is the number of samples correctly predicted as positive by the model, TN is the number of samples correctly predicted as negative by the model, FP is the number of samples incorrectly predicted as positive by the model, and FN is the number of samples incorrectly predicted as negative by the model.

[0028] The performance metrics of the training set are compared with those of the validation set. If there is a large deviation in the prediction results or a model error, the cause is analyzed. Based on the evaluation results, the diversity of the dataset is increased by increasing the number of training rounds, adjusting the model parameters, and using data augmentation techniques. The model is continuously optimized to obtain the target model.

[0029] Compared to existing technologies, this innovative achievement demonstrates significant advantages, as detailed below:

[0030] (1) This invention achieves more comprehensive information integration. By fusing multimodal data such as images and physiological signals, the system can capture information comprehensively from multiple perspectives. At the same time, the complementarity of different modal data significantly enhances the comprehensiveness and accuracy of the information.

[0031] (2) The present invention significantly improves the robustness of the system. When faced with complex situations where single-modal data is disturbed or missing, the multimodal data fusion mechanism can effectively mitigate its negative impact, thereby improving the stability and robustness of the system operation.

[0032] (3) This invention exhibits better generalization ability. Multimodal fusion technology enables the system to adapt more flexibly to diverse application scenarios. Even if some modal data cannot be obtained under specific conditions, the system can still maintain basic performance and exhibit good fault tolerance. In addition, the system can intelligently select the most suitable modal data for processing according to actual needs, thereby enhancing the system's adaptability and flexibility. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0034] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0035] like Figure 1 As shown in the figure, this embodiment discloses a method for monitoring dangerous driving behavior of pilots based on multimodal data fusion, including the following steps:

[0036] Step 1: Acquire and process physiological signal data collected from multiple sensors by the pilot, and use Acqknowledge-based physiological signal analysis and processing software to determine the judgment indicators for various signals; the specific process is as follows:

[0037] Step S11, use the electroencephalogram cap, electrocardio sensor, electromyography sensor, pulse detector to detect the electroencephalogram signal, electrocardio signal, electromyography signal and pulse signal of the pilot in the take-off and landing state respectively;

[0038] Step S12, in the electroencephalogram signal processing analysis, the fatigue degree can be evaluated by analyzing the frequency and amplitude changes of the electroencephalogram signal. This project plans to convert the electroencephalogram signal into a frequency distribution graph by using the frequency spectrum analysis method, and analyze the changes of δ wave and θ wave, α wave and β wave. At the same time, the amplitude distribution graph is made by using the ERP analysis method to process the electroencephalogram signal amplitude, and the response of the brain to the specific event is obtained, and then the reaction time extension or weakening of the brain to the event is analyzed. The combination of frequency spectrum analysis method and ERP analysis method can obtain more accurate fatigue state degree of the pilot.

[0039] In the aspect of electrocardio signal state monitoring, the fatigue state is judged by analyzing the heart rate variability HRV (the slight variation between successive heartbeats of a healthy human body). By processing and analyzing the frequency domain and time domain characteristics of HRV, the information related to fatigue such as LF / HF, SDNN, RMSSD, PNN50, etc. is extracted, and the fatigue state is analyzed.

[0040] In the electromyography signal processing analysis, the muscle fatigue degree can be evaluated by analyzing the frequency and amplitude changes of the electromyography signal. This project plans to process the electromyography signal by using continuous wavelet transform to obtain the sEMG power spectral density, and analyze the median frequency change to judge the muscle fatigue degree. At the same time, the time domain characteristics are obtained by processing the time representation of the original signal, and the changes of signal amplitude and sign slope are analyzed according to the root mean square value and integral electromyography value in the time domain characteristics, to further judge the muscle activation intensity and fatigue state.

[0041] In the pulse data processing analysis, the pulse data is imported into the MATLAB workspace in the form of a data table, the data of a certain column of the pulse signal data table is accessed and assigned to a variable x to obtain the data source of a certain signal, the plot instruction is used to take the time column in the pulse data table as the horizontal axis variable and the single pulse signal data as the vertical axis variable to display the time domain change graph of the pulse signal, and the pilot state is judged by analyzing the pulse peak value and amplitude change.

[0042] Step S13, import the raw data file obtained from step S12 into AcqKnowledge software; check the raw data using the graphical display function in AcqKnowledge to ensure that the data quality is good and there is no obvious noise or abnormality; if there is noise or abnormality, remove high-frequency noise (such as muscle activity, electromagnetic interference), slow-changing direct current offset and low-frequency noise (such as drift) and specific frequency interference (such as 50 / 60Hz power line noise); adjust the sampling rate as needed to reduce the amount of data or improve accuracy; identify and mark or delete data segments containing significant noise; calculate statistical indicators such as mean, standard deviation, slope index, etc.; perform spectral analysis using techniques such as Fast Fourier Transform (FFT); for complex signals (such as EEG), nonlinear analysis methods such as entropy, fractal dimension, etc. can be used; save the processed data and extracted features in a format suitable for subsequent analysis;

[0043] Step 2, obtain pilot facial and motion data through the eye tracker and camera device, use the YOL0v5s model to detect and recognize pilot motion from image data, including blink frequency, eyelid closure and pupil changes; as well as the number of yawns, nodding frequency, joystick average movement speed, and movement direction; further judge the pilot's fatigue state according to the recognition results and mark them, preparing for training the network based on the fusion of multi-modal data of Transformer.

[0044] The training process of the YOL0v5s model is as follows:

[0045] Step S21, obtain pilot face image, facial expression data set and motion data set from existing public network;

[0046] Step S22, use the eye tracker to collect image data such as blink frequency, eyelid closure and pupil changes; use video monitoring images to obtain facial and motion data such as the number of yawns, nodding frequency, joystick average movement speed, and movement direction;

[0047] Step S23, divide the image data obtained in steps S21 and S22 into 60% training set, 20% validation set, and 20% test set; perform grayscale, median filter noise reduction, Gamma correction, contrast enhancement, and histogram equalization preprocessing on the images, mainly using the weighted average method for grayscale processing, the formula is as follows:

[0048]

[0049] In the above formula, R is the red channel, G is the green channel, and B is the blue channel.

[0050] In order to remove the noise of the image, smooth the image while protecting the edge information of the image, the image needs to be subjected to median filtering noise reduction processing after the grayscale processing is completed. For the current pixel point (x, y) to be processed, a template is selected, all pixel points in the template region are arranged according to the size of the gray value to generate a monotonic sequence, and then the median of the sequence is taken out and assigned to the current pixel point as the pixel value g(x, y) of the processed image at (x, y), that is respectively represent the displacement amount in the horizontal direction and the vertical direction, W is a two-dimensional template, a linear template is selected for median filtering, and the formula of the median filtering adopted in the present application is as follows:

[0051]

[0052] In the above formula, g(x, y) represents the gray value of the pixel point at (x, y) in the processed image.

[0053] In order to solve the problem of the decrease in the image definition caused by the median filtering noise reduction, strengthen the key information of the image, and after the median filtering noise reduction processing is completed, the image should be subjected to Gamma correction for adjusting the contrast of the image and reducing the light unevenness and local shadow in the image. The basic formula of the Gamma correction adopted in the present application is as follows:

[0054]

[0055] In the above formula, s represents the pixel value of the output image, r represents the pixel value of the input image, is the Gamma value, usually between 0.1 and 10. When γ<1, the image becomes brighter; when γ>1, the image becomes darker.

[0056] In order to make the details in the image more clear and prominent, and the visual effect better, solve the problem of the decrease in the brightness after the Gamma correction, and adopt the technology of enhancing the contrast to suppress the useless information, so that the key information is prominent;

[0057] Finally, the histogram equalization is adopted for the image, which effectively solves the problem of the uneven distribution of the gray value of the image, enhances the spatial sense and definition of the image, and makes the useful information in the image more prominent;

[0058] Step S24, input the training set into the YOL0v5s model for training, input the test set into the trained YOL0v5s model, compare the output result with the verification set to verify the accuracy of the result, and finally obtain the trained YOL0v5s model through repeated training.

[0059] ​Step 3, according to the physiological and image data obtained in steps 1 and 2, and publicly available datasets such as OpenCV face image dataset, FER2013 dataset, WIDERFACE dataset, Dlib library, CK+ dataset, etc. are obtained from the network, and these data are divided into training set and test set, and the feature selection and feature dimension reduction of data samples are carried out. The physiological data is preprocessed by deleting invalid symbols, extracting effective data and removing noise by appropriate filter, and the signal is converted into a format suitable for the input of the Transformer model. The image is preprocessed by uniform image size, inversion, cropping, toning, normalization, etc. The image is split into blocks, and linear variation is used to make the dimension consistent with the physiological data dimension, so as to ensure that all images have the same resolution and pixel value range.

[0060] Step 4, construct a network based on Transformer multi-modal data fusion, which includes feature extraction module, Transformer multi-modal data fusion module and Softmax classifier.

[0061] Firstly, the visual features of the image data (including face data and action data. Face data: blink frequency, eyelid closure degree, pupil change, yawn frequency; action data: nodding frequency, joystick average moving speed, moving direction) obtained by step S31 are extracted using convolutional neural network (CNNs), and the features of the physiological data (electroencephalogram, electrocardiogram, electromyogram and pulse signal) obtained by step S31 are extracted using recurrent neural network (RNNs);

[0062] Then the extracted physiological features are input into the Transformer encoder 1, and the extracted image features are input into the Transformer encoder 2. The Transformer encoder is stacked by multiple identical layers, each layer contains multi-head self-attention layer and position-based feedforward neural network connected by residual connection. The sequence data is processed by self-attention mechanism using Transformer model, the relationship between features is learned, the complementary information between physiological features and image features is captured, etc. Then the attention scores between different modalities are calculated, and the final attention weight is obtained by information weighting integration, so as to generate the fused feature representation.

[0063] Finally, the fused features are input into the Transformer decoder to generate the final fusion result, and the pilot state classification is completed by the Softmax classifier.

[0064] Step 5, train the Transformer-based multi-modal data fusion network using the data in step 3 to obtain the trained Transformer-based multi-modal data fusion network.

[0065] Step S51, view the prediction results of the model and calculate the performance indicators of the model, such as accuracy, precision, recall and F1 score. The calculation process of the performance indicators is shown in the formula.

[0066]

[0067] Wherein, TP is the number of samples correctly predicted as positive examples by the model, TN is the number of samples correctly predicted as negative examples by the model, FP is the number of samples incorrectly predicted as positive examples by the model, and FN is the number of samples incorrectly predicted as negative examples by the model.

[0068] Step S52, compare with the validation set, if the prediction result deviation is large or the model is wrong, analyze the reason, such as whether the model can distinguish different driving states, whether there is overfitting and other problems.

[0069] Step S53, according to the evaluation result, increase the number of training rounds, adjust the model parameters and use data enhancement technology to increase the diversity of the data set, continuously optimize the model, and obtain the target model.

[0070] Step 6, use multi-element sensor, eye tracker and camera to collect data of pilots in real time, and input the processed data into the trained Transformer-based multi-modal data fusion model to obtain the state of the pilot and monitor the dangerous driving behavior of the pilot.

[0071] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art in the technical range disclosed by the present application, according to the technical scheme and the invention concept of the present application, equivalent replacement or change, should be covered in the protection scope of the present application.

Claims

1. A method for monitoring dangerous driving behavior of a pilot based on multi-modal data fusion, characterized in that, The method comprises the following steps: (1) acquiring physiological data of pilots in take-off and landing states by using a multi-sensor, the physiological data including electroencephalogram signals, electrocardiogram signals, electromyogram signals and pulse signals; and collecting image data of the pilots in the take-off and landing states by using an eye tracker and a camera; (2) preliminarily judging a fatigue state of the pilot according to the electroencephalogram signals, the electrocardiogram signals, the electromyogram signals and the pulse signals, and marking the fatigue state, and then importing the fatigue state into AcqKnowledge software for data processing, extraction and storage; (3) pre-processing the image data collected by the eye tracker and the camera, detecting and recognizing, by using a YOL0v5s model, behaviors of the pilot, such as blink frequency, eyelid closure degree, pupil change, yawning frequency, nodding frequency, average moving speed of a joystick, moving direction, and further judging the fatigue state of the pilot according to the recognized results and marking the fatigue state; (4) establishing a database by using the data obtained in steps (1) and (2) and a public data set, and dividing the database into a training set, a test set and a verification set; (5) constructing a network based on multi-modal data fusion of the Transformer, the network comprising a feature extraction module, a multi-modal data fusion module based on the Transformer and a Softmax classifier; obtaining physiological data features and image data features by using the extraction module, respectively, inputting the physiological data features and the image data features into the multi-modal data fusion module based on the Transformer, respectively, obtaining fusion features, and completing pilot state classification by the Softmax classifier; (6) training the network based on the multi-modal data fusion of the Transformer by using the database of step (4), and obtaining the trained network based on the multi-modal data fusion of the Transformer; (7) collecting data of the pilot in real time by using the multi-sensor, the eye tracker and the camera, inputting the processed data into the trained network based on the multi-modal data fusion of the Transformer, and obtaining a pilot state to monitor dangerous driving behaviors of the pilot.

2. The method of claim 1, wherein the method further comprises: In step (2), the electroencephalogram signals are judged by using a frequency spectrum analysis method and an ERP analysis method, the electrocardiogram signals are judged by processing and analyzing frequency domain and time domain characteristics of HRV, the electromyogram signals are processed by using continuous wavelet transform to obtain sEMG power spectral density and analyze median frequency changes, and the pulse signals are analyzed by using a time domain change graph in MATLAB to judge the fatigue state. 3.The method for monitoring dangerous driving behavior of a pilot based on multi-modal data fusion according to claim 1, characterized in that, In step (3), the pre-processing of the image data comprises grayscale processing, median filter noise reduction, Gamma correction, contrast enhancement and histogram equalization pre-processing.

4. The method of claim 1, wherein the method further comprises: In step (3), the YOL0v5s model is used to detect and recognize pilot actions from the image data.

5. The method of claim 1, wherein the method further comprises: The classification process of the network based on the multi-modal data fusion of the Transformer is as follows. 1) using convolutional neural network to extract features from image data collected by eye tracker and camera, obtaining image data features, including blink frequency, eyelid closure, pupil change, yawn frequency, nodding frequency, joystick average moving speed and joystick moving direction features; using recurrent neural network to extract features from physiological data collected by multi-element sensor, obtaining physiological data features, including electroencephalogram, electrocardiogram, electromyogram and pulse signal features; 2) input the physiological data features and image data features of step 1) into the Transformer-based multi-modal data fusion module, wherein the physiological data features are input into the Transformer encoder 1 and the image data features are input into the Transformer encoder 2; both the Transformer encoder 1 and the Transformer encoder 2 are stacked by multiple identical layers, each layer containing a multi-head self-attention layer and a position-based feedforward neural network connected by residual connection; the self-attention mechanism is used to process sequence data to learn the relationship between features and capture the complementary information relationship between physiological data features and image data features, then calculate the attention score between different modalities and integrate the information according to the attention score to obtain the final attention weight, thereby generating the fused features; 3) input the fused features into the Transformer decoder to generate the final fusion result, and complete the pilot state classification through the Softmax classifier.

6. The method for monitoring dangerous driving behavior of a pilot based on multi-modal data fusion according to claim 1, characterized in that, In the network training process of the Transformer-based multi-modal data fusion, the accuracy, precision, recall and F1 score are used as performance indicators Accuracy: ; Precision: ; Recall rate: ; F1 score: ; where TP is the number of samples correctly predicted as positive examples by the model, TN is the number of samples correctly predicted as negative examples by the model, FP is the number of samples incorrectly predicted as positive examples by the model, and FN is the number of samples incorrectly predicted as negative examples by the model; Compare the performance indicators of the training set with the validation set, if the prediction result deviation is large or the model is wrong, analyze the reason, according to the evaluation result, increase the training rounds, adjust the model parameters and use the data enhancement technology to increase the diversity of the data set, optimize the model constantly, and obtain the target model.

Images