Operation driver fatigue detection method based on multi-source information fusion

Abstract

The invention discloses an operating driver fatigue detection method based on multi-source information fusion, and relates to a data acquisition module, a data analysis processing module, a fatigue state classification module, a fatigue state real-time detection and evaluation module and a fatigue early warning module, and the fatigue state classification module is used for constructing a fatigue state classification model. The fatigue state real-time detection and evaluation module is connected with the data analysis and processing module and is used for summarizing real-time driver physiology, driving behaviors and external environment characteristics; and the lower connection fatigue state classification module inputs the multi-source indexes into a fatigue state classification model to obtain a fatigue state grade. And the fatigue early warning module is used for issuing early warning information to the driver according to the fatigue state grade of the driver so as to relieve the fatigue of the driver. And an automatic driving module of the vehicle is started when necessary, so that the driving safety of a driver is protected. According to the invention, real-time monitoring, evaluation and prevention of the fatigue of the operating driver are realized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation, and aims to train a driver fatigue status classification model by collecting multi-source data: driver physiology, driving behavior and external environment information, so as to detect the fatigue status level of operating drivers in real time and provide a fatigue warning method. Background Art

[0002] Operational drivers are prone to fatigue due to long-term driving operations, and fatigue driving poses a great threat to the driver's driving safety. With the development of human-machine co-driving technology, it is expected that accidents caused by driver fatigue will be reduced through vehicle-assisted driving in the future. And driver fatigue status detection is gradually becoming an important research direction in the field of human-machine co-driving. Existing studies have conducted research on driver fatigue detection based on driver physiological indicators, such as heart rate, breathing rate, etc., as well as driving behavior data, such as steering wheel angle and speed during driving. At the same time, studies have found that monotonous road environments can affect the driver's fatigue state.

[0003] Although the driver fatigue detection method has made some progress, the existing detection methods are often based on a single indicator and fail to consider the individual differences of drivers and the interference of the external environment, resulting in low accuracy of the detection results. In addition, the existing detection is mainly static recognition, lacking real-time dynamic continuous detection methods.

[0004] To overcome the above problems, the present invention is based on existing research conclusions and comprehensively judges the driver's fatigue state by widely combining multi-source information, including driver's physiological information, driving behavior and external environment characteristics, to improve the accuracy of the detection results. In addition, the pre-trained fatigue state classification model is used to achieve real-time detection of the fatigue state of the operating driver. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for detecting fatigue of an operating driver based on multi-source information fusion in view of the deficiencies in the above-mentioned prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Reference Figure 1 , a method for detecting fatigue of an operational driver based on multi-source information fusion, including a data acquisition module, a data analysis and processing module, a fatigue state classification module, a fatigue state real-time detection and evaluation module, and a fatigue warning module.

[0008] The data acquisition module is composed of a driver physiological signal data acquisition module, a driving behavior data acquisition module and an external environment data acquisition module, and is used to collect driver physiological signals, driving behavior and external environment information:

[0009] An ECG sensor with an operating frequency of 200 Hz is used to collect the driver's ECG;

[0010] A breathing sensor with an operating frequency of 50 Hz is used to collect the driver's breathing signal;

[0011] Eye tracker Tobii Pro X2-30, used to collect driver’s eye movement signals;

[0012] WLJ880 steering wheel angle detector, used to collect the vehicle steering wheel angle;

[0013] On-vehicle sensors to collect vehicle speed;

[0014] The camera is placed at eye level with the driver to record the vehicle's external environment information from the driver's perspective.

[0015] The data analysis and processing module is used to reduce noise and filter the data collected by the data acquisition module, and discard the missing data. Among them, the ECG signal is easily disturbed by environmental factors, and the original noisy ECG signal is purified by wavelet transform filtering; the noise in the respiratory data is concentrated on the high frequency by wavelet multi-layer noise reduction technology, and then effectively removed by threshold processing and signal reconstruction; for eye movement data, it is imported into the Tobii Pro Lab software and segmented according to the established time period, and data is marked. After the data is marked, it is exported and stored in Excel format to further extract the subject's gaze, saccade, pupil diameter and other characteristic parameters; for external environment data, semantic segmentation technology is used to extract external environment features and obtain the pixel ratio of objects in the external environment. Finally, the driver's physiological, driving behavior and external environment indicators are calculated: ECG indicators: heart rate (HR): indicates the average rate of heart beats, in (times / min); RR interval difference root mean square (RMSSD): indicates the root mean square of the difference between adjacent RR intervals, in (ms); respiratory indicators: respiratory rate (RR), in (times / min); respiratory amplitude (RA), in (mm); eye movement indicators: fixation duration (FIXT) mean, that is, the eyes focus on an object until a clear image is obtained in order to obtain road environment information. The time spent on the image is in (ms); the standard deviation of the scan amplitude (SACV), that is, the ratio of the distance of each scan of the eyes to the duration, in (° / s); the percentage of eyes closed (PERCLOS), that is, the proportion of time that the eyes are in a closed state; the mean pupil diameter (PLD), in (mm); driving behavior indicators: steering wheel steering angle retention rate, in (%); vehicle speed retention rate, in (%); external environment indicators: pixel ratios of green plants, sky, buildings, roads, and traffic participants (pedestrians, motor vehicles, and non-motor vehicles).

[0016] Taking the driver's driving time of 5 minutes as a unit, the mean values ​​of the corresponding electrocardiogram, breathing, eye movement, driving behavior and external environment indicators are taken as indicators to characterize the driver's fatigue state in this time period; due to the different dimensions of the data, the min-max normalization method is used to normalize the data.

[0017] The fatigue state classification module is used to construct a fatigue state classification model by using the indicators obtained by the data analysis and processing module as training samples. The driver's physiology, driving behavior and external environment indicators are used as input variables of the model. The state level of the corresponding driver in the training data is obtained through the Karolinska Sleepiness Scale (KSS), which is used as the prediction variable of the model to establish an Attention-GRU fatigue state classification model based on the particle swarm optimization algorithm (PSO).

[0018] The fatigue state real-time detection and evaluation module is used to summarize the driver's physiological, driving behavior and vehicle external environment indicators collected and calculated in real time by the data acquisition module and the data analysis and processing module, and transmit them to the trained fatigue state classification model to obtain the driver's fatigue state level fed back by the fatigue state classification module.

[0019] The fatigue warning module is used to issue warning information according to the fatigue status level obtained by the real-time fatigue status detection and evaluation module. If the driver's fatigue status level is "fatigue" and "severe fatigue", the corresponding warning information will be issued. At the same time, if the driver's fatigue status level is "severe fatigue", the automatic driving module will be activated and the vehicle will take over the automatic driving.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention first uses the collected driver's physiological, driving behavior and external environment information, as well as the driver's Karolinska Sleepiness Scale as training samples to train a driver fatigue state classification model; then, the driver's physiological, driving behavior and external environment indicators are summarized in real time through a fatigue state real-time detection and evaluation module, and the indicators are input into the trained fatigue state classification model to obtain the driver's fatigue state level; the fatigue warning module issues warning information to drivers who are in "fatigue" and "severe fatigue", reminding the driver to adjust his or her sitting posture or stop to rest according to the warning information, and at the same time, the fatigue warning module switches the vehicle to automatic driving to protect the driver's driving safety and avoid collisions due to fatigue. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a module structure diagram of the present invention;

[0022] Figure 2 This is a schematic diagram of a driving simulator;

[0023] Figure 3Schematic diagram of simulated driving scenario;

[0024] Figure 4 This is a schematic diagram of the Attention-GRU network model structure;

[0025] Figure 5 Schematic diagram of the neuron structure of the GRU layer;

[0026] Figure 6 This is a flow chart of the PSO-Attention-GRU algorithm;

[0027] Figure 7 Schematic diagram of the prediction accuracy of the fatigue state classification model. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention, which mainly include two embodiments: 1. Construction of fatigue state classification model; 2. Real-time detection and early warning of fatigue state.

[0029] Example 1: Construction of fatigue state classification model

[0030] Building a high-precision classification model requires rich and comprehensive training samples. In order to obtain a large number of training samples with rich scenarios, this case conducted a simulated driving experiment. By conducting a driving simulation experiment and using the data acquisition module to collect training samples for the model, including driver physiological signals, driving behavior, and external environment information, KSS was used to investigate the subjective fatigue status of the corresponding driver. The driving simulation experiment includes:

[0031] (1) Instruments and equipment: The experiment was carried out using a driving simulator. Figure 2 The hardware includes a Logitech G923 steering wheel, a brake pedal, an accelerator pedal, and a 27-inch monitor with a resolution of 1920×1080 pixels. The traffic scene simulation software is UC-Win / Road. The experimental vehicle is an automatic transmission vehicle, and the entire experimental process is manually driven. The traffic scene is an urban area, including 10 two-way six-lane urban roads ( Figure 3 a) 8 two-way four-lane urban roads ( Figure 3 b) 31 road intersections ( Figure 3 c). The lane width is 3.15m, the traffic volume is 3-10veh / min, the speed limit of 2 roads is 60km / h, the speed limit of 3 roads is 50km / h, and the speed limit of 9 roads is 40km / h.

[0032] (2) Experimental participants: 51 male drivers were recruited from the public. They were aged 23 to 29 years old (average age 26.8 years) and had 4 to 7 years of driving experience (average driving experience 5.3 years). All participants had normal or corrected vision and no chronic or acute respiratory or heart disease.

[0033] (3) Before the experiment, the purpose of the experiment was explained to the subjects, who were asked to fill out a basic questionnaire including information such as age, gender, driving experience, and personal driving habits, and were asked to sign an informed consent form. The subjects were asked to practice driving in front of the simulator for 10 minutes to familiarize themselves with the driving conditions. After the practice, three ECG electrodes were connected under the left and right clavicles and under the left pectoral muscle of the subjects, respectively. The breathing belt was placed at the level of the sternal diaphragm and fully tightened without causing discomfort. At the same time, the WLJ880 steering wheel steering angle detector and vehicle sensors were used to collect the steering wheel steering angle and vehicle speed during driving, and the camera was used to record the vehicle's external environment information from the driver's perspective.

[0034] (4) After the screen was time-calibrated and the subjects were confirmed to be ready, the formal experiment began. Each driving scenario in the experiment lasted about 60 minutes. During the experiment, the staff asked the driver about his current subjective fatigue level every 5 minutes. The driver was required to report his fatigue level with reference to the KSS, as shown in Table 1. After the experiment, all subjects received a reward of 100 yuan.

[0035] Table 1 Karolinska Sleepiness Scale

[0036]

[0037] The data analysis and processing module is used to reduce noise and filter the experimental data, and the missing data is discarded. Among them, the ECG signal is easily disturbed by environmental factors. The original noisy ECG signal is purified by wavelet transform filtering; the wavelet multi-layer noise reduction technology is used to concentrate the noise in the respiratory data at high frequencies, and then it is effectively removed by threshold processing and signal reconstruction; for the eye movement data, it is imported into the Tobii Pro Lab software and segmented according to the established time period, and the data is marked. After all the data is marked, it is exported and stored in Excel format to further extract the subject's gaze, saccade, pupil diameter and other characteristic parameters; semantic segmentation technology is used to extract external environmental features and obtain the pixel ratio of each object in the external environment. Finally, the driver's physiological, driving behavior and external environmental indicators are calculated. The specific indicators include:

[0038] ECG indicators:

[0039] Heart rate (HR): Indicates the average rate of heart beats, measured in (beats / min). The calculation formula is as follows:

[0040]

[0041] Here, heart rate variability is called the RR interval, which refers to the change in the time interval between consecutive heartbeats. It represents the mean RR interval.

[0042] RR interval root mean square difference (RMSSD): represents the root mean square difference of adjacent RR intervals, in milliseconds. The calculation formula is as follows:

[0043]

[0044] In the formula, RR i+1 and RR i They represent the i+1th and ith RR intervals respectively; N represents the number of RR intervals.

[0045] Respiratory indicators: respiratory rate (RR), unit is (times / min); respiratory amplitude (RA), unit is (mm).

[0046] Eye movement indicators: mean value of fixation duration (FIXT), that is, the time it takes for the eyes to focus on an object until a clear image is obtained in order to obtain road environment information, in milliseconds; standard deviation of saccadic amplitude (SACV), that is, the ratio of the distance and duration of each saccade of the eyes, in degrees per second; percentage of eyes closure (PERCLOS), that is, the proportion of time that the eyes are in a closed state; mean value of pupil diameter (PLD), in millimeters.

[0047] Driving behavior indicators:

[0048] The steering wheel steering angular velocity retention rate is calculated as follows:

[0049]

[0050] In the formula, ω t is the steering wheel angle at the tth second.

[0051] The vehicle speed maintenance rate is calculated as follows:

[0052]

[0053] In the formula, v t is the vehicle speed at the tth second.

[0054] External environment indicators: pixel ratios of green plants, sky, buildings, roads, and traffic participants (pedestrians, motor vehicles, and non-motor vehicles).

[0055] Taking the driver's driving time of 5 minutes as a unit, the mean values ​​of the corresponding electrocardiogram, breathing, eye movement, driving behavior and external environment indicators are taken as indicators to characterize the driver's fatigue state in this time period; due to the different dimensions of the data, the min-max normalization method is used to normalize the data.

[0056] The fatigue state classification module was used to train the PSO-Attention-GRU fatigue state classification model. The Karolinska Sleepiness Scale (KSS) was used to evaluate the subjective wakefulness of the corresponding drivers in the training data. The collected KSS was used as the basis for judging the fatigue state, and the driver's state was divided into three levels: awake (KSS=1~5), fatigue (KSS=6~7) and severe fatigue (KSS=8~9), which were quantified as 1, 2, and 3 respectively and used as the predictor variables of the model. The training data obtained by the data analysis and processing module: driver physiology, driving behavior and external environment indicators were used as the input variables of the model to establish the PSO-Attention-GRU fatigue state classification model.

[0057] The Attention-GRU network structure is as follows Figure 4 The model consists of an input layer, a GRU layer, an Attention layer, a fully connected layer, and an output layer. The neuron structure of the GRU layer is as follows: Figure 5 As shown. GRU contains memory unit C t , Update gate Z t and reset gate R t , where the memory unit is used to process the input data and update the hidden state at each time step. The update gate determines how much information of the hidden state at the previous moment needs to be retained to the current moment, and the reset gate determines how much input information at the previous moment needs to be ignored. Let X t is the input at time t, H t is the output of the hidden layer at time t, and the information transmission process within the unit is as follows:

[0058] Z t =σ(W XZ X t +W HZ H t-1 )(5)

[0059] R t =σ(W XR X t +W HR H t-1 )(6)

[0060]

[0061] Where σ is the sigmoid function; act is the activation function of GRU; W XZ, W HZ , W XR , W HR , W XC , W HC is the weight matrix.

[0062] The embodiment of the present application uses the Attention mechanism to assign weights to the hidden layer states of the GRU network, so that the fatigue state classification model adaptively focuses on features that better reflect the fatigue state, thereby improving the prediction accuracy of the model. The fully connected layer uses the Softmax activation function to map the output of the neuron in the [0,1] interval, convert it into a probability output, and obtain the probability values ​​of different fatigue states.

[0063] In addition, the particle swarm optimization algorithm (PSO) can be introduced to optimize the parameters of the Attention-GRU model (model parameters include: learning rate, number of neurons in the GRU hidden layer, and L2 regularization coefficient) to improve the classification accuracy of the Attention-GRU model. The PSO-Attention-GRU algorithm process is as follows: Figure 6 As shown. The particle swarm optimization algorithm (PSO) originates from the study of bird predation behavior. The PSO algorithm converts each solution of the problem to be optimized into a particle swarm containing N particles in an M-dimensional space. In each iteration, the particles move according to the current individual optimal position and the group optimal position, and finally obtain the optimal solution. The speed and position of the particles are calculated by equations (9) and (10).

[0064]

[0065] In the formula, i is the number of particles in each iteration; k is the current iteration number; ω is the inertia coefficient; the moving speed of the particle is v i ; The current position of the particle is x i ; is the optimal position of an individual; is the optimal position of the group; r 1 and r 2 is a random number between (0,1); c 1 and c 2 is the acceleration factor.

[0066] 70% of the collected data set is used as the training sample set, and 30% is used as the test sample set. The optimizer uses the Adam optimization algorithm, the loss function uses the cross entropy, and the model iteration number is 150. The number of particles in the particle swarm is set to 5, the number of evolutions is set to 10, and the dimension of each particle is 3, which represent the parameters to be optimized: learning rate, number of neurons in the GRU hidden layer, and L2 regularization coefficient. At the same time, the learning factor is set to c 1 =1.5, c 2=1.7, and the inertia coefficient is set to 0.8. The accuracy of the model in classifying the test set is: the recognition rates of awake state (fatigue level 1), fatigue state (fatigue level 2) and severe fatigue state (fatigue level 3) are 98.6%, 91.7% and 98.0% respectively, and the total accuracy reaches 96.1957%. The prediction results are shown in Figure 7 From the model results, it can be seen that the model has a high classification accuracy for the awake state (fatigue level 1) and the severe fatigue state (fatigue level 3), while the classification accuracy for the fatigue state (fatigue level 2) is the lowest, which may be related to the driver's self-awakening regulation when the driver is in the fatigue state (fatigue level 2).

[0067] Thus, a fatigue state classification model with higher accuracy is obtained.

[0068] Example 2: Real-time detection and early warning of fatigue status

[0069] The data acquisition module is used to collect the driver's physiological signals, driving behavior and external environment information in real time:

[0070] An ECG sensor with an operating frequency of 200 Hz is used to collect the driver's ECG;

[0071] A breathing sensor with an operating frequency of 50 Hz is used to collect the driver's breathing signal;

[0072] Eye tracker Tobii Pro X2-30, used to collect driver’s eye movement signals;

[0073] WLJ880 steering wheel angle detector, used to collect the vehicle steering wheel angle;

[0074] On-vehicle sensors to collect vehicle speed;

[0075] The camera is placed at eye level with the driver sitting in the car to record the vehicle's external environment information from the driver's perspective.

[0076] The data analysis and processing module is used to reduce noise and filter the data obtained by the data acquisition module, and the missing data is discarded. Among them, the ECG signal is easily disturbed by environmental factors, and the original noisy ECG signal is purified by wavelet transform filtering; the wavelet multi-layer noise reduction technology is used to concentrate the noise in the respiratory data on the high frequency, and then it is effectively removed by threshold processing and signal reconstruction; for the eye movement data, it is imported into the Tobii Pro Lab software and segmented according to the established time period, and the data is marked. After all the data are marked, they are exported and stored in Excel format to further extract the subject's gaze, saccade, pupil diameter and other characteristic parameters; semantic segmentation technology is used to extract external environmental features and obtain the pixel ratio of objects in the external environment. Finally, the driver's physiological, driving behavior and external environment indicators were calculated: heart rate (HR), root mean square difference of RR interval (RMSSD); respiratory rate (RR), respiratory amplitude (RA); mean of fixation duration (FIXT), standard deviation of saccade amplitude (SACV), percentage of closed eyes (PERCLOS), mean of pupil diameter (PLD); steering wheel steering angle retention rate, vehicle speed retention rate; pixel ratios of green plants, sky, buildings, roads, and traffic participants (pedestrians, motor vehicles and non-motor vehicles). Due to the different dimensions of the data, the data were normalized using the min-max normalization method.

[0077] The real-time fatigue state detection and evaluation module is used to summarize the driver's physiological, driving behavior and vehicle external environment indicators collected and calculated in real time by the data acquisition module and the data analysis and processing module, and input them into the trained fatigue state classification model to obtain the fatigue state level fed back by the fatigue state classification module.

[0078] The fatigue warning module is used to issue warning information based on the fatigue status level obtained by the real-time fatigue status detection and evaluation module. If the driver's fatigue status level is "awake", the vehicle drives normally and no information will be issued. If the driver's fatigue status level is "fatigue", the vehicle voice system will announce "You are currently in a state of fatigue, please adjust your sitting position or stop to rest if necessary." If the driver's fatigue status level is "severe fatigue", the vehicle voice system will announce "You are currently in a state of severe fatigue, the vehicle has turned on the automatic driving mode, if you need to switch to manual driving, please lightly step on the brakes", and the vehicle will start the automatic driving module.

[0079] The present invention realizes real-time collection and processing of driver's physiological signals, driving behavior and external environment information through the above-mentioned data acquisition module, data analysis and processing module, fatigue status classification module, fatigue status real-time detection and evaluation module, and fatigue warning module, and provides fatigue warning by evaluating the driver's fatigue status level.

[0080] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for detecting fatigue of an operating driver based on multi-source information fusion, characterized in that: include: The data acquisition module consists of a driver physiological signal data acquisition module, a driving behavior data acquisition module, and an external environment data acquisition module; Driver physiological signal data acquisition module: used to collect driver physiological signals, including ECG, respiration and eye movement data; Driving behavior data collection module: used to collect vehicle steering wheel angle and vehicle speed; External environment data acquisition module: used to collect vehicle external environment information within the driver's field of view; Data analysis and processing module: used to perform noise reduction, filtering and calculation on the data collected by the data acquisition module to obtain the driver's physiological, driving behavior and external environment indicators; and use semantic segmentation technology to extract external environment features to obtain the pixel ratio of each object in the external environment; Fatigue state classification module: This module uses the indicators obtained by the data analysis and processing module as training samples to build a fatigue state classification model; the driver's physiological, driving behavior and external environment indicators are used as input variables of the model, and the state level of the corresponding driver in the training data is obtained through the Karolinska Sleepiness Scale (KSS). It is used as the prediction variable of the model to establish an Attention-GRU fatigue state classification model based on the particle swarm optimization algorithm (PSO); Fatigue status real-time detection and evaluation module: used to summarize the indicators collected and calculated in real time by the data acquisition module and the data analysis and processing module, and transmit them to the trained fatigue status classification model to obtain the driver's fatigue status level fed back by the fatigue status classification module; Fatigue warning module: Receives the fatigue status level obtained by the real-time fatigue status detection and evaluation module, issues corresponding warning information according to the fatigue status level, and starts the vehicle's automatic driving takeover when necessary.

2. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The driver physiological signal data acquisition module comprises: An ECG sensor with an operating frequency of 200 Hz is used to collect the driver's ECG; A breathing sensor with an operating frequency of 50 Hz is used to collect the driver's breathing signal; The eye tracker Tobii Pro X2-30 is used to collect the driver’s eye movement signals.

3. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The driving behavior data collection module includes: WLJ880 steering wheel angle detector, used to collect the vehicle steering wheel angle; On-board sensor to collect vehicle speed.

4. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The external environment data acquisition module includes: The camera is placed at eye level with the driver to record the vehicle's external environment information from the driver's perspective.

5. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The data analysis and processing module includes: The data obtained by the data acquisition module is subjected to noise reduction processing, and the missing data is discarded. Among them, the ECG signal is easily disturbed by environmental factors, and the original noisy ECG signal is purified by wavelet transform filtering method; the wavelet multi-layer noise reduction technology is used to concentrate the noise in the respiratory data on the high frequency, and then it is effectively removed by threshold processing and signal reconstruction; for the eye movement data, it is imported into the Tobii Pro Lab software and segmented according to the established time period and marked. After the data is marked, it is exported and stored in Excel format to further extract the subject's gaze, saccade, pupil diameter and other characteristic parameters; semantic segmentation technology is used to extract external environmental features, and the pixel ratio of each object in the external environment is obtained. Finally, the driver's physiological, driving behavior and external environmental indicators are calculated. The specific indicators are as follows: ECG indicators: heart rate (HR), which indicates the average rate of heart beats, in (beats / min); root mean square difference of RR intervals (RMSSD), which indicates the root mean square difference of adjacent RR intervals, in (ms); Respiratory indexes: respiratory rate (RR), in (times / min); respiratory amplitude (RA), in (mm); Eye movement indicators: mean value of fixation duration (FIXT), i.e. the time it takes for the eyes to focus on an object until a clear image is obtained in order to obtain road environment information, in milliseconds; standard deviation of saccadic amplitude (SACV), i.e. the ratio of the distance and duration of each saccade of the eyes, in degrees per second; percentage of eye closure (PERCLOS), i.e. the proportion of time that the eyes are closed; mean value of pupil diameter (PLD), in millimeters; Driving behavior indicators: steering wheel angle retention rate, unit is (%); vehicle speed retention rate, unit is (%); External environment indicators: pixel ratios of green plants, sky, buildings, roads, and traffic participants (pedestrians, motor vehicles, and non-motor vehicles); Taking the driver's driving time of 5 minutes as a unit, the mean values ​​of the corresponding electrocardiogram, breathing, eye movement, driving behavior and external environment indicators are taken as indicators to characterize the driver's fatigue state in this time period; due to the different dimensions of the data, the min-max normalization method is used to normalize the data.

6. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The fatigue state classification module comprises: The driver's physiological, driving behavior and external environmental indicators calculated by the data analysis and processing module are used as training samples to build a fatigue state classification model, and the Karolinska Sleepiness Scale (KSS) is used to evaluate the subjective wakefulness of the corresponding driver in the training data; the driver is asked about his fatigue state every 5 minutes of driving, and the collected KSS is used as the basis for judging the fatigue state. The driver's state is divided into three levels: awake (KSS=1-5), fatigue (KSS=6-7) and severe fatigue (KSS=8-9), which are quantified by 1, 2, and 3 respectively, and used as the prediction variables of the model; the training data obtained by the data analysis and processing module: driver's physiological, driving behavior and external environmental indicators are used as input variables of the model, and an Attention-GRU fatigue state classification model based on the particle swarm optimization algorithm (PSO) is established; The Gated Recurrent Unit (GRU) neural network is proposed by merging the forget gate and input gate in LSTM into an update gate on the basis of solving the gradient explosion and gradient disappearance in the RNN network. Compared with the LSTM network, the GRU network requires fewer parameters, takes less time, and is more suitable for solving real-time time series problems. The Attention mechanism is used to focus on important features in time series data and give greater weight to key information. The Attention mechanism is used to assign weights to the hidden layer states of the GRU network, so that the fatigue state classification model can adaptively focus on features that better reflect the fatigue state, thereby improving the prediction accuracy of the model. In addition, the Attention-GRU model parameters (model parameters include: learning rate, number of neurons in the GRU hidden layer, and L2 regularization coefficient) can be optimized by introducing the particle swarm optimization algorithm (PSO) to improve the classification accuracy of the Attention-GRU model. The particle swarm optimization algorithm (PSO) originated from the study of bird predation behavior. Each solution to the problem to be optimized is converted into a particle swarm containing N particles in M-dimensional space. In each iteration, the particles move according to the current individual optimal position and the group optimal position, and finally the optimal solution is obtained.

7. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The real-time detection and assessment module of fatigue status includes: The driver's physiological, driving behavior and vehicle external environment indicators collected and calculated in real time by the data acquisition module and the data analysis and processing module are summarized and transmitted to the trained fatigue state classification model to obtain the driver's fatigue state level fed back by the fatigue state classification module.

8. The method for detecting fatigue of an operating driver based on multi-source information fusion according to claim 1, characterized in that: The fatigue warning module includes: Warning information will be issued based on the fatigue status level obtained by the real-time fatigue status detection and evaluation module. If the driver's fatigue status level is "fatigue" and "severe fatigue", the corresponding warning information will be issued. At the same time, if the driver's fatigue status level is "severe fatigue", the automatic driving module will be started and the vehicle will take over the automatic driving.

Images