Driver state evaluation method and system based on multi-mode signal monitoring

Abstract

The invention relates to the technical field of vehicles, and discloses a driver state evaluation method and system based on multi-mode signal monitoring, electronic equipment, a storage medium and a vehicle, and the method comprises the steps that multi-mode signals of a driver are collected in real time, and the multi-mode signals comprise video signals, audio signals, electrocardiosignals and vehicle state signals; the multi-modal signals are preprocessed, key features are extracted, multi-modal features are obtained, and the multi-modal features comprise driver and driving behavior features; the multi-modal features are fused to obtain fused features, and the fused features are used for describing the state of the driver; and inputting the fusion feature into a preset model to divide an abnormal state of the driver, and when the abnormal state exceeds a preset condition, correspondingly reminding the driver. The state of the driver can be detected in a multi-dimensional mode, the accuracy of state evaluation is improved, reminding can be correspondingly carried out, and driving safety is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a driver status assessment method, system, electronic equipment, storage medium and vehicle based on multimodal signal monitoring. Background Art

[0002] At present, existing driver status monitoring mainly obtains the driver's physiological or behavioral information through a single sensor or device, such as a camera, heart rate sensor, etc., and then analyzes it through a specific algorithm to determine the driver's status.

[0003] For example, some technologies determine whether the driver is tired or distracted by analyzing visual information such as the driver's facial expressions and eye movements; some technologies assess the driver's health by monitoring physiological information such as the driver's heart rate and blood pressure. However, existing driver status monitoring technologies usually rely on only a single sensor or device, and the information obtained is limited, which may lead to insufficient accuracy and reliability of status detection. At the same time, when processing multi-source data, it is often impossible to effectively fuse and utilize information from different modalities, which may lose some important information and affect the results of driving status detection. Finally, after discovering abnormal driving status, existing technologies often fail to take effective intervention measures in a timely manner to ensure driving safety.

[0004] Therefore, the present application provides a driver status assessment method based on multimodal signal monitoring to solve the above technical problems. Summary of the invention

[0005] The purpose of the present invention is to provide a driver status assessment method, system, electronic device, storage medium and vehicle based on multimodal signal monitoring, so as to solve the technical problems in the prior art that the status detection information is relatively single, multi-source data is easily lost, and the accuracy of status detection is insufficient.

[0006] In order to solve the above technical problems, the present invention provides a driver status assessment method based on multimodal signal monitoring, comprising:

[0007] The signal acquisition step includes acquiring multimodal signals of the driver in real time, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals, and vehicle status signals;

[0008] The feature extraction step includes preprocessing the multimodal signal and extracting key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features;

[0009] a feature fusion step, comprising fusing the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state;

[0010] The state evaluation step includes inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, giving a corresponding reminder to the driver.

[0011] In some specific embodiments, the signal acquisition step includes real-time acquisition of multimodal signals of the driver, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals, and vehicle status signals, and further includes:

[0012] The video signal acquisition includes capturing the driver's facial and motion signals in real time;

[0013] The audio signal collection includes capturing the driver's voice instructions and conversation content in real time;

[0014] The electrocardiogram signal collection includes real-time monitoring of the driver's electrocardiogram signal;

[0015] The vehicle status signal collection includes real-time reading of vehicle speed, accelerator pedal, brake pedal and steering wheel angle signals.

[0016] In some specific embodiments, the feature extraction step includes preprocessing the multimodal signal and extracting key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features, and further includes:

[0017] Preprocessing the video signal and extracting key features, including extracting a frontal face view, performing histogram equalization, resolution scaling, and facial feature point detection based on the frontal face view, and calculating head posture and eye and mouth activity features;

[0018] Preprocessing the electrocardiogram signal and extracting key features, including noise reduction and heart rate variability calculation, to obtain characteristic parameters of cardiac activity;

[0019] The vehicle state signal is preprocessed and key features are extracted, including calculating characteristic parameters of the driving behavior and analyzing driving habits.

[0020] In some specific embodiments, the feature fusion step includes fusing the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state, and further includes:

[0021] splicing the multimodal features into multidimensional vectors in sequence;

[0022] The multidimensional vector is subjected to dimensionality reduction processing based on the Pearson correlation coefficient method and the principal component analysis method to obtain the fusion feature.

[0023] In some specific embodiments, the state assessment step includes inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, giving a corresponding reminder to the driver, further including:

[0024] Input the fusion features into a pre-trained LSTM model for state evaluation and classification of the abnormal state, wherein the abnormal state includes a fatigue state, a distracted state, and an angry state;

[0025] When the abnormal state of the driver exceeds the preset condition, the driver is reminded in at least one of visual, tactile, olfactory or auditory ways.

[0026] In some specific embodiments, configuration and model training steps are also included:

[0027] Configuring real-time acquisition, preprocessing, feature extraction and state assessment tasks of the multimodal signals;

[0028] The LSTM model is trained based on the labeled driver status data and vehicle status data, and the generalization ability and accuracy of the LSTM model are evaluated by cross-validation method;

[0029] The trained LSTM model is deployed for real-time status evaluation.

[0030] Based on the same concept, the present invention also provides a driver status assessment system based on multimodal signal monitoring, comprising:

[0031] A signal acquisition module, configured to acquire multimodal signals of the driver in real time, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals, and vehicle status signals;

[0032] A feature extraction module is configured to preprocess the multimodal signal and extract key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features;

[0033] a feature fusion module, configured to fuse the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state;

[0034] The state assessment module is configured to include inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, the driver is reminded accordingly.

[0035] Based on the same concept, the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a driver status assessment method based on multimodal signal monitoring.

[0036] Based on the same concept, the present invention also provides a computer-readable storage medium, which stores a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a driver status assessment method based on multimodal signal monitoring.

[0037] Based on the same concept, the present invention also provides a vehicle, wherein the vehicle is provided with the driver status assessment system based on multimodal signal monitoring as described above.

[0038] Compared with the prior art, the beneficial effects are:

[0039] The present invention discloses a driver status assessment method, system, electronic device, storage medium and vehicle based on multimodal signal monitoring, which can detect the driver status in multiple dimensions, improve the accuracy of status assessment, and give corresponding reminders to ensure driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a driver state assessment method based on multimodal signal monitoring in some specific embodiments of the present invention;

[0041] Figure 2 It is a schematic diagram of a signal acquisition process of a driver status assessment method based on multimodal signal monitoring in some applications of the present invention;

[0042] Figure 3 It is a schematic diagram of the overall process of a driver state assessment method based on multimodal signal monitoring in some applications of the present invention;

[0043] Figure 4 It is a schematic diagram of the structure of a driver state assessment system based on multimodal signal monitoring in some specific embodiments of the present invention;

[0044] Figure 5 It is a schematic diagram of the structure of an electronic device in some specific embodiments of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0047] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0049] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0050] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0051] It should be particularly noted that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0052] Reference Figure 1 , a driver state assessment method based on multimodal signal monitoring, comprising:

[0053] S101, a signal collection step, comprising collecting multimodal signals of the driver in real time, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals and vehicle status signals;

[0054] It is understood that a camera is installed inside the vehicle, such as above the dashboard or on the driver's side. The camera resolution is set to 1920x1080 pixels and the frame rate is set to 30fps to ensure the clarity and continuity of the video signal. A microphone array is installed inside the vehicle, such as above the driver's head or near the steering wheel. The microphone sampling rate is set to 44.1kHz and the bit depth is set to 16bit to capture audio details. Provide the driver with a wearable ECG monitoring device (such as a smart watch or chest strap) with a built-in PPG sensor for real-time monitoring of the driver's ECG signal. Set the sampling rate of the PPG sensor to 100Hz to accurately capture subtle changes in the ECG signal. Use the vehicle's built-in CAN bus interface to access the vehicle control system and read the vehicle status information in real time. Set the frequency of CAN bus data acquisition to 10Hz to obtain enough vehicle status data points for analysis. The collected data includes key parameters such as vehicle speed, accelerator pedal position, brake pedal status, and steering wheel angle.

[0055] In some of the applications, the signal acquisition step includes real-time acquisition of the driver's multimodal signals, which include video signals, audio signals, ECG signals and vehicle status signals. Video signal acquisition includes real-time capture of the driver's facial and motion signals; audio signal acquisition includes real-time capture of the driver's voice commands and conversation content; ECG signal acquisition includes real-time monitoring of the driver's ECG signals; vehicle status signal acquisition includes real-time reading of vehicle speed, accelerator pedal, brake pedal and steering wheel angle signals.

[0056] It is understandable that the driver's facial expressions and movements will change during driving. These changes are captured in real time, including the driver's blinking, yawning, head turning and other movements, as well as facial micro-expression changes. The driver will issue voice commands, talk to passengers or answer calls while driving. These sound signals are captured in real time, including the driver's voice content, tone changes and possible background noise. The driver may feel tired after driving for a long time, and the driver's ECG signal changes are captured in real time, including features such as increased heart rate variability (HRV) and decreased heart rate. Vehicle status information is read in real time, including key parameters such as vehicle speed (unit: km / h), accelerator pedal position (unit: %), brake pedal status (0 is not pressed, 1 is pressed), and steering wheel angle (unit: degree).

[0057] S102, a feature extraction step, comprising preprocessing the multimodal signal and extracting key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features;

[0058] It can be understood that in the preprocessing stage, image filtering is used to remove noise and interference in the video signal to improve the image quality. The face detection algorithm (such as Haar feature + AdaBoost, HOG + SVM) is used to detect and locate the driver's facial area from the video frame. The detected facial area is aligned through affine transformation to eliminate the influence of head posture changes on feature extraction. Spectral subtraction and adaptive filtering are used to remove background noise in the audio signal to improve speech clarity. The quality of the speech signal is further improved through speech enhancement algorithms (such as Wiener filtering). Energy threshold, zero crossing rate and other features are used to detect speech endpoints and determine the effective interval of the speech signal. A digital filter is used to remove high-frequency noise and baseline drift in the ECG signal. The R wave detection algorithm (such as Pan-Tompkins algorithm) is used to locate the R wave position in the ECG signal. Outliers and missing values ​​in the vehicle status signal are removed to ensure the integrity and accuracy of the data. The vehicle status signal (such as vehicle speed, accelerator pedal position, etc.) is normalized to the same dimension.

[0059] The driver's facial features, such as eye closure degree, blinking frequency, facial expressions, etc., are extracted from the preprocessed video signals to assess the driver's fatigue level and emotional state.

[0060] The driver's voice features, such as tone, volume, speaking speed, etc., are extracted from the preprocessed audio signal to assess the driver's emotional state and attention level.

[0061] Features such as heart rate and heart rate variability are extracted from the preprocessed ECG signals to assess the driver's physiological state and fatigue level.

[0062] Features such as vehicle speed change rate, accelerator pedal operation frequency, and brake pedal usage are extracted from the vehicle status signal to evaluate the driver's driving style and driving skills.

[0063] By combining vehicle status signals and video signals (such as lane line recognition), features such as the vehicle's lateral offset and lane keeping ability are extracted to evaluate the driver's driving stability and safety.

[0064] In some of the applications, the feature extraction step includes preprocessing the multimodal signal and extracting key features to obtain multimodal features. The multimodal features include driver and driving behavior characteristics. The video signal is preprocessed and key features are extracted, including extracting the front face view, and based on the front face view, histogram equalization, resolution scaling, and facial feature point detection are performed, and the head posture and eye and mouth activity characteristics are calculated; the electrocardiogram signal is preprocessed and key features are extracted, including noise reduction and heart rate variability calculation to obtain characteristic parameters of heart activity; the vehicle status signal is preprocessed and key features are extracted, including calculating characteristic parameters of driving behavior and analyzing driving habits.

[0065] It can be understood that, first, the facial area of ​​the driver is located from the real-time captured video frame. The detected facial area is adjusted to a frontal view to reduce the impact of head posture changes on subsequent feature extraction. The facial front view is histogram equalized to enhance the contrast of the image. As required, the facial front view image is scaled to a uniform resolution (such as 128x128 pixels). A facial feature point detection algorithm (such as the 68-point facial feature point detection of the D lib library) is used to detect feature points of key parts such as eyes, nose, and mouth on the facial front view. Based on the detected facial feature points, the pitch angle, yaw angle, and roll angle of the head are calculated to evaluate the driver's head posture. According to the position changes of the eye and mouth feature points, features such as the degree of eye closure, blinking frequency, and degree of mouth opening and closing are calculated to evaluate the driver's fatigue and emotional state.

[0066] For example, in a video, the driver's face view was detected and the image contrast was enhanced by histogram equalization. The facial image was scaled to 128x128 pixels and 68 facial feature points were detected. Based on these feature points, the driver's head posture was calculated to be slightly forward, and the blinking frequency was monitored to increase to 15 times per minute (the normal blinking frequency is 10-15 times per minute). Combined with other features, it was judged that the driver might be in a state of fatigue.

[0067] A digital filter (such as a bandpass filter) is used to reduce the noise of the original ECG signal. The R wave detection algorithm is used to locate the R wave position in the ECG signal and calculate the time interval (RR interval) between adjacent R waves. Based on the change of the RR interval, the heart rate variability (HRV) is calculated as an important parameter for evaluating the cardiac autonomic nervous regulation function and the driver's physiological state.

[0068] For example, the R-wave position is detected and a heart rate variability indicator of 50 milliseconds is calculated (lower HRV may indicate reduced autonomic nervous system regulation or a stressed driver).

[0069] Remove outliers and missing values ​​from vehicle status signals to ensure data integrity and accuracy. Based on vehicle status signals (such as vehicle speed, accelerator pedal position, brake pedal status, etc.), calculate characteristic parameters such as acceleration, deceleration, and throttle operation frequency to evaluate the driver's driving behavior. Combined with long-term collected vehicle status data, analyze the driver's driving habits, such as whether they frequently change lanes and whether they maintain a safe distance.

[0070] S103, a feature fusion step, comprising fusing the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state;

[0071] In some of the applications, the feature fusion step includes fusing multimodal features to obtain fused features, which are used to describe the driver's status, and concatenating the multimodal features into multidimensional vectors in sequence; and performing dimensionality reduction processing on the multidimensional vectors based on the Pearson correlation coefficient method and the principal component analysis method to obtain the fused features.

[0072] It is understandable that the obtained multimodal features are spliced. The features of each modality are expressed as vectors or matrices of different dimensions. In this embodiment, it is assumed that the facial features and driving behavior features extracted from the video signal are encoded as N1-dimensional vectors, the features such as heart rate variability extracted from the electrocardiogram signal are encoded as N2-dimensional vectors, and the driving behavior features extracted from the vehicle state signal are encoded as N3-dimensional vectors.

[0073] These multimodal features are sequentially concatenated into a multidimensional vector, denoted as X, whose dimension is N = N1 + N2 + N3. For example, if the facial features are 100-dimensional, the heart rate variability features are 20-dimensional, and the driving behavior features are 30-dimensional, then the concatenated multidimensional vector X is 150-dimensional.

[0074] For example, the facial feature vector V extracted from the video signal is [f1,f2,...,f100], the heart rate variability feature vector H extracted from the ECG signal is [h1,h2,...,h20], and the driving behavior feature vector D extracted from the vehicle state signal is [d1,d2,...,d30]. Then the concatenated multidimensional vector X is [f1,f2,...,f100,h1,h2,...,h20,d1,d2,...,d30].

[0075] After obtaining the spliced ​​multidimensional vector X, in order to remove redundant and irrelevant features and improve the efficiency and effect of subsequent processing, the Pearson correlation coefficient method is used for feature screening. The Pearson correlation coefficient is used to measure the degree of linear correlation between two variables, and its value ranges from -1 to 1. In this step, the Pearson correlation coefficient between each dimensional feature in the multidimensional vector X and all other features is calculated, and a threshold value (such as 0.7 or adjusted according to the specific situation) is set to remove features with low correlation with other features.

[0076] For example, after calculation, if the average correlation coefficient between the 50th dimension feature (from the facial feature vector V) and other features is lower than the set threshold, it will be removed from the multidimensional vector X.

[0077] After screening by the Pearson correlation coefficient method, although some redundant features are removed, the dimension of the multidimensional vector may still be high. In order to further reduce the dimension, improve the calculation efficiency, and retain the main information of the original data as much as possible, the principal component analysis (PCA) method is used for dimensionality reduction.

[0078] PCA projects the multidimensional vector into a new low-dimensional space through linear transformation, so that the variance of the projected data in the new space is maximized, that is, the main change direction of the original data is retained. In this step, PCA is performed on the screened multidimensional vector, and the first K principal components (K is determined according to actual needs and the amount of information retained) are selected as fusion features.

[0079] For example, after PCA processing, the first 10 principal components are selected as fusion features, and the final fusion feature is a 10-dimensional vector, which greatly reduces the dimension of the data while retaining the main information of the original data.

[0080] S104, a state evaluation step, includes inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, giving a corresponding reminder to the driver.

[0081] In some of the applications, the state assessment step includes inputting the fused features into a preset model to classify the driver's abnormal state, and when the abnormal state exceeds the preset condition, the driver is reminded accordingly, and the fused features are input into a pre-trained LSTM model for state assessment and classification of abnormal states, which include fatigue state, distraction state and anger state; when the driver's abnormal state exceeds the preset condition, the driver is reminded in at least one of visual, tactile, olfactory or auditory ways.

[0082] It can be understood that the obtained fused features are passed as input to the pre-trained LSTM (Long Short-Term Memory Network) model.

[0083] For example, the fusion feature contains a 10-dimensional vector that integrates multimodal information such as video signals, ECG signals, and vehicle status signals.

[0084] After receiving the fused features, the LSTM model learns and infers through its internal network structure, and outputs an evaluation result of the current driver state. In this embodiment, the LSTM model is trained to identify and classify three main abnormal states: fatigue state, distracted state and angry state.

[0085] Fatigue state: usually judged by detecting eye activities (such as blinking frequency, eye closing time, etc.) and head posture (such as nodding frequency).

[0086] Distracted state: Identified through facial expressions, eye direction, and sudden changes in driving behavior (such as sudden braking or acceleration).

[0087] Angry state: Assessed by dramatic changes in facial expression, vocal characteristics (if available), and aggressiveness of driving behavior.

[0088] For example, the LSTM model outputs a vector containing three probability values, corresponding to the probabilities of fatigue state, distracted state and angry state. For example, the output vector is [0.05, 0.85, 0.10], which means that the current driver is most likely to be in a distracted state.

[0089] According to the output of the LSTM model, it is determined whether the driver is in an abnormal state, and whether the abnormal state exceeds the preset conditions (such as the probability threshold of a certain abnormal state). If the abnormal state exceeds the preset conditions, the reminder mechanism is triggered.

[0090] For example, the preset condition is that the probability of any abnormal state exceeds 0.7. In the above example, the probability of the distracted state (0.85) exceeds this threshold, so it is determined that the driver is currently in a distracted state.

[0091] When the abnormal state of the driver exceeds the preset condition, the driver is reminded by at least one of vision, touch, smell or hearing. The choice of reminder method can be set according to the actual situation and the driver's preference.

[0092] Visual reminder: Display warning messages or flashing lights on the vehicle's display screen.

[0093] Haptic reminder: reminds the driver through steering wheel vibration or seat vibration.

[0094] Olfactory reminder: Reminders can be assisted by releasing specific scents (such as air fresheners).

[0095] Auditory reminder: A warning sound or voice prompt is played through the car audio.

[0096] For example, in the above-mentioned distracted state example, visual and auditory reminders can be triggered at the same time, that is, a warning message "Do not drive distracted" is displayed on the car display screen, and a voice prompt "Please drive safely" is played through the car audio.

[0097] Some of these applications also include configuration and model training steps, configuring real-time acquisition, preprocessing, feature extraction and state assessment tasks of multimodal signals; training the LSTM model based on labeled driver state data and vehicle state data, and evaluating the generalization ability and accuracy of the LSTM model through cross-validation; and deploying the trained LSTM model for real-time state assessment.

[0098] It can be understood that: the camera is configured to collect video signals at a speed of 30 frames per second, and the face detection algorithm is set to detect the driver's facial features once per second. The ECG monitor is configured to collect ECG signals at a sampling frequency of 1000 times per second, and a bandpass filter is applied to remove noise. The feature extraction cycle is set to once every 5 seconds, and features such as blinking frequency and expression changes are extracted from the video signal, heart rate variability features are extracted from the ECG signal, and acceleration, speed and other features are extracted from the vehicle status signal.

[0099] Collect and annotate driver status data and vehicle status data. The annotated data includes samples of normal status and various abnormal statuses, such as fatigue driving, distracted driving, angry driving, etc.

[0100] Based on the labeled data, the LSTM model is used for training. During the training process, multimodal features are used as input and driver status is used as output, and the model parameters are optimized through the back propagation algorithm. Cross-validation methods (such as K-fold cross-validation) are used to evaluate the generalization ability and accuracy of the LSTM model. The data set is divided into K subsets, and K-1 subsets are used as training sets each time, and the remaining subset is used as the validation set. Repeat K times and take the average result as the final evaluation indicator.

[0101] For example, a data set containing 1000 samples is prepared, of which 700 samples are used for training and 300 samples are used for validation. The LSTM model is evaluated using the 5-fold cross validation method, and the average accuracy of the model on the validation set is 95%.

[0102] The trained LSTM model is deployed into the driver status assessment system and integrated with real-time acquisition, preprocessing, feature extraction and other modules to form a complete real-time status assessment process.

[0103] Combine the following Figure 2 and Figure 3 The following describes some examples of the driver status assessment method based on multimodal signal monitoring in some applications of the present invention:

[0104] In this embodiment, the multimodal signal is presented to the terminal through the target driver's video signal, voice signal, PPG ECG signal, and CAN bus signal, so as to realize the multimodal perception of the driver. At the same time, the multimodal signal is preprocessed and key features are extracted to obtain multimodal features; the multimodal features are fused to obtain fusion features; the fusion features are input into the long short-term memory network, and the abnormal state of the driver is divided, such as fatigue state, distraction state, and anger state; if the driver's fatigue state exceeds the preset requirements, the vehicle terminal will actively remind the driver.

[0105] like Figure 2 and Figure 3 As shown, the driver's multimodal signals are collected through vehicle-mounted devices and wearable devices. The multimodal signals include the driver's video signal, the driver's audio signal, the driver's electrocardiogram signal and the vehicle status signal. After the collection is completed, the multimodal signals are transmitted to the processing terminal.

[0106] The driver's actions and facial signals are acquired through the vehicle video monitoring. In this embodiment, the facial signal is a video signal, and the action and facial signal are transmitted to the vehicle-mounted processing terminal through a wiring harness.

[0107] The driver's voice is acquired through a vehicle voice system such as a microphone. In this embodiment, the audio signal is a vehicle signal, and the sound signal is transmitted to the vehicle processing terminal through a wiring harness.

[0108] The driver's ECG signal is obtained through a wearable device. In this embodiment, the driver's ECG signal is measured and obtained by a bracelet with a PPG sensor worn on the driver's wrist, and the ECG signal is transmitted to the vehicle-mounted processing terminal via Bluetooth.

[0109] The vehicle status signal, including vehicle speed, accelerator pedal signal, brake pedal signal and steering wheel angle, is obtained through the CAN bus analyzer, and the vehicle signal is transmitted to the on-board processing terminal through the wiring harness.

[0110] Preprocess the multimodal signal and extract key features to obtain multimodal features:

[0111] Preprocessing and feature extraction of driver video signals: extracting the frontal face view in the driver video signal; performing histogram equalization on the frontal face view; scaling the resolution of the frontal face view of the driver after the histogram equalization process;

[0112] In this embodiment, the resolution of the front view is scaled to 512×512, and is input into a convolutional neural network, which outputs the 2D coordinates of the facial feature points;

[0113] In this embodiment, a total of 468 2D coordinates of facial feature points are output; key points are selected from the 2D coordinates of the feature points, including 6 key points, namely, the outer corner of the eye, the center of the brow, the tip of the nose and the corner of the mouth, and the Euler angles of the head posture are calculated by the PnP algorithm; eye key points are selected, and the eye aspect ratio EAR is calculated;

[0114] In this embodiment, a total of 32 eye key points are screened; mouth key points are screened, and the mouth length-width ratio MAR is calculated;

[0115] In this embodiment, a total of 20 mouth key points are selected; the above steps are repeated to collect the head Euler angles, AR and MAR within the specified duration, and the nodding frequency, eyelid closure PERCLOS and yawn frequency are calculated according to the preset thresholds;

[0116] The reference threshold of the head Euler angle for judging nodding is: the pitch angle is 25-35 degrees. If it is greater than this threshold, it is judged as nodding; the AR reference threshold is 0.2-0.25. If it is lower than this threshold, it is judged as closing the eyes;

[0117] The reference threshold of AR is 0.55-0.65. If it is higher than this threshold, it is judged as an open mouth.

[0118] In this embodiment, the specified duration is 300 seconds. In other implementations, different durations may be specified according to different requirements.

[0119] Preprocessing and feature extraction of driver's ECG signal. Collect the driver's ECG signal within a specified duration (300s), perform noise reduction on the ECG signal through wavelet transform and threshold analysis, and calculate the heart rate variability HRV;

[0120] Preprocessing and feature extraction of vehicle status signals: Collect vehicle status signals within a specified duration (300s), which include vehicle speed, accelerator pedal signal, brake pedal signal and steering wheel angle; calculate vehicle acceleration, steering wheel angular velocity, steering wheel angular acceleration average, steering wheel angular acceleration standard deviation, braking frequency and acceleration frequency within the specified duration.

[0121] The multimodal features are fused to obtain the fused features:

[0122] The multimodal features are sequentially spliced ​​into a multidimensional vector; that is, the nodding frequency, eyelid closure PERCLOS, yawning frequency, heart rate variability HRV, vehicle acceleration, steering wheel angular velocity, steering wheel angular acceleration average, steering wheel angular acceleration standard deviation, braking frequency and acceleration frequency are sequentially spliced ​​into a multidimensional vector.

[0123] Based on the Pearson correlation coefficient method and principal component analysis method, the multidimensional vector is reduced in dimension to obtain the fusion features.

[0124] Input the fused features into the long short-term memory network and classify the driver's abnormal state;

[0125] In this embodiment, the abnormal state of the driver is divided into three states: fatigue state, distraction state, and anger state;

[0126] If the driver's abnormal state exceeds the preset requirements, the on-board equipment will remind the driver through the alarm device;

[0127] In this embodiment, if the driver's fatigue state is set to be one of fatigue state, distraction state, and anger state, active reminder service is provided to the driver through visual, tactile, olfactory, visual and other means.

[0128] This embodiment is described below in conjunction with application scenarios:

[0129] Embodiment 1: Through the target driver's video signal, voice signal, PPG ECG signal, and CAN bus signal, the multimodal signal is transmitted to the terminal to realize the multimodal perception of the driver. For example, the driver's real-time video signal is obtained through the camera, the driver's voice signal is obtained through the microphone, the driver's PPG ECG signal is obtained through the heart rate monitoring device, and the vehicle status signal is obtained through the vehicle CAN bus, so as to accurately obtain and understand the driving environment information.

[0130] Embodiment 2: Signal preprocessing and key feature extraction: Preprocess multimodal signals and extract key features to obtain multimodal features; fuse multimodal features to obtain fused features. For example, perform face recognition and expression analysis on video signals, perform emotion recognition on voice signals, perform heart rate and heart rate variability analysis on PPG electrocardiogram signals, and perform vehicle status monitoring on CAN bus signals, so as to obtain multimodal features and fuse them.

[0131] Example 3: Abnormal state identification and reminder: The fused features are input into the long short-term memory network (LSTM), and the driver's abnormal state is divided, such as fatigue, distraction, and anger. For example, the driver's multimodal features are analyzed in time series through the LSTM network to identify the driver's abnormal state, and when the driver's fatigue state exceeds the preset requirements, the vehicle terminal actively reminds the driver, such as opening the window for ventilation or voice prompts.

[0132] Embodiment 4: Multimodal feature fusion: This embodiment fuses multimodal features, which has higher state detection accuracy and reliability, stronger data processing and utilization capabilities, and better driving safety assurance capabilities. For example, by fusing multimodal features of video, voice, ECG and vehicle signals, the driver's state can be understood more comprehensively, improving the accuracy and reliability of state detection.

[0133] For the method steps disclosed in the above embodiments, the method steps are described as a series of action combinations for the purpose of simple description, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0134] like Figure 4 As shown, the present invention also provides a driver status assessment system based on multimodal signal monitoring, comprising:

[0135] The signal acquisition module 201 is configured to acquire multimodal signals of the driver in real time, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals and vehicle status signals;

[0136] A feature extraction module 202 is configured to pre-process the multimodal signal and extract key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features;

[0137] A feature fusion module 203 is configured to fuse the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state;

[0138] The state evaluation module 204 is configured to input the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, give a corresponding reminder to the driver.

[0139] It is worth noting that although only some basic functional modules are disclosed in the embodiments of the present invention, it does not mean that the composition of the present system is limited to the above basic functional modules. On the contrary, what the present embodiment wants to express is that on the basis of the above basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the protection scope of the claims of the present invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described in terms of functions and are described separately in various units and modules. Of course, when implementing the present invention, the functions of each unit and module can be implemented in the same or one or more software and / or hardware.

[0140] like Figure 5 As shown, the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of a driver status assessment method based on multimodal signal monitoring.

[0141] Figure 5 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 5 The electronic device provided in the embodiment of the present invention includes: one or more processors 710 and a storage device 720; the processor 710 in the electronic device may be one or more, Figure 5 A processor 710 is taken as an example; the storage device 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement the driver state assessment method based on multimodal signal monitoring as described in any one of the embodiments of the present invention.

[0142] The electronic device may further include: an input device 730 and an output device 740 .

[0143] The processor 710, storage device 720, input device 730 and output device 740 in the electronic device can be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0144] The storage device 720 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the driver status assessment method based on multimodal signal monitoring provided in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the storage device 720, that is, realizes the driver status assessment method based on multimodal signal monitoring in the above method embodiment.

[0145] The storage device 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 720 may further include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0147] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a driver status assessment method based on multimodal signal monitoring.

[0148] Specifically, the computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0149] The present invention also provides a vehicle, wherein the vehicle is provided with the driver status assessment system based on multimodal signal monitoring as described above.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver status assessment method based on multimodal signal monitoring, characterized in that: include: The signal acquisition step includes acquiring multimodal signals of the driver in real time, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals, and vehicle status signals; The feature extraction step includes preprocessing the multimodal signal and extracting key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features; a feature fusion step, comprising fusing the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state; The state evaluation step includes inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, giving a corresponding reminder to the driver.

2. The driver status assessment method based on multimodal signal monitoring according to claim 1 is characterized in that: The signal acquisition step includes real-time acquisition of the driver's multimodal signals, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals, and vehicle status signals, and further includes: The video signal acquisition includes capturing the driver's facial and motion signals in real time; The audio signal collection includes capturing the driver's voice instructions and conversation content in real time; The electrocardiogram signal collection includes real-time monitoring of the driver's electrocardiogram signal; The vehicle status signal collection includes real-time reading of vehicle speed, accelerator pedal, brake pedal and steering wheel angle signals.

3. The driver status assessment method based on multimodal signal monitoring according to claim 1, characterized in that: The feature extraction step includes preprocessing the multimodal signal and extracting key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features, and further includes: Preprocessing the video signal and extracting key features, including extracting a frontal face view, performing histogram equalization, resolution scaling, and facial feature point detection based on the frontal face view, and calculating head posture and eye and mouth activity features; Preprocessing the electrocardiogram signal and extracting key features, including noise reduction and heart rate variability calculation, to obtain characteristic parameters of cardiac activity; The vehicle state signal is preprocessed and key features are extracted, including calculating characteristic parameters of the driving behavior and analyzing driving habits.

4. The driver status assessment method based on multimodal signal monitoring according to claim 1, characterized in that: The feature fusion step includes fusing the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state, and further includes: splicing the multimodal features into multidimensional vectors in sequence; The multidimensional vector is subjected to dimensionality reduction processing based on the Pearson correlation coefficient method and the principal component analysis method to obtain the fusion feature.

5. The driver status assessment method based on multimodal signal monitoring according to claim 1, characterized in that: The state evaluation step includes inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, giving a corresponding reminder to the driver, further including: Input the fusion features into a pre-trained LSTM model for state evaluation and classification of the abnormal state, wherein the abnormal state includes a fatigue state, a distracted state, and an angry state; When the abnormal state of the driver exceeds the preset condition, the driver is reminded in at least one of visual, tactile, olfactory or auditory ways.

6. The driver status assessment method based on multimodal signal monitoring according to claim 5 is characterized in that: It also includes configuration and model training steps: Configuring real-time acquisition, preprocessing, feature extraction and state assessment tasks of the multimodal signals; The LSTM model is trained based on the labeled driver status data and vehicle status data, and the generalization ability and accuracy of the LSTM model are evaluated by cross-validation method; The trained LSTM model is deployed for real-time status evaluation.

7. A driver status assessment system based on multimodal signal monitoring, characterized in that: include: A signal acquisition module, configured to acquire multimodal signals of the driver in real time, wherein the multimodal signals include video signals, audio signals, electrocardiogram signals, and vehicle status signals; A feature extraction module is configured to preprocess the multimodal signal and extract key features to obtain multimodal features, wherein the multimodal features include driver and driving behavior features; a feature fusion module, configured to fuse the multimodal features to obtain fused features, wherein the fused features are used to describe the driver state; The state assessment module is configured to include inputting the fusion feature into a preset model to classify the abnormal state of the driver, and when the abnormal state exceeds a preset condition, the driver is reminded accordingly.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method described in any one of claims 1 to 6.

10. A vehicle, characterized in that: The vehicle is provided with a driver status assessment system based on multimodal signal monitoring as described in claim 7.

Images