Driver state monitoring and intelligent early warning method and device for human-machine co-driving

Abstract

The application relates to the computer technical field, and in particular to a driver state monitoring and intelligent early warning method and device for human-machine co-driving, which comprises the following steps: collecting facial image data, eye movement data and physiological signals of a driver in real time through a multi-modal sensor; extracting facial visual features, eye movement time sequence features and physiological signal time-frequency features through a CNN model and an LSTM model; weighting and fusing the facial visual features, the eye movement time sequence features and the physiological signal time-frequency features through a Transformer self-attention mechanism to obtain a multi-modal feature vector; acquiring vehicle driving environment data, and predicting a dangerous probability in a future preset time through an LSTM model according to the multi-modal feature vector and the vehicle driving environment data; and performing multi-modal early warning according to the predicted dangerous probability and a preset early warning threshold. The application helps to realize real-time evaluation of the driver state and intelligent early warning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a driver state monitoring and intelligent early warning method and device for human-machine co-driving. BACKGROUND

[0002] With the rapid development of advanced driving assistance system (ADAS) and automatic driving technology, vehicles are evolving from traditional manual driving to human-machine co-driving mode. In the human-machine co-driving mode, the driver still needs to pay attention to the safety of vehicle driving and take over the control of the vehicle when necessary.

[0003] However, in real driving scenarios, the driver may be distracted, fatigued, drowsy, and inefficient, which may lead to traffic accidents. In the prior art, most driver monitoring methods only focus on facial expression or eye movement detection, and lack comprehensive assessment of the driver's overall physiological and psychological state. The early warning methods also mostly rely on single alarm with fixed threshold, which cannot effectively prompt the potential risks of the driver. Therefore, there is an urgent need for a solution that can multi-dimensionally and multi-modally evaluate the driver's state in real time and provide intelligent early warning. SUMMARY

[0004] Therefore, it is necessary to provide a driver state monitoring and intelligent early warning method and device for human-machine co-driving, which can realize multi-modal real-time evaluation of the driver's state and intelligent early warning.

[0005] In a first aspect, the present application provides a driver state monitoring and intelligent early warning method for human-machine co-driving, which comprises:

[0006] real-time collection of facial image data, eye movement data, and physiological signals of the driver by multi-modal sensors;

[0007] extraction of facial visual features, eye movement time sequence features, and physiological signal time-frequency features from the facial image data, eye movement data, and physiological signals by CNN and LSTM models;

[0008] weighting fusion of the facial visual features, eye movement time sequence features, and physiological signal time-frequency features by the Transformer self-attention mechanism to obtain a multi-modal feature vector;

[0009] acquisition of vehicle driving environment data, and prediction of the risk probability of a future preset time by the LSTM model based on the multi-modal feature vector and the vehicle driving environment data;

[0010] multi-modal early warning according to the predicted risk probability and a preset early warning threshold.

[0011] In one of the embodiments, the extracting, according to the facial image data, the eye movement data and the physiological signal, the facial visual features, the eye movement time sequence features and the physiological signal time-frequency features through the CNN model and the LSTM model comprises:

[0012] The facial visual features are extracted from the facial image data through the CNN model, and the facial visual features comprise facial state information;

[0013] The eye movement time sequence features are extracted from the eye movement data through the LSTM model, and the eye movement time sequence features comprise blink frequency, gaze duration and eye movement trajectory;

[0014] The physiological signal time-frequency features are obtained by performing time-frequency analysis on the physiological signal.

[0015] In one of the embodiments, the acquiring vehicle driving environment data and predicting, according to the multi-modal feature vector and the vehicle driving environment data, the dangerous probability in a future preset time through the LSTM model comprises:

[0016] The vehicle driving environment data is obtained by performing binarization, character segmentation and recognition on the text on the instrument panel or head-up display in the in-vehicle environment image through the OCR technology, and the vehicle driving environment data is kept synchronous with the facial image data, the eye movement data and the physiological signal through a time stamp;

[0017] The dangerous probability in a future preset time is predicted according to the multi-modal feature vector and the vehicle driving environment data through the LSTM model.

[0018] In one of the embodiments, the predicting, according to the multi-modal feature vector and the vehicle driving environment data, the dangerous probability in a future preset time through the LSTM model comprises:

[0019] The semantic correlation features are extracted according to the multi-modal feature vector and the vehicle driving environment data, wherein the semantic correlation features are used to mine the correlation between the multi-modal features;

[0020] The semantic correlation features are weighted and fused into the multi-modal feature vector through the Transformer self-attention mechanism to obtain a final multi-modal feature vector;

[0021] The dangerous probability in a future preset time is predicted according to the final multi-modal feature vector and the vehicle driving environment data through the LSTM model.

[0022] In one of the embodiments, the predicting, by the LSTM model, the dangerous probability in the future preset time according to the final multi-modal feature vector and the vehicle driving environment data comprises:

[0023] classifying the multi-modal feature vector by a full connection classifier to obtain a current state of the driver;

[0024] predicting, by the LSTM model, the dangerous probability in the future preset time according to the current state of the driver, the final multi-modal feature vector and the vehicle driving environment data.

[0025] In one of the embodiments, the multi-level pre-warning according to the predicted dangerous probability and the preset pre-warning threshold further comprises:

[0026] regularly updating the parameters of the LSTM model, the pre-warning threshold and the feature vector weight related to individual differences in the multi-modal feature vector by using the collected driver data based on an incremental learning algorithm.

[0027] In one of the embodiments, the driver state monitoring and intelligent pre-warning method for human-machine co-driving further comprises:

[0028] pruning and quantifying each model to realize lightweight of the model;

[0029] using a special instruction set of the vehicle-mounted NPU to accelerate the inference process of each model.

[0030] In a second aspect, the application further provides a driver state monitoring and intelligent pre-warning device for human-machine co-driving. The device comprises:

[0031] a data collection module configured to collect facial image data, eye movement data and physiological signals of a driver in real time through multi-modal sensors;

[0032] a feature extraction module configured to extract facial visual features, eye movement time sequence features and physiological signal time-frequency features by CNN and LSTM models according to the facial image data, the eye movement data and the physiological signals;

[0033] a multi-modal feature acquisition module configured to obtain a multi-modal feature vector by weighting and fusing the facial visual features, the eye movement time sequence features and the physiological signal time-frequency features through a Transformer self-attention mechanism;

[0034] a dangerous probability prediction module configured to obtain vehicle driving environment data, and predict, by the LSTM model, a dangerous probability in a future preset time according to the multi-modal feature vector and the vehicle driving environment data;

[0035] A multi-level early warning module is configured to perform multi-level early warning according to a predicted danger probability and a preset early warning threshold.

[0036] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0037] Real-time collection of facial image data, eye movement data and physiological signals of the driver through a multi-modal sensor;

[0038] According to the facial image data, the eye movement data and the physiological signals, facial visual features, eye movement time sequence features and physiological signal time-frequency features are extracted through a CNN model and an LSTM model;

[0039] The facial visual features, the eye movement time sequence features and the physiological signal time-frequency features are weighted and fused through a Transformer self-attention mechanism to obtain a multi-modal feature vector;

[0040] Collection of vehicle driving environment data, and prediction of a danger probability at a future preset time through the LSTM model according to the multi-modal feature vector and the vehicle driving environment data;

[0041] Multi-modal early warning according to the predicted danger probability and a preset early warning threshold.

[0042] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0043] Real-time collection of facial image data, eye movement data and physiological signals of the driver through a multi-modal sensor;

[0044] According to the facial image data, the eye movement data and the physiological signals, facial visual features, eye movement time sequence features and physiological signal time-frequency features are extracted through a CNN model and an LSTM model;

[0045] The facial visual features, the eye movement time sequence features and the physiological signal time-frequency features are weighted and fused through a Transformer self-attention mechanism to obtain a multi-modal feature vector;

[0046] Collection of vehicle driving environment data, and prediction of a danger probability at a future preset time through the LSTM model according to the multi-modal feature vector and the vehicle driving environment data;

[0047] Multi-modal early warning according to the predicted danger probability and a preset early warning threshold.

[0048] In summary, the present application includes the following beneficial technical effects:

[0049] The driver's facial image, eye movement, and physiological signal multi-source data are collected by the multi-modal sensor, and the CNN model, LSTM model, and Transformer model are used for multi-dimensional feature extraction and fusion, which effectively integrates different modal data and avoids the limitations of single modal data in complex environments. Through attention mechanism weighted fusion, the weight of different modal features can be dynamically adjusted, so as to better capture the complex relationships in multi-modal data. The LSTM model is introduced to jointly model the fused multi-modal feature vector and vehicle driving environment data, which can predict the risk probability of the future preset time, make the early warning decision have foresight, and effectively reduce the incidence of traffic accidents. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 FIG. 1 is a flowchart of a driver state monitoring and intelligent early warning method for human-machine co-driving in an embodiment;

[0051] Figure 2 FIG. 4 is a schematic diagram of a multi-modal input processing flow;

[0052] Figure 3 FIG. 5 is a schematic diagram of a multi-modal fusion flow;

[0053] Figure 4 FIG. 6 is a schematic diagram of a feature extraction flow;

[0054] Figure 5 FIG. 7 is a structural block diagram of a driver state monitoring and intelligent early warning device for human-machine co-driving in an embodiment. DETAILED DESCRIPTION

[0055] Therefore, it is necessary to provide a driver state monitoring and intelligent early warning method and device for human-machine co-driving, which can realize real-time evaluation of driver state and intelligent early warning based on multi-modal.

[0056] In a first aspect, the present application provides a driver state monitoring and intelligent early warning method for human-machine co-driving, which comprises:

[0057] S100, real-time collection of facial image data, eye movement data, and physiological signals of the driver by a multi-modal sensor.

[0058] Specifically, the facial image data, eye movement data and physiological signals of the driver are collected in real time by a multi-modal sensor. The multi-modal sensor includes a high-resolution RGB camera, an infrared camera, an eye tracker, a heart rate sensor and a skin conductance sensor. The RGB camera and the infrared camera are installed in front of the cockpit to obtain the color image and the thermal image of the driver's face, respectively. All collected images are automatically oriented and corrected to ensure consistency in image orientation and are adjusted to 512x512 pixels to ensure the accuracy and standardization of subsequent feature extraction. The eye tracker is used to record the eye movement data of the driver, such as gaze trajectory, blink frequency and gaze duration. The heart rate sensor and the skin conductance sensor can collect the heart rate signal and the skin conductance signal of the driver through a wearable device (such as a heart rate band) to reflect the physiological state changes of the driver. All sensors are connected to the vehicle's Ethernet or CAN bus and the vehicle's computing unit to ensure real-time and synchronous transmission of the collected data to the computing unit, providing basic data support for subsequent processing.

[0059] S200, according to the facial image data, eye movement data and physiological signals, the facial visual features, eye movement time series features and physiological signal time-frequency features are extracted by the CNN model and the LSTM model.

[0060] Specifically, as shown in Figure 2 The collected facial color image, thermal image, eye movement data and physiological signals are preprocessed, including denoising and pseudo-color mapping of the facial color image and the thermal image, band-pass filtering and normalization of the physiological signals, interpolation filling and normalization of the eye movement data. Then, according to the preprocessed facial image data, eye movement data and physiological signals, the facial visual features, eye movement time series features and physiological signal time-frequency features are extracted by the CNN (convolutional neural network) model and the LSTM (long short-term memory network) model, laying the foundation for subsequent multi-modal fusion.

[0061] S300, the facial visual features, eye movement time series features and physiological signal time-frequency features are weighted and fused by the Transformer self-attention mechanism to obtain a multi-modal feature vector.

[0062] Specifically, the Transformer self-attention mechanism includes multiple attention heads, each of which calculates the correlation weight between different modal features. First, as shown in Figure 3As shown, the facial visual features, eye movement timing features, and physiological signal time-frequency features are unified and mapped to a feature space through a multi-head self-attention fusion layer to ensure consistent feature dimensions; then, each attention head calculates a similarity matrix of any two modal features, and assigns weights according to the similarity (the greater the weight value, the higher the contribution of the modal feature to the driver state evaluation, such as the increase in the weight of the eye movement timing feature in the fatigue state and the increase in the weight of the physiological signal time-frequency feature in the emotional fluctuation state); finally, the output features of the attention heads are spliced, and the features are integrated through a feedforward fully connected layer (the hidden layer dimension is 4d_model, and the activation function uses ReLU), and a multi-modal feature vector is output. In the fusion process, multi-head self-attention is used instead of single-head attention, which can capture multi-scale correlation relationships between different modalities (such as the strong correlation between "eye closure" in facial visual features and "blinking frequency" in eye movement timing features, and the weak correlation between "blinking frequency" in eye movement timing features and "heart rate low frequency component" in physiological signal time-frequency features), further improving the effectiveness of the fused features.

[0063] In this embodiment, the multi-modal feature vectors are obtained by weighting and fusing multi-modal features through the Transformer self-attention mechanism, which can effectively avoid the limitations of single modal data, improve the feature expression ability, and realize efficient interaction and complementarity of cross-modal information.

[0064] S400, vehicle driving environment data is obtained, and the future preset time danger probability is predicted through the LSTM model according to the multi-modal feature vector and the vehicle driving environment data.

[0065] Specifically, the in-vehicle environment image containing the instrument panel or the head-up display (HUD) is collected by using the front-mounted camera, the instrument panel or the HUD text area in the in-vehicle environment image is identified, and the vehicle driving environment data is obtained; the multi-modal feature vector and the vehicle driving environment data are combined, and the future preset time danger probability prediction is realized through the LSTM model, wherein the preset time can be set according to actual needs, and is usually 1-3s, which takes into account the prediction foresight and accuracy.

[0066] S500, multi-modal warning is performed according to the predicted danger probability and the preset warning threshold.

[0067] Specifically, the preset warning threshold can be dynamically adjusted according to different driving scenes and driving habits of the driver. For example, in a highway driving scene, the warning threshold can be set relatively low to achieve earlier warning; in a city road driving scene, the warning threshold can be appropriately increased. When the predicted risk probability is lower than the first warning threshold, it is determined that the current driving state is safe, and the system does not give a warning; when the risk probability reaches or exceeds the first warning threshold but is lower than the second warning threshold, a first-level warning is triggered, and the driver is reminded to pay attention to the current state through visual warning (such as displaying a yellow warning icon on the vehicle display screen) and auditory warning (such as playing a soft prompt tone); when the risk probability reaches or exceeds the second warning threshold, a second-level warning is triggered, and more intense warning methods are adopted, such as visual warning (displaying a red warning icon and flashing), auditory warning (playing an urgent alarm sound), and tactile warning (such as through steering wheel vibration, seat vibration), to forcibly remind the driver to take timely measures to adjust the driving state and avoid traffic accidents.

[0068] In one embodiment, according to the facial image data, the eye movement data and the physiological signals, the facial visual features, the eye movement time sequence features and the physiological signal time-frequency features are extracted through the CNN model and the LSTM model, including:

[0069] According to the facial image data, the facial visual features are extracted through the CNN model, and the facial visual features include facial state information; according to the eye movement data, the eye movement time sequence features are extracted through the LSTM model, and the eye movement time sequence features include blink frequency, gaze duration and eye movement trajectory; the physiological signals are subjected to time-frequency analysis to obtain the physiological signal time-frequency features.

[0070] Specifically, as Figure 4As shown, for the facial image data, the preprocessed facial color image and the thermal image are input into the CNN model, and the CNN model can automatically learn and extract facial visual features, including the driver's facial state information, such as the degree of eye closure, the mouth state (whether yawning), the facial expression (such as anxiety, relaxation, etc.), and the head posture (whether to lower the head, turn the head, etc.), which can intuitively reflect the driver's fatigue degree and attention concentration. For the eye movement data, since the eye movement process has obvious time sequence characteristics, it is input into the LSTM model for processing. The LSTM model can effectively capture the long-term dependence relationship in the time sequence data, and through the analysis of the continuous eye movement data, the eye movement time sequence features are extracted, including the blink frequency (the number of blinks per unit time), the fixation duration (the fixation duration of a specific area), and the eye movement trajectory (the movement path of the eyeball in different time periods), which can further assist in determining whether the driver has a state of attention distraction, fatigue, etc. For the physiological signals (heart rate signal and skin electricity signal), the preprocessed physiological signals are first subjected to time-frequency analysis (such as wavelet transform, short-time Fourier transform, etc.), converting the time domain signal into a time-frequency domain signal, and then extracting the physiological signal time-frequency features, which can reflect the variation law of the heart rate, skin electricity signal with time and frequency, such as heart rate variability, skin electricity signal fluctuation amplitude, etc., which can indirectly reflect the physiological and psychological changes of the driver's tension, fatigue state, etc.

[0071] In one embodiment, vehicle driving environment data is acquired, and a dangerous probability in a future preset time is predicted according to the multi-modal feature vector and the vehicle driving environment data through an LSTM model, including:

[0072] An in-vehicle environment image is acquired, and text on an instrument panel or a head-up display in the in-vehicle environment image is binarized, character segmented, and recognized through an OCR technology to obtain vehicle driving environment data, wherein the vehicle driving environment data is kept synchronous with the facial image data, eye movement data, and physiological signals through a time stamp; a dangerous probability in a future preset time is predicted according to the multi-modal feature vector and the vehicle driving environment data through an LSTM model.

[0073] Specifically, the in-vehicle environment image is collected by the in-vehicle camera, the text information on the instrument panel or the head-up display in the in-vehicle environment image is processed by using an OCR (Optical Character Recognition) technology, binarization (separating the text from the background by using an adaptive threshold method), character segmentation (segmenting a single character based on connected domain analysis) and character recognition are sequentially completed, and thus the vehicle driving environment data such as the vehicle driving speed, the engine speed, the lane deviation information, the front vehicle distance and the like are extracted. Meanwhile, by using the time stamp technology, the vehicle driving environment data is kept time-synchronized with the previously collected facial image data, eye movement data and physiological signals, and thus the consistency and timeliness of the data in subsequent prediction and analysis are ensured. According to the multi-modal feature vector and the vehicle driving environment data, the risk probability in a future preset time is predicted by using an LSTM model.

[0074] In one embodiment, predicting the risk probability in a future preset time according to the multi-modal feature vector and the vehicle driving environment data by using an LSTM model comprises:

[0075] According to the multi-modal feature vector and the vehicle driving environment data, a semantic correlation feature is extracted, wherein the semantic correlation feature is used to mine the correlation between the multi-modal features; the semantic correlation feature is weighted and fused into the multi-modal feature vector by using a Transformer self-attention mechanism to obtain a final multi-modal feature vector; and according to the final multi-modal feature vector and the vehicle driving environment data, the risk probability in a future preset time is predicted by using an LSTM model.

[0076] Specifically, first, according to the multi-modal feature vector and the vehicle driving environment data, a semantic correlation feature is extracted, which is used to mine the potential correlation between the multi-modal features and the vehicle driving environment data, such as the correlation between the driver fatigue state and the vehicle driving speed, the correlation between the attention distraction state and the lane deviation, and the like; then, the semantic correlation feature is weighted and fused into the multi-modal feature vector by using a Transformer self-attention mechanism to obtain a final multi-modal feature vector, which further enriches the semantic information and correlation of the features; finally, the obtained final multi-modal feature vector and the vehicle driving environment data are input into an LSTM model to predict the risk probability in a future preset time.

[0077] In this embodiment, the final vector not only contains the multi-modal physiological and behavioral features of the driver himself, but also closely relates to the vehicle driving environment state, which greatly improves the representation ability of the feature vector for the driver state and the potential danger, and provides a more valuable and more comprehensive and accurate data basis for subsequent risk probability prediction and intelligent early warning.

[0078] In one embodiment, predicting the risk probability in a future preset time according to the final multi-modal feature vector and the vehicle driving environment data by using an LSTM model comprises:

[0079] The multi-modal feature vector is classified by a fully connected classifier to obtain the current state of the driver; and the dangerous probability in the future preset time is predicted by an LSTM model according to the current state of the driver, the final multi-modal feature vector and the vehicle driving environment data.

[0080] Specifically, first, the multi-modal feature vector is generated by the Transformer self-attention mechanism; then, the final multi-modal feature vector is classified by a fully connected classifier to determine the specific state of the driver at present, wherein each neuron in the fully connected classifier is connected with each element of the input vector, and the multi-modal feature vector is mapped to different state categories through comprehensive processing of the information, so as to determine the specific state of the driver at present, such as fatigue state, attention concentration state, anxiety state, etc.; then, the current state of the driver, the final multi-modal feature vector and the vehicle driving environment data are input into the LSTM model, and the LSTM model predicts the dangerous probability in the future preset time by using its modeling ability for time series data and combining historical data and current data.

[0081] In one embodiment, according to the predicted dangerous probability and the preset warning threshold, after multi-level warning, it further includes:

[0082] Based on the incremental learning algorithm, the parameters of the LSTM model, the warning threshold and the feature vector weight related to individual differences in the multi-modal feature vector are updated regularly by using the collected driver data.

[0083] Specifically, the incremental learning algorithm is used to input the collected samples and historical training samples into the LSTM model to update the parameters of the LSTM model; the warning threshold is dynamically adjusted according to the historical response data of the driver, for example, if the response rate (effective response times / warning trigger times) of the driver to a certain level of warning is lower than 60% (such as not responding in time to the first level of warning), the threshold of this level is reduced by 10% to trigger the warning in advance; the feature weight related to individual differences in the multi-modal feature vector is adjusted through the weight feedback of the Transformer self-attention mechanism, for example, the eye movement data of a certain driver has stronger representation ability for the fatigue state (such as the change amplitude of the blink frequency of the driver when being tired is significantly higher than the average level), and the attention weight of the eye movement time series feature in the fusion is improved through incremental learning. The collected driver feedback data is uploaded to the cloud training platform through a secure channel for incremental learning and model updating; the updated model is distributed through OTA, and is automatically verified on the vehicle side before being switched to use.

[0084] In this embodiment, based on the incremental learning algorithm, the parameters of the LSTM model, the early warning threshold and the feature vector weights related to individual differences in the multi-modal feature vector are updated regularly using the collected driver data, realizing continuous self-adaptive optimization of individual driver behavior differences and improving the personalization and reliability of early warning.

[0085] In one embodiment, the driver state monitoring and intelligent early warning method for human-machine co-driving further comprises:

[0086] Pruning and quantizing each model to realize lightweight of the model; using the special instruction set of the vehicle-mounted NPU to accelerate the inference process of each model.

[0087] Specifically, a structured pruning algorithm is used to delete or shield parameters that contribute less to the model, making the network structure sparse; at the same time, the model is quantized, and the model weights and activations are compressed from 32-bit floating point (FP32) to low-bit integer (INT8, INT4) or mixed precision (FP16 / BF16), thereby reducing storage and computing power requirements; the lightweight model is exported in ONNX format and integrated into the vehicle-mounted ROS system. The special instruction set of the vehicle-mounted NPU is used to optimize the model inference process, for example, for the convolution operation of the CNN model, the special convolution instruction of the NPU is used to reduce the amount of calculation; for the gate calculation of the LSTM model, the vector operation instruction of the NPU is used to process multiple sets of time series data in parallel, improving the calculation efficiency; for the self-attention calculation of the Transformer model, the matrix multiplication acceleration unit of the NPU is used to reduce the time consumption of attention weight calculation. In addition, a fault detection module is added to the system, which automatically switches to a backup detection strategy when the camera or sensor data is interrupted, ensuring continuous operation of the system.

[0088] In this embodiment, model pruning and quantization significantly reduce the number of model parameters and the occupation of computing resources, ensuring real-time operation on low-end vehicle-mounted hardware; at the same time, the special instruction set of the vehicle-mounted NPU is used to accelerate the inference process of each model, ensuring that the system meets the real-time requirements.

[0089] In one embodiment, as shown in Figure 5 A driver state monitoring and intelligent early warning device for human-machine co-driving is provided, comprising a data acquisition module 10, a feature extraction module 20, a multi-modal feature acquisition module 30, a danger probability prediction module 40 and a multi-level early warning module 50, wherein:

[0090] The data acquisition module 10 is used to acquire real-time facial image data, eye movement data and physiological signals of the driver through multi-modal sensors;

[0091] The feature extraction module 20 is configured to extract facial visual features, eye movement time sequence features, and physiological signal time-frequency features by using a CNN model and an LSTM model according to the facial image data, the eye movement data, and the physiological signal.

[0092] The multi-modal feature acquisition module 30 is configured to obtain a multi-modal feature vector by using a Transformer self-attention mechanism to weight and fuse the facial visual features, the eye movement time sequence features, and the physiological signal time-frequency features.

[0093] The danger probability prediction module 40 is configured to obtain vehicle driving environment data, and predict a danger probability at a future preset time by using an LSTM model according to the multi-modal feature vector and the vehicle driving environment data.

[0094] The multi-level warning module 50 is configured to perform multi-level warning according to the predicted danger probability and a preset warning threshold.

[0095] In an embodiment, the feature extraction module 20 is further configured to extract facial visual features by using a CNN model according to the facial image data, wherein the facial visual features include facial state information; extract eye movement time sequence features by using an LSTM model according to the eye movement data, wherein the eye movement time sequence features include blink frequency, gaze duration, and eye movement trajectory; and perform time-frequency analysis on the physiological signal to obtain physiological signal time-frequency features.

[0096] In an embodiment, the danger probability prediction module 40 is further configured to obtain an in-vehicle environment image, perform binarization, character segmentation, and recognition on text on an instrument panel or a head-up display in the in-vehicle environment image by using an OCR technology to obtain vehicle driving environment data, wherein the vehicle driving environment data is synchronized with the facial image data, the eye movement data, and the physiological signal by using a time stamp; and predict a danger probability at a future preset time by using an LSTM model according to the multi-modal feature vector and the vehicle driving environment data.

[0097] In an embodiment, the danger probability prediction module 40 is further configured to extract semantic association features according to the multi-modal feature vector and the vehicle driving environment data, wherein the semantic association features are used to mine association relationships among the multi-modal features; weight and fuse the semantic association features into the multi-modal feature vector by using a Transformer self-attention mechanism to obtain a final multi-modal feature vector; and predict a danger probability at a future preset time by using an LSTM model according to the final multi-modal feature vector and the vehicle driving environment data.

[0098] In an embodiment, the dangerous probability prediction module 40 is further configured to classify the multi-modal feature vector by a fully connected classifier to obtain a current state of the driver; and predict a dangerous probability of a future preset time by an LSTM model according to the current state of the driver, the final multi-modal feature vector, and the vehicle driving environment data.

[0099] In an embodiment, the driver state monitoring and intelligent early warning device for human-machine co-driving further comprises an updating module configured to periodically update parameters of the LSTM model, the early warning threshold, and feature vector weights of the multi-modal feature vector related to individual differences based on an incremental learning algorithm using the collected driver data.

[0100] In an embodiment, the driver state monitoring and intelligent early warning device for human-machine co-driving further comprises a lightweight and acceleration module configured to perform pruning and quantization on each model to realize lightweight of the model; and accelerate the inference process of each model by using a special instruction set of an NPU.

[0101] In an embodiment, the present application discloses a computer device, comprising a memory and a processor, the memory is configured to store a computer program capable of running on the processor, and when the processor loads the computer program, a driver state monitoring and intelligent early warning method for human-machine co-driving is executed.

[0102] In an embodiment, the present application discloses a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein when the computer program is loaded by a processor, a driver state monitoring and intelligent early warning method for human-machine co-driving is executed.

[0103] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application, therefore: any equivalent changes made according to the structure, shape, principle of the present application shall be covered within the protection scope of the present application.

Claims

1. A method for driver status monitoring and intelligent early warning in human-machine co-driving, characterized in that, include: Real-time acquisition of driver's facial image data, eye movement data, and physiological signals using multimodal sensors; Based on the facial image data, the eye movement data, and the physiological signals, facial visual features, eye movement temporal features, and physiological signal time-frequency features are extracted using CNN and LSTM models. A multimodal feature vector is obtained by weighted fusion of the facial visual features, the eye movement temporal features, and the physiological signal time-frequency features using the Transformer self-attention mechanism. Acquire vehicle driving environment data, and based on the multimodal feature vector and the vehicle driving environment data, use the LSTM model to predict the probability of danger at a preset time in the future; Based on the predicted probability of danger and the preset warning threshold, multi-level warnings are issued. The step of acquiring vehicle driving environment data and, based on the multimodal feature vectors and the vehicle driving environment data, predicting the probability of danger at a predetermined future time using the LSTM model includes: The in-vehicle environment image is acquired, and the text on the dashboard or head-up display in the in-vehicle environment image is binarized, segmented and recognized using OCR technology to obtain vehicle driving environment data. The vehicle driving environment data is synchronized with the facial image data, the eye movement data and the physiological signals through timestamps. The step of predicting the probability of danger at a predetermined time in the future using the LSTM model based on the multimodal feature vectors and the vehicle driving environment data includes: Based on the multimodal feature vectors and the vehicle driving environment data, semantic association features are extracted, wherein the semantic association features are used to mine the association relationships between multimodal features; The semantic association features are weighted and fused into the multimodal feature vector through the Transformer self-attention mechanism to obtain the final multimodal feature vector. Based on the final multimodal feature vector and the vehicle driving environment data, the LSTM model predicts the probability of danger at a predetermined time in the future, including: The driver's current state is obtained by classifying the multimodal feature vectors using a fully connected classifier. Based on the driver's current state, the final multimodal feature vector, and the vehicle driving environment data, the LSTM model is used to predict the probability of danger at a predetermined time in the future.

2. The method for driver status monitoring and intelligent early warning for human-machine co-driving as described in claim 1, characterized in that, The step of extracting facial visual features, eye movement temporal features, and physiological signal time-frequency features based on the facial image data, the eye movement data, and the physiological signals using CNN and LSTM models includes: Based on the facial image data, facial visual features are extracted using a CNN model, and the facial visual features include facial state information. Based on the eye movement data, eye movement temporal features are extracted using an LSTM model. These features include blink frequency, fixation duration, and eye movement trajectory. Time-frequency analysis was performed on the physiological signal to obtain its time-frequency characteristics.

3. The method for driver status monitoring and intelligent early warning for human-machine co-driving as described in claim 1, characterized in that, After implementing multi-level early warnings based on the predicted risk probability and preset warning thresholds, the method further includes: Based on the incremental learning algorithm, the parameters of the LSTM model, the warning threshold, and the weights of the feature vectors related to individual differences in the multimodal feature vectors are updated periodically using the collected driver data.

4. The method for driver status monitoring and intelligent early warning for human-machine co-driving as described in claim 1, characterized in that, Also includes: Each model is pruned and quantized to achieve lightweight model implementation; The dedicated instruction set of the onboard NPU is used to accelerate the inference process of each model.

5. A driver status monitoring and intelligent early warning device for human-machine co-driving, characterized in that, include: The data acquisition module is used to perform the operation of "real-time acquisition of the driver's facial image data, eye movement data and physiological signals through a multimodal sensor" in the method of claim 1. The feature extraction module is used to perform the operation of "extracting facial visual features, eye movement temporal features and physiological signal time-frequency features based on the facial image data, the eye movement data and the physiological signal through CNN model and LSTM model" in the method of claim 1. The multimodal feature acquisition module is used to perform the operation of "obtaining a multimodal feature vector by weighted fusion of the facial visual features, the eye movement temporal features and the physiological signal time-frequency features through the Transformer self-attention mechanism" in the method of claim 1. The danger probability prediction module is used to perform the operation of "acquiring vehicle driving environment data and predicting the danger probability at a preset time in the future based on the multimodal feature vector and the vehicle driving environment data through the LSTM model" in the method of claim 1. The multi-level early warning module is used to perform the operation of "performing multi-level early warnings based on the predicted risk probability and the preset early warning threshold" in the method of claim 1.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images