Driver fatigue state real-time monitoring system and method based on multi-source data

Abstract

The invention relates to the field of intelligent fatigue driving detection, and provides a driver fatigue state real-time monitoring system and method based on multi-source data. Feature extraction is carried out on eye and mouth states in driver face data obtained by shooting of a camera to obtain real state features of a driver, and feature extraction is carried out on related information, collected by various sensors, of a vehicle driven by the driver to obtain vehicle driving features; and judging whether the driver has fatigue driving or not by combining the vehicle driving characteristics and the real state characteristics of the driver. Therefore, the fatigue state of the driver can be monitored in real time and accurately judged.

Description

Technical Field

[0001] The present application relates to the field of intelligent fatigue driving detection, and more specifically, to a real-time monitoring system and method for driver fatigue status based on multi-source data. Background Art

[0002] In today's society, with the rapid development of the transportation industry, cars have become an indispensable means of transportation in people's lives. However, the traffic safety issues that come with it are becoming increasingly prominent. Among them, driver fatigue driving has become a serious social problem.

[0003] In the fast-paced modern life, people are facing various pressures, and drivers are no exception. Long driving hours, high-intensity work tasks, and irregular work and rest schedules can all lead to driver fatigue. When a driver is in a fatigued state, his or her reaction ability, judgment ability, and attention will be significantly reduced, which can lead to traffic accidents in serious cases. These traffic accidents not only bring great pain and losses to the parties involved, but also bring a heavy burden to society.

[0004] In traditional driver fatigue status monitoring, there are often situations where judgments are made based on a single feature. For example, judgments are made based on only one item of eye status or vehicle driving data. However, a single data source is easily interfered by noise, individual differences, or special situations. For example, eye status may be affected by factors such as light; vehicle driving data may deviate due to factors such as road conditions and vehicle performance. Therefore, this single-feature monitoring method is prone to misjudgment and cannot accurately judge the driver's fatigue status. An optimized real-time monitoring solution for driver fatigue status is expected. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present application provides a real-time monitoring system and method for driver fatigue status based on multi-source data.

[0006] According to one aspect of the present application, a real-time monitoring system for driver fatigue status based on multi-source data is provided, which includes:

[0007] A fatigue status monitoring related data acquisition module is used to obtain the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, vehicle following distance and driver's face image at multiple predetermined time points within a predetermined time period;

[0008] A vehicle monitoring data feature encoding module, used for performing time series encoding on the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the plurality of predetermined time points to obtain a vehicle monitoring time series feature matrix;

[0009] A driver monitoring data feature encoding module, used for performing feature encoding on the driver's facial images at the plurality of predetermined time points to obtain a driver monitoring state temporal feature enhancement matrix;

[0010] A multi-source monitoring feature fusion module, used to fuse the driver monitoring state time series feature enhancement matrix and the vehicle monitoring time series feature matrix to obtain a fatigue state monitoring related feature matrix;

[0011] The fatigue monitoring result generating module is used to obtain the driver fatigue monitoring result based on the fatigue state monitoring related feature matrix.

[0012] According to another aspect of the present application, a method for real-time monitoring of driver fatigue status based on multi-source data is provided, which includes:

[0013] Acquire the vehicle speed, the vehicle instantaneous acceleration, the vehicle steering angular velocity, the vehicle following distance and the driver's face image at multiple predetermined time points within a predetermined time period;

[0014] Performing time series coding on the vehicle speed, the vehicle instantaneous acceleration, the vehicle steering wheel steering angular velocity, and the vehicle following distance at the plurality of predetermined time points to obtain a vehicle monitoring time series feature matrix;

[0015] Performing feature encoding on the driver's facial images at the plurality of predetermined time points to obtain a driver monitoring state temporal feature enhancement matrix;

[0016] Fusion of the driver monitoring state time series feature enhancement matrix and the vehicle monitoring time series feature matrix to obtain a fatigue state monitoring related feature matrix;

[0017] Based on the fatigue state monitoring related feature matrix, a driver fatigue monitoring result is obtained.

[0018] This application has significant technical effects due to the adoption of the above technical solutions:

[0019] The real-time monitoring system and method of driver fatigue status based on multi-source data provided by the present application adopts data processing technology based on deep learning, extracts features of the driver's eyes and mouth in the driver's facial data captured by the camera to obtain the driver's real state features, and extracts features of the relevant information of the driver driving the vehicle collected by various sensors to obtain the vehicle driving features, and combines the vehicle driving features and the driver's real state features to determine whether the driver is driving fatigued. In this way, real-time monitoring and accurate judgment of the driver's fatigue state can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0021] Figure 1 This is a flowchart of a method for real-time monitoring of driver fatigue status based on multi-source data according to an embodiment of the present application.

[0022] Figure 2 In the real-time monitoring method for driver fatigue status based on multi-source data according to an embodiment of the present application, the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the multiple predetermined time points are time-series encoded to obtain a flow chart of a vehicle monitoring time-series feature matrix.

[0023] Figure 3 The present invention is a flowchart of performing feature encoding on the driver's facial images at the plurality of predetermined time points to obtain a driver monitoring status temporal feature enhancement matrix in a real-time monitoring method for driver fatigue status based on multi-source data according to an embodiment of the present application.

[0024] Figure 4 In the real-time monitoring method of driver fatigue status based on multi-source data according to an embodiment of the present application, a flowchart is provided for extracting time series features based on eye areas and mouth areas of the driver's facial images at the multiple predetermined time points to obtain a time series feature diagram of the driver monitoring status.

[0025] Figure 5 The present invention is a flowchart of a method for real-time monitoring of driver fatigue status based on multi-source data according to an embodiment of the present application, in which the driver monitoring status time series feature graph is feature enhanced and then pooled to obtain the driver monitoring status time series feature enhancement matrix.

[0026] Figure 6 The present invention provides a flowchart for obtaining a driver fatigue monitoring result based on the fatigue status monitoring-related feature matrix in a real-time driver fatigue status monitoring method based on multi-source data according to an embodiment of the present application.

[0027] Figure 7 4 is a system block diagram of a real-time monitoring system for driver fatigue status based on multi-source data according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0029] Severe driver fatigue driving can lead to traffic accidents, and fatigue driving has become a serious social problem. If drivers are reminded when they are driving fatigued, 90% of similar traffic accidents can be avoided. In traditional driver fatigue status monitoring, there is only reliance on a single feature, such as judging only by the eye state or one item of vehicle driving data. A single data source is easily disturbed by noise, individual differences or special situations, which can easily lead to misjudgment. Therefore, an optimized real-time monitoring solution for driver fatigue status is expected.

[0030] In recent years, deep learning and neural networks have been widely used in computer vision, natural language processing, text signal processing and other fields. In addition, deep learning and neural networks have also demonstrated a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields. The development of deep learning and neural networks provides new solutions and solutions for real-time monitoring of driver fatigue status.

[0031] Figure 1 Flow chart of a method for real-time monitoring of driver fatigue status based on multi-source data according to an embodiment of the present application. Figure 1 As shown, according to the real-time monitoring method of driver fatigue status based on multi-source data of an embodiment of the present application, the method includes: S110, obtaining the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, vehicle following distance and driver facial images at multiple predetermined time points within a predetermined time period; S120, time-series encoding the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity and vehicle following distance at the multiple predetermined time points to obtain a vehicle monitoring time-series feature matrix; S130, feature encoding the driver facial images at the multiple predetermined time points to obtain a driver monitoring status time-series feature enhancement matrix; S140, fusing the driver monitoring status time-series feature enhancement matrix and the vehicle monitoring time-series feature matrix to obtain a fatigue status monitoring-related feature matrix; S150, obtaining a driver fatigue monitoring result based on the fatigue status monitoring-related feature matrix.

[0032] Specifically, in the technical solution of the present application, the vehicle speed, the instantaneous acceleration, the steering angular velocity of the vehicle steering wheel, the following distance of the vehicle and the facial images of the driver at the predetermined time points within a predetermined time period are first obtained. It should be understood that the vehicle speed is one of the important parameters reflecting the driver's driving behavior. In the state of fatigue driving, the driver's reaction speed and judgment ability will decrease, which may lead to large fluctuations in vehicle speed, such as frequent acceleration and deceleration or maintaining too low or too high speed for a long time. The instantaneous acceleration reflects the change in vehicle power performance and is also closely related to the driver's driving habits and operating status. In the case of fatigue driving, the driver's control ability of the vehicle is weakened, which may lead to unstable acceleration and deceleration operations and large fluctuations in instantaneous acceleration. The steering angular velocity of the steering wheel is an important indicator to measure the flexibility and stability of the driver's control of the vehicle. In the state of fatigue, the driver's control ability of the steering wheel decreases, and there may be situations such as untimely steering, too large or too small steering angle, etc. The following distance is the safe distance that the driver maintains with the vehicle in front during driving. In the case of fatigue driving, the driver's attention is distracted, and the driving state of the vehicle in front may not be accurately judged, resulting in a too close or too far following distance. That is, obtaining data such as vehicle speed, vehicle instantaneous acceleration, vehicle steering angular velocity, vehicle following distance, etc. at multiple predetermined time points can reflect the driver's driving behavior and state changes from different angles, and provide comprehensive and accurate information support for judging whether he is driving fatigued. The driver's facial image contains rich facial feature information, such as eyes, mouth, etc. In the state of fatigue driving, the driver's facial features will change significantly, such as prolonged eye closure time and frequent yawning. Considering that a person's facial features will change with the changes of time, emotions, environment and other factors, it is difficult to accurately judge the driver's fatigue state by relying solely on a facial image at a single time point. Therefore, it is necessary to obtain the driver's facial images at multiple predetermined time points and perform dynamic analysis. By comparing the changing trends and differences of facial features at different time points, the driver's fatigue state can be more accurately evaluated. In general, by dynamically analyzing the changing trends and differences of facial features, combined with other information sources such as vehicle driving data, real-time monitoring and accurate judgment of the driver's fatigue state can be achieved.

[0033] In an embodiment of the present application, a possible implementation method for obtaining the vehicle speed, the vehicle instantaneous acceleration, the vehicle steering wheel steering angular velocity, the vehicle following distance at multiple predetermined time points within a predetermined time period, and the driver's facial image at the multiple predetermined time points may be: a. The vehicle speed, the vehicle instantaneous acceleration, the vehicle steering wheel steering angular velocity, and the vehicle following distance data may be obtained respectively by a speed sensor, an acceleration sensor, a steering wheel angle sensor, and a radar; b. The driver's facial image data may be obtained by a camera installed at a suitable position in the vehicle.

[0034] Figure 2 The present invention is a flowchart of a method for real-time monitoring of driver fatigue status based on multi-source data according to an embodiment of the present application, in which the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel angular velocity, and vehicle following distance at the plurality of predetermined time points are time-series encoded to obtain a vehicle monitoring time-series feature matrix. Figure 2 As shown, the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the multiple predetermined time points are time-series encoded to obtain a vehicle monitoring time series feature matrix, including: S121, arranging the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the multiple predetermined time points according to the time dimension to obtain a vehicle monitoring time series input matrix; S122, passing the vehicle monitoring time series input matrix through a vehicle monitoring parameter time series feature encoder to obtain the vehicle monitoring time series feature matrix.

[0035] Considering that the driver's fatigue state is a process that changes over time, his driving behavior will also change dynamically accordingly. In order to clearly capture dynamic changes such as speed fluctuations, acceleration changes, unstable steering wheel operation, etc., so as to evaluate the driver's fatigue state, the technical solution of this application needs to arrange the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the multiple predetermined time points according to the time dimension to obtain a vehicle monitoring timing input matrix. It is also worth mentioning that by integrating multiple vehicle driving data at multiple time points into one matrix, the data processing process can be simplified and the processing efficiency can be improved.

[0036] The vehicle monitoring time series input matrix contains a large amount of raw data, but not all data are equally important for judging the driver's fatigue state. In order to extract the most critical feature information for judging fatigue driving from the raw matrix data, such as vehicle speed change mode, acceleration stability, steering wheel operation rules, etc., it is necessary to pass the vehicle monitoring time series input matrix through the vehicle monitoring parameter time series feature encoder to obtain the vehicle monitoring time series feature matrix. Through the processing of the vehicle monitoring parameter time series feature encoder, the noise and redundant information in the raw data can be filtered out, and more pure and useful feature information can be retained. These feature information are closer to the essential characteristics of the vehicle operation under the driver's fatigue state, so the accuracy of subsequent judgments can be improved. The structure of the vehicle monitoring parameter time series feature encoder in this application is a convolutional neural network model using convolution kernels of different scales, wherein the scale of the first convolution kernel is greater than the scale of the second convolution kernel. It should be understood that convolution kernels of different scales can capture feature information of different scales in the vehicle monitoring time series input matrix. Larger convolution kernels can extract global and overall feature information, while smaller convolution kernels are better at capturing local and detailed feature information. By combining these two convolution kernels, multi-scale feature extraction of the vehicle monitoring time series input matrix can be achieved, thereby more comprehensively reflecting the vehicle driving status under different driving conditions of the driver.

[0037] In an embodiment of the present application, a possible implementation method of passing the vehicle monitoring timing input matrix through a vehicle monitoring parameter timing feature encoder to obtain a vehicle monitoring timing feature matrix may be: using each layer of the vehicle monitoring parameter timing feature encoder to respectively perform the following on the input data in the forward pass of the layer: using a first convolution kernel to perform convolution processing on the input data to obtain a first convolution feature map; using a second convolution kernel to perform convolution processing on the first convolution feature map to obtain a second convolution feature map, wherein the size of the first convolution kernel is larger than the size of the second convolution kernel; performing pooling processing on the second convolution feature map to obtain a pooling feature map; performing activation processing on the pooling feature map to obtain an activation feature map; wherein the output of the last layer of the vehicle monitoring parameter timing feature encoder is the vehicle monitoring timing feature matrix, the input of the first layer of the vehicle monitoring parameter timing feature encoder is the vehicle monitoring timing input matrix, the size of the first convolution kernel is 5×5, and the size of the second convolution kernel is 3×3.

[0038] Figure 3 The present invention is a flowchart of a method for real-time monitoring of driver fatigue status based on multi-source data according to an embodiment of the present application, in which the driver's facial images at the plurality of predetermined time points are feature encoded to obtain a driver monitoring status temporal feature enhancement matrix. Figure 3As shown, the feature encoding of the driver's facial images at the multiple predetermined time points to obtain a driver monitoring status temporal feature enhancement matrix includes: S131, extracting the temporal features of the driver's facial images at the multiple predetermined time points based on the eye area and the mouth area to obtain a driver monitoring status temporal feature map; S132, performing feature enhancement and pooling on the driver monitoring status temporal feature map to obtain the driver monitoring status temporal feature enhancement matrix.

[0039] Specifically, Figure 4 The present invention is a flowchart of extracting the time series features of the driver's facial images at the plurality of predetermined time points based on the eye area and the mouth area to obtain the driver's monitoring state time series feature diagram in the real-time monitoring method of the driver's fatigue state based on multi-source data according to an embodiment of the present application. Figure 4 As shown, the driver face images at the multiple predetermined time points are subjected to time series feature extraction based on the eye area and the mouth area to obtain a driver monitoring status time series feature map, including: S1311, the driver face images at the multiple predetermined time points are passed through an eye target monitoring network to obtain a plurality of eye interest region images; S1312, the plurality of eye interest region images are respectively passed through an eye state feature encoder to obtain a plurality of eye state feature matrices; S1313, the driver face images at the multiple predetermined time points are passed through a mouth target monitoring network to obtain a plurality of mouth interest region images; S1314, the plurality of mouth interest region images are respectively passed through a mouth state feature encoder to obtain a plurality of mouth state feature matrices; S1315, each eye state feature matrix in the multiple eye state feature matrices and each mouth state feature matrix at the corresponding time point in the multiple mouth state feature matrices are respectively fused to obtain a plurality of driver state time series feature matrices, and the plurality of driver state time series feature matrices are aggregated along the channel dimension into the driver monitoring status time series feature map.

[0040] The state of the eyes is an important basis for assessing the driver's fatigue. When the driver is in a fatigued state, the eyes will show obvious characteristics, such as reduced blinking frequency, prolonged eye closure time, and wandering vision. These characteristic information is highly sensitive and accurate for judging the driver's fatigue state. In order to quickly and accurately extract the eye area from the driver's face image and eliminate the interference of other non-critical information, it is necessary to pass the driver's face images at the multiple predetermined time points through the eye target monitoring network to obtain multiple eye interest area images. By extracting the eye interest area images, the pertinence of data analysis can be improved, the efficiency of data processing can be improved, and unnecessary computing resource consumption can be reduced. The eye target monitoring network in this application is a target monitoring network based on anchor windows. The target monitoring network with anchor windows performs sliding scanning on the image to detect possible target areas by presetting a series of anchor windows of different scales and aspect ratios. This method can more comprehensively cover the potential target areas in the driver's face image, thereby improving the accuracy of detection. At the same time, due to the diversity and flexibility of the anchor window, the network can adapt to eye areas of different shapes and sizes, thereby improving the robustness of detection.

[0041] In order to automatically extract key features related to fatigue status from the eye region of interest image, these features may include the degree of opening and closing of the eyelids, the movement trajectory of the eyeball, etc., it is necessary to pass the multiple eye region of interest images through the eye state feature encoder to obtain multiple eye state feature matrices. It should be easy to understand that compared with directly analyzing the original image, the eye state feature encoder can automatically extract key features, which can significantly improve the analysis efficiency and speed up the judgment speed. In this application, the eye state feature encoder is a deep separable convolutional neural network model. The deep separable convolutional neural network model decomposes the standard convolution into two steps: deep convolution and point-by-point convolution. This decomposition method can significantly reduce the amount of calculation and the number of parameters, thereby reducing the computational complexity and storage requirements. This advantage is particularly obvious when processing large-scale image data. Although deep separable convolution reduces the computational complexity, this structure has advantages in maintaining or even improving model performance. By extracting feature information more finely, deep separable convolution can capture more subtle and complex image features, thereby improving the accuracy and robustness of eye state feature encoding.

[0042] When the driver is tired, in addition to the obvious fatigue characteristics of the eyes, the mouth may also show some movements that are different from the normal state. For example, yawning is a common physiological reaction when tired. By monitoring the degree of opening and duration of the mouth, it is possible to indirectly determine whether the driver is in a state of fatigue. In addition, long-term open mouth breathing, drooping corners of the mouth, etc. may also be related to fatigue. In order to analyze the driver's mouth state, it is necessary to first pass the driver's face images at the multiple predetermined time points through the mouth target monitoring network to obtain multiple mouth interest area images. The mouth target monitoring network in this application is a target monitoring network based on an anchor window. The target monitoring network based on the anchor window can capture the contour and position of the mouth more accurately, especially when the mouth is in different forms (such as open, closed, smiling, etc.), it can also maintain a high detection accuracy. At the same time, the use of a target monitoring network with an anchor window can adapt to the mouth area of ​​different expressions by adjusting the scale and aspect ratio of the anchor window, thereby improving the flexibility and robustness of the detection.

[0043] Taking into account the changes in the state of the mouth, such as yawning, mouth breathing, etc., these are common physiological reactions when the driver is tired. These changes can be captured by image analysis and converted into quantifiable features for subsequent fatigue state judgment. Therefore, the multiple mouth area of ​​interest images are respectively passed through the mouth state feature encoder to obtain multiple mouth state feature matrices. The mouth state feature encoder in this application is a deep separable convolutional neural network model. In the extraction of mouth state features, the deep separable convolutional neural network model can more accurately capture key features such as the contour of the mouth, the degree of opening, and changes in expression, thereby improving the accuracy and robustness of feature extraction. And because the deep separable convolutional neural network has a lower number of parameters, it is easier to avoid overfitting, which enables the model to better adapt to the changes in mouth states of different drivers and different driving scenarios.

[0044] Considering that the state of the eyes and mouth are both important indicators for assessing the degree of driver fatigue. The state of the eyes can reflect the degree of opening and closing of the eyelids, the frequency of blinking, etc., while the state of the mouth can reflect fatigue-related behaviors such as yawning and mouth breathing. In order to comprehensively assess the driver's fatigue state and improve the accuracy and comprehensiveness of the judgment, it is necessary to fuse each eye state feature matrix in the multiple eye state feature matrices and each mouth state feature matrix at the corresponding time point in the multiple mouth state feature matrices to obtain multiple driver state time series feature matrices. Considering that the multiple driver state time series feature matrices represent the driver monitoring features at different time points. The driver's fatigue state is a dynamically changing process, and the monitoring data at different time points can reflect the changing trend of its fatigue degree. In order to capture this dynamic change and more accurately judge the driver's fatigue state, the multiple driver state time series feature matrices can be aggregated along the channel dimension into a driver monitoring state time series feature diagram.

[0045] Figure 5 The present invention is a flowchart of a method for real-time monitoring of driver fatigue status based on multi-source data according to an embodiment of the present application, wherein the driver monitoring status time series feature graph is subjected to feature enhancement and then pooling to obtain the driver monitoring status time series feature enhancement matrix. Figure 5 As shown, the driver monitoring state timing feature graph is feature enhanced and then pooled to obtain the driver monitoring state timing feature enhancement matrix, including: S1321, the driver monitoring state timing feature graph is passed through a driver monitoring state timing feature enhancer to obtain a driver monitoring state timing feature enhancement graph; S1322, the driver monitoring state timing feature enhancement graph is pooled along the channel dimension to obtain the driver monitoring state timing feature enhancement matrix.

[0046] Although the driver monitoring state time series feature graph aggregates the feature information of multiple time points, it may still contain redundant or insignificant features, which may not be helpful for the subsequent fatigue state analysis, and may even interfere. In order to further screen and optimize the features in the feature graph, improve the expressiveness and discrimination of the features, and more accurately capture the key features related to the fatigue state, it is necessary to pass the driver monitoring state time series feature graph through the driver monitoring state time series feature enhancer to obtain the driver monitoring state time series feature enhanced graph. In this application, the driver monitoring state time series feature enhancer is a convolutional neural network model using a channel attention mechanism. The channel attention mechanism can automatically learn the importance of each channel feature and assign different weights to it. It should be understood that in the driver monitoring state time series feature graph, the contribution of features at different time points to the fatigue state analysis may be different. The features at some time points may be more critical and can more directly reflect the driver's fatigue state. The driver monitoring state time series feature enhancer can automatically emphasize the features of these important time points through learning, and suppress the features of those unimportant time points, thereby improving the accuracy and efficiency of fatigue state analysis.

[0047] In the driver monitoring status time series feature enhancement map, although the features have been optimized by the feature enhancer, the number of channels of the feature map may still be large, resulting in a large amount of calculation for subsequent processing. In order to reduce the number of features while retaining important information, thereby reducing the computational complexity and increasing the processing speed, the driver monitoring status time series feature enhancement map needs to be pooled along the channel dimension to obtain the driver monitoring status time series feature enhancement matrix. It should be understood that the pooling operation is not only for reducing the dimension, but more importantly, it can extract more important features from the feature map. Through the pooling operation, the features of the local area in the feature map can be aggregated to form more representative features, which can often better reflect the driver's fatigue state.

[0048] The driver's fatigue state is not only reflected in his physiological characteristics (such as the degree of eye closure, mouth state, etc.), but may also be indirectly reflected through his behavior and vehicle status. For example, when the driver is tired, he may have unstable steering wheel grip, large speed fluctuations, lane deviation, etc. In order to achieve multi-dimensional information complementarity, the driver monitoring state time series feature enhancement matrix and the vehicle monitoring time series feature matrix are fused in the technical solution of the present application to obtain a fatigue state monitoring related feature matrix. The fused fatigue state monitoring related feature matrix contains information on both the driver and the vehicle, so that the driver's fatigue state can be comprehensively evaluated. This comprehensive evaluation capability is more comprehensive and reliable than relying solely on a single data source, and can better adapt to the influence of different driving environments and individual differences of drivers.

[0049] Figure 6 The present invention is a flowchart of obtaining a driver fatigue monitoring result based on the fatigue state monitoring-related feature matrix in a real-time driver fatigue state monitoring method based on multi-source data according to an embodiment of the present application. Figure 6 As shown, the driver fatigue monitoring result is obtained based on the fatigue state monitoring related feature matrix, including: S151, expanding the fatigue state monitoring related feature matrix into a fatigue state monitoring related feature vector; S152, performing local fluctuation compensation on the fatigue state monitoring related feature vector based on the shift information representation to obtain an optimized fatigue state monitoring related feature vector; S153, passing the optimized fatigue state monitoring related feature vector through a classifier-based fatigue result monitoring generator to obtain a classification result, and the classification result is used to indicate whether the driver is in fatigue driving at the current time point.

[0050] Considering that the fatigue state monitoring-related feature matrix has a high dimension, directly using the feature matrix for classification will increase the computational complexity in the data processing process, thereby affecting the response speed of real-time monitoring. Therefore, in the technical solution of this application, it is necessary to expand the fatigue state monitoring-related feature matrix into a fatigue state monitoring-related feature vector. By expanding into a feature vector, the key information in the original feature matrix is ​​still retained, but the memory usage and calculation time in the calculation process can be reduced, thereby improving the overall efficiency and real-time performance of fatigue driving monitoring.

[0051] In particular, considering that the vehicle speed, acceleration, steering wheel steering angular velocity and other data are usually measured by on-board sensors, the sensors have precision errors, resulting in deviations between the measured data and the actual values, and the vehicle is also affected by the external environment during driving, such as road conditions, weather changes, etc. These factors will cause fluctuations in sensor data, thereby introducing noise interference in the encoding process of vehicle monitoring related data. In addition, the acquisition and processing of the driver's face image is also easily affected by factors such as illumination, angle, and expression changes, resulting in the extracted driver monitoring features being inaccurate. These factors will cause local feature input disturbances in the fatigue state monitoring related feature vector obtained by the final fusion. In practical applications, the fatigue monitoring system needs to work stably in various environments and conditions. In order to ensure that the system can still accurately identify the driver's fatigue state in the face of uncertainty and reduce false alarms or missed reports, in the technical solution of the present application, the fatigue state monitoring related feature vector is compensated for local fluctuations based on the shift information representation to obtain an optimized fatigue state monitoring related feature vector.

[0052] Specifically, in an embodiment of the present application, the fatigue state monitoring related feature vector is compensated for local fluctuations based on the shift information representation to obtain an optimized fatigue state monitoring related feature vector, including: determining the first fatigue classification weight matrix and the second fatigue classification weight matrix of the classifier before and after each iterative update; matrix multiplying the first fatigue classification weight matrix and the second fatigue classification weight matrix with the fatigue state monitoring related feature vector respectively to obtain a first fatigue state monitoring modulation feature vector and a second fatigue state monitoring modulation feature vector; calculating the position difference between the first fatigue state monitoring modulation feature vector and the second fatigue state monitoring modulation feature vector to obtain a fatigue state monitoring shift information. information representation vector; calculating the F norm of the position mean vector between the first fatigue state monitoring modulation feature vector and the second fatigue state monitoring modulation feature vector as a shift compensation scaling factor; linearly scaling the fatigue state monitoring shift information representation vector with the shift compensation scaling factor to obtain a scaled fatigue state monitoring shift information representation vector, and inputting the scaled fatigue state monitoring shift information representation vector into a Sigmoid activation function to obtain a fatigue state monitoring backpropagation shift compensation representation vector; calculating the position point multiplication of the fatigue state monitoring backpropagation shift compensation representation vector and the fatigue state monitoring related feature vector to obtain the optimized fatigue state monitoring related feature vector.

[0053] In an embodiment of the present application, another expression of performing local fluctuation compensation based on the shift information representation on the fatigue state monitoring related feature vector to obtain an optimized fatigue state monitoring related feature vector may be: processing the fatigue state monitoring related feature vector according to the following formula to obtain the optimized fatigue state monitoring related feature vector; wherein the formula is:

[0054]

[0055]

[0056] Among them, M 1 is the first fatigue classification weight matrix, M 2 is the second fatigue classification weight matrix, V is the fatigue state monitoring related feature vector, represents matrix multiplication, Indicates adding by position point, ||·|| F represents the F norm of the feature vector, S represents the shift compensation scaling factor, ⊙ represents the point multiplication by position, represents point-by-point subtraction, sigmoid represents an activation function, and V' represents the optimized fatigue state monitoring-related feature vector.

[0057] In order to enhance the robustness of the classifier to local disturbances in the input feature vector, the feature vector related to fatigue state monitoring is subjected to displacement compensation based on back propagation representation. The key to the displacement compensation based on back propagation representation of the feature vector related to fatigue state monitoring is to capture the information of the fatigue classification weight matrix changing with the training process, and to use this information to generate the fatigue state monitoring displacement information representation vector, and then generate the fatigue state monitoring back propagation displacement compensation representation vector through scaling and nonlinear transformation (Sigmoid function). In this way, the feature vector related to fatigue state monitoring is described by the local displacement of the feature vector supported by the distribution difference of different classification scenarios of the classifier during the training process, so that not only the displacement of the feature vector caused by the change of the fatigue classification weight matrix can be quantified, but also a compensation vector can be generated according to the displacement information. The role of the compensation vector is to weight and adjust each element in the feature vector according to the degree of influence of the disturbance, thereby reducing the influence of the disturbance on the classification result. In this way, the classifier's robustness to local disturbances in the input data can be significantly improved, the classifier's stability and generalization ability can be enhanced, and by reducing the impact of local disturbances on the classification results, the classifier can maintain a high classification accuracy when facing noisy or variant data.

[0058] Finally, the optimized fatigue state monitoring related feature vector is passed through the classifier-based fatigue result monitoring generator to obtain a classification result, and the classification result is used to indicate whether the driver is in fatigue driving at the current time point. It should be understood that the classifier is a machine learning model that can learn from the input data and automatically classify the data into predefined categories. In this scenario, the goal of the classifier is to automatically determine whether the driver is in a fatigue driving state based on the information of the optimized fatigue state monitoring related feature vector. It is not difficult to understand that after training, the classifier can learn the complex relationship between the feature matrix and the fatigue state, and make accurate judgments based on this. Compared with traditional threshold or rule-based methods, the classifier can handle more complex data patterns and feature combinations, thereby improving the accuracy of judgment. The classification result can tell the driver in real time whether he is currently in a fatigue driving state. When it is found that he is in fatigue driving, he can be reminded by issuing a sound alarm, a vibration prompt, etc. This real-time feedback helps the driver to adjust his driving state in time and avoid traffic accidents caused by fatigue.

[0059] In summary, the real-time monitoring method of driver fatigue status based on multi-source data according to the embodiment of the present application is explained, which adopts data processing technology based on deep learning, extracts features of the driver's eyes and mouth in the driver's facial data captured by the camera to obtain the driver's real state features, and extracts features of the relevant information of the driver driving the vehicle collected by various sensors to obtain the vehicle driving features, and combines the vehicle driving features and the driver's real state features to determine whether the driver is driving fatigued. In this way, real-time monitoring and accurate judgment of the driver's fatigue state can be achieved.

[0060] Figure 7 : is a system block diagram of a real-time monitoring system for driver fatigue status based on multi-source data according to an embodiment of the present application. Figure 7 As shown, according to the embodiment of the present application, the real-time monitoring system 100 of the driver fatigue state based on multi-source data includes: a fatigue state monitoring related data acquisition module 110, which is used to obtain the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, vehicle following distance and the driver's face image at the multiple predetermined time points within a predetermined time period; a vehicle monitoring data feature encoding module 120, which is used to perform time series encoding on the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity and vehicle following distance at the multiple predetermined time points to obtain a vehicle monitoring time series feature matrix; a driver monitoring data feature encoding module 130, which is used to perform feature encoding on the driver's face images at the multiple predetermined time points to obtain a driver monitoring state time series feature enhancement matrix; a multi-source monitoring feature fusion module 140, which is used to fuse the driver monitoring state time series feature enhancement matrix and the vehicle monitoring time series feature matrix to obtain a fatigue state monitoring related feature matrix; a fatigue monitoring result generation module 150, which is used to obtain a driver fatigue monitoring result based on the fatigue state monitoring related feature matrix.

[0061] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the above-mentioned driver fatigue status real-time monitoring system 100 based on multi-source data have been described in the above reference. Figures 1 to 6 The description of the real-time monitoring method of driver fatigue status based on multi-source data has been introduced in detail, and therefore, its repeated description will be omitted.

[0062] In summary, the real-time monitoring system 100 for driver fatigue status based on multi-source data according to the embodiment of the present application is explained, which adopts data processing technology based on deep learning, extracts features of the eye and mouth states in the driver's facial data captured by the camera to obtain the driver's real state features, and extracts features of the relevant information of the driver driving the vehicle collected by various sensors to obtain the vehicle driving features, and combines the vehicle driving features and the driver's real state features to determine whether the driver is driving fatigued. In this way, real-time monitoring and accurate judgment of the driver's fatigue state can be achieved.

[0063] As described above, the real-time monitoring system 100 for driver fatigue status based on multi-source data according to the embodiment of the present application can be implemented in various wireless terminals, such as a server for real-time monitoring of driver fatigue status based on multi-source data, etc. In one example, the real-time monitoring system 100 for driver fatigue status based on multi-source data according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the real-time monitoring system 100 for driver fatigue status based on multi-source data can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the real-time monitoring system 100 for driver fatigue status based on multi-source data can also be one of the many hardware modules of the wireless terminal.

[0064] Alternatively, in another example, the real-time monitoring system 100 for driver fatigue status based on multi-source data and the wireless terminal may also be separate devices, and the real-time monitoring system 100 for driver fatigue status based on multi-source data may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

Claims

1. A real-time monitoring method for driver fatigue status based on multi-source data, characterized in that: include: Acquire the vehicle speed, the vehicle instantaneous acceleration, the vehicle steering angular velocity, the vehicle following distance and the driver's face image at multiple predetermined time points within a predetermined time period; Performing time series coding on the vehicle speed, the vehicle instantaneous acceleration, the vehicle steering wheel steering angular velocity, and the vehicle following distance at the plurality of predetermined time points to obtain a vehicle monitoring time series feature matrix; Performing feature encoding on the driver's facial images at the plurality of predetermined time points to obtain a driver monitoring state temporal feature enhancement matrix; Fusion of the driver monitoring state time series feature enhancement matrix and the vehicle monitoring time series feature matrix to obtain a fatigue state monitoring related feature matrix; Based on the fatigue state monitoring related feature matrix, a driver fatigue monitoring result is obtained.

2. The method for real-time monitoring of driver fatigue status based on multi-source data according to claim 1 is characterized in that: The vehicle speed, the vehicle instantaneous acceleration, the vehicle steering angular velocity, and the vehicle following distance at the plurality of predetermined time points are time-series encoded to obtain a vehicle monitoring time-series feature matrix, including: Arrange the vehicle speed, vehicle instantaneous acceleration, vehicle steering angular velocity, and vehicle following distance at the plurality of predetermined time points according to the time dimension to obtain a vehicle monitoring timing input matrix; The vehicle monitoring timing input matrix is ​​passed through a vehicle monitoring parameter timing characteristic encoder to obtain the vehicle monitoring timing characteristic matrix.

3. The method for real-time monitoring of driver fatigue status based on multi-source data according to claim 2 is characterized in that: Performing feature encoding on the driver's facial images at the plurality of predetermined time points to obtain a driver monitoring state temporal feature enhancement matrix includes: Extracting time series features based on eye regions and mouth regions from the driver's face images at the plurality of predetermined time points to obtain a driver monitoring state time series feature graph; The driver monitoring state time series feature graph is feature enhanced and then pooled to obtain the driver monitoring state time series feature enhancement matrix.

4. The method for real-time monitoring of driver fatigue status based on multi-source data according to claim 3 is characterized in that: Extracting the time series features of the driver's face images at the plurality of predetermined time points based on the eye region and the mouth region to obtain a driver monitoring state time series feature graph includes: Passing the driver's face images at the plurality of predetermined time points through an eye target monitoring network to obtain a plurality of eye region of interest images; Passing the plurality of eye region of interest images through eye state feature encoders respectively to obtain a plurality of eye state feature matrices; Passing the driver's face images at the plurality of predetermined time points through a mouth target monitoring network to obtain a plurality of mouth region of interest images; Passing the plurality of mouth region of interest images through a mouth state feature encoder respectively to obtain a plurality of mouth state feature matrices; Each eye state feature matrix in the multiple eye state feature matrices and each mouth state feature matrix at the corresponding time point in the multiple mouth state feature matrices are fused separately to obtain multiple driver state timing feature matrices, and the multiple driver state timing feature matrices are aggregated along the channel dimension into the driver monitoring state timing feature graph.

5. The method for real-time monitoring of driver fatigue status based on multi-source data according to claim 4 is characterized in that: The driver monitoring state time series feature graph is subjected to feature enhancement and then pooling to obtain the driver monitoring state time series feature enhancement matrix, including: Passing the driver monitoring state time series characteristic diagram through a driver monitoring state time series characteristic enhancer to obtain a driver monitoring state time series characteristic enhanced diagram; The driver monitoring state temporal feature enhancement map is pooled along the channel dimension to obtain the driver monitoring state temporal feature enhancement matrix.

6. The method for real-time monitoring of driver fatigue status based on multi-source data according to claim 5 is characterized in that: Based on the fatigue state monitoring related feature matrix, a driver fatigue monitoring result is obtained, including: Expanding the fatigue state monitoring related feature matrix into a fatigue state monitoring related feature vector; Performing local fluctuation compensation based on the shift information representation on the fatigue state monitoring related feature vector to obtain an optimized fatigue state monitoring related feature vector; The optimized fatigue state monitoring related feature vector is passed through a classifier-based fatigue result monitoring generator to obtain a classification result, and the classification result is used to indicate whether the driver is in fatigue driving at the current time point.

7. The method for real-time monitoring of driver fatigue status based on multi-source data according to claim 6 is characterized in that: The fatigue state monitoring related feature vector is subjected to local fluctuation compensation based on the shift information representation to obtain an optimized fatigue state monitoring related feature vector, including: Determine a first fatigue classification weight matrix and a second fatigue classification weight matrix of the classifier before and after each iterative update; Performing matrix multiplication of the first fatigue classification weight matrix and the second fatigue classification weight matrix with the fatigue state monitoring related feature vector respectively to obtain a first fatigue state monitoring modulation feature vector and a second fatigue state monitoring modulation feature vector; Calculating the position difference between the first fatigue state monitoring modulation feature vector and the second fatigue state monitoring modulation feature vector to obtain a fatigue state monitoring shift information representation vector; Calculating the F-norm of the position mean vector between the first fatigue state monitoring modulation feature vector and the second fatigue state monitoring modulation feature vector as a shift compensation scaling factor; Linearly scaling the fatigue state monitoring shift information representation vector by the shift compensation scaling factor to obtain a scaled fatigue state monitoring shift information representation vector, and inputting the scaled fatigue state monitoring shift information representation vector into a Sigmaid activation function to obtain a fatigue state monitoring back-propagation shift compensation representation vector; The optimized fatigue state monitoring related feature vector is obtained by calculating the point-by-point multiplication of the fatigue state monitoring back-propagation shift compensation representation vector and the fatigue state monitoring related feature vector.

8. A real-time monitoring system for driver fatigue status based on multi-source data, characterized in that: include: A fatigue status monitoring related data acquisition module is used to obtain the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, vehicle following distance and driver's face image at multiple predetermined time points within a predetermined time period; A vehicle monitoring data feature encoding module, used for performing time series encoding on the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the plurality of predetermined time points to obtain a vehicle monitoring time series feature matrix; A driver monitoring data feature encoding module, used for performing feature encoding on the driver's facial images at the plurality of predetermined time points to obtain a driver monitoring state temporal feature enhancement matrix; A multi-source monitoring feature fusion module, used to fuse the driver monitoring state time series feature enhancement matrix and the vehicle monitoring time series feature matrix to obtain a fatigue state monitoring related feature matrix; The fatigue monitoring result generating module is used to obtain the driver fatigue monitoring result based on the fatigue state monitoring related feature matrix.

9. The real-time monitoring system for driver fatigue status based on multi-source data according to claim 8 is characterized in that: The vehicle monitoring data feature encoding module includes: A vehicle monitoring data structured arrangement unit, used for arranging the vehicle speed, vehicle instantaneous acceleration, vehicle steering wheel steering angular velocity, and vehicle following distance at the plurality of predetermined time points according to the time dimension to obtain a vehicle monitoring time series input matrix; The vehicle monitoring time series feature generating unit is used to pass the vehicle monitoring time series input matrix through a vehicle monitoring parameter time series feature encoder to obtain the vehicle monitoring time series feature matrix.

10. The real-time monitoring system for driver fatigue status based on multi-source data according to claim 9, wherein the fatigue monitoring result generating module comprises: A fatigue state monitoring related feature representation dimension reduction unit, used for expanding the fatigue state monitoring related feature matrix into a fatigue state monitoring related feature vector; A fatigue state monitoring related feature optimization unit, used for performing local fluctuation compensation based on the shift information representation on the fatigue state monitoring related feature vector to obtain an optimized fatigue state monitoring related feature vector; The optimized fatigue state monitoring related feature classification unit is used to pass the optimized fatigue state monitoring related feature vector through a classifier-based fatigue result monitoring generator to obtain a classification result, and the classification result is used to indicate whether the driver is in fatigue driving at the current time point.

Images