Real-time Monitoring Method for Driver Fatigue State Based on Multimodal Biometric Fusion

Abstract

The present application provides a real-time monitoring method for driver fatigue state based on multimodal biometric fusion, which relates to the technical field of fatigue alarm. The method includes: deploying monitoring components to sample the driver's posture; embedding and deploying a first decision-making unit and a second decision-making unit; connecting a fatigue alarm, and according to continuous monitoring frames, coordinating the first decision-making unit and the second decision-making unit to give a status reminder to the driver. Among them, the first decision-making unit makes a spatio-temporal dimension decision through the Euler angles of the head posture, and the second decision-making unit makes a spatio-temporal dimension decision through the correction based on the Euler angles of the head posture and the multimodal biometric fusion determined based on the aiming frame vector. By means of the present application, the technical problem that the monitoring accuracy is low and false alarms and missed alarms are likely to occur due to relying on a single facial feature and not considering individual characteristics is solved. Through multimodal biometric fusion and two-level decision-making units, the accuracy and efficiency of fatigue early warning are improved.

Description

Technical Field

[0001] This application relates to the technical field of fatigue alarm, and particularly to a real-time monitoring method for driver fatigue state based on multimodal biometric fusion. Background Art

[0002] With the rapid development of the modern transportation industry, vehicle driving safety has gradually become the focus of social attention. The fatigue state of drivers is one of the important factors affecting driving safety. Long-term driving or night driving is likely to cause drivers to be inattentive and have slow reactions, which may lead to traffic accidents. Currently, fatigue driving detection technologies mainly include methods based on physiological signal detection, driving behavior analysis, and facial feature recognition. They mainly rely on single facial parameters such as eye closure rate and mouth opening rate, and are easily affected by factors such as light, individual differences of drivers (such as eye size and face shape), and posture differences (such as tilting the head, which cannot effectively measure the state of the upper and lower eyes), resulting in misjudgment and false alarms.

[0003] In summary, there are technical problems in the prior art that due to relying on single facial features and not considering individual characteristics, the monitoring accuracy is low, and false alarms and missed alarms are likely to occur. Summary of the Invention

[0004] The purpose of this application is to provide a real-time monitoring method for driver fatigue state based on multimodal biometric fusion, so as to solve the technical problems in the prior art that due to relying on single facial features and not considering individual characteristics, the monitoring accuracy is low, and false alarms and missed alarms are likely to occur.

[0005] In view of the above problems, this application provides a real-time monitoring method for driver fatigue state based on multimodal biometric fusion. Among them, the real-time monitoring method for driver fatigue state based on multimodal biometric fusion includes: deploying monitoring components, establishing channel interaction between the monitoring components and the vehicle control center, controlling the monitoring components to sample the driver's posture to determine continuous monitoring frames; embedding and deploying a first decision-making device and a second decision-making device, where the first decision-making device is deployed in the scanning component and the second decision-making device is deployed in the vehicle control center; connecting a fatigue alarm, and according to the continuous monitoring frames, coordinating the first-order fuzzy decision-making of the first decision-making device and the fatigue state alarm, and the second-order decision-making of the second decision-making device and the fatigue state alarm to give a state reminder to the driver; among them, the first decision-making device makes a spatio-temporal dimension decision through the Euler angles of the head posture, and the second decision-making device makes a spatio-temporal dimension decision based on the correction of the head posture Euler angles and the multimodal biometric fusion determined based on the aiming frame vector.

[0006] Optionally, the Euler angles of the head posture include pitch angle, yaw angle and roll angle, which are determined with reference to the baseline three-dimensional coordinate system in the standard posture; for the continuous monitoring frames, the angular vector in the time and space dimension is measured based on the Euler angles of the head posture, wherein the angular vector is determined by the frame angle value and the angle trend characteristics based on adjacent frames; fatigue status decision and alarm are made based on the angular vector.

[0007] Optionally, a posture threshold is set, wherein the posture threshold comprises a dynamic posture threshold and a static posture threshold set based on the Euler angle of the head posture; and it is determined whether the angular vector satisfies the posture threshold. If so, the fatigue alarm is controlled to execute an alarm based on the fatigue state.

[0008] Optionally, if it is not satisfied, the continuous monitoring frame is transmitted back, and combined with the second decision maker, a posture aiming frame is determined, wherein the posture aiming frame is determined based on the fatigue key posture part and corresponds one to one with the continuous monitoring frame; according to the angle vector, the posture aiming frame is corrected to determine a corrected posture aiming frame, wherein the correction requirement is based on the standard posture; based on the corrected posture aiming frame, an aiming frame vector is determined; based on the aiming frame vector, multimodal biometric features are extracted.

[0009] Optionally, historical posture data is called and posture data is extracted, the Euler angle of the head posture is used as the independent variable, and the trend of the key posture part is used as the dependent variable to mine the linear variation relationship; according to the linear variation relationship, the posture aiming frame is subjected to the aiming frame point cloud correction based on the angle vector to determine the corrected posture aiming frame.

[0010] Optionally, the correction posture aiming frame is traversed to determine an aiming frame trend change feature based on the trend change of the mapped aiming frame point cloud of adjacent frames; and the aiming frame vector in the spatiotemporal dimension is measured according to the aiming frame trend change feature.

[0011] Optionally, a multi-level alarm mode is introduced and a fatigue alarm is constructed, wherein the multi-level alarm mode is limited based on the alarm mode and the alarm intensity; the fatigue alarm device is deployed and connected with the first decision maker and the second decision maker; the fatigue alarm responds to the output of the first decision maker and the second decision maker to warn the driver in a fatigue state.

[0012] Optionally, the monitoring component includes a visual monitoring component and a physiological monitoring component, wherein the visual monitoring is primary and the physiological monitoring is secondary; the continuous monitoring frames obtained by the visual monitoring component are combined with the physiological signals obtained by the physiological monitoring component to make fatigue status decisions and alarm reminders.

[0013] Optionally, according to the driving time, a periodic sampling frequency based on fatigue trend is mined; and according to the periodic sampling frequency, the monitoring component is configured.

[0014] Optionally, track and monitor the driver's response status; adjust the alarm mode according to the response status, including increasing-direction continuous alarm in the non-response state, continuous alarm in the weak-response state, and alarm termination in the expected-response state.

[0015] One or more technical solutions provided in this application have at least the following beneficial effects:

[0016] By deploying a monitoring component, establishing channel interaction between the monitoring component and the vehicle control center, controlling the monitoring component to sample the driver's posture, and determining continuous monitoring frames; embedding and deploying a first decision maker and a second decision maker, where the first decision maker is deployed in the scanning component and the second decision maker is deployed in the vehicle control center; connecting a fatigue alarm, and according to the continuous monitoring frames, coordinating the first-order fuzzy decision of the first decision maker and the fatigue status alarm, and the second-order decision of the second decision maker and the fatigue status alarm to give a status reminder to the driver; where the first decision maker makes a spatio-temporal dimension decision through the Euler angles of the head posture, and the second decision maker makes a spatio-temporal dimension decision by fusing multi-modal biometrics based on the correction of the Euler angles of the head posture and the aiming frame vector. That is to say, by fusing multiple biometrics, sampling the driver's posture, comprehensively evaluating the driver's fatigue status, deploying the first decision maker and the second decision maker, and evaluating the fatigue status at different levels, false alarms and missed alarms are reduced, and the accuracy and efficiency of fatigue warning are improved.

[0017] The above description is only an overview of the technical solution of this application. In order to be able to understand the technical means of this application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of this application more obvious and understandable, the following specific embodiments of this application are specifically given. It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of this application, nor is it used to limit the scope of this application. Other features of this application will become easily understood through the following description. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in this application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0019] Figure 1 It is a schematic flowchart of the method for real-time monitoring of the driver's fatigue status by multi-modal biometric fusion in this application.

[0020] Figure 2 This is a schematic flowchart for fatigue state alarm in the real-time monitoring method of driver fatigue state with multimodal biometric fusion in this application. Specific embodiments

[0021] By providing a real-time monitoring method for driver fatigue state with multimodal biometric fusion, this application solves the technical problems existing in the prior art. By fusing multiple biometric features, driver posture sampling is carried out to comprehensively evaluate the fatigue state of the driver. The first decision maker and the second decision maker are deployed to evaluate the fatigue state at different levels, thereby reducing false alarms and missed alarms and improving the accuracy and efficiency of fatigue warning.

[0022] Next, the technical solutions in this application will be described clearly and completely with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments of this application. It should be understood that this application is not limited by the example embodiments described here. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application. In addition, it should be noted that for the convenience of description, only the parts related to this application are shown in the accompanying drawings rather than all of them.

[0023] Embodiment, please refer to the attached Figure 1 , this application provides a real-time monitoring method for driver fatigue state with multimodal biometric fusion. Among them, the real-time monitoring method for driver fatigue state with multimodal biometric fusion specifically includes the following steps:

[0024] S100: Deploy monitoring components, establish channel interaction between the monitoring components and the vehicle control center, control the monitoring components to perform driver posture sampling, and determine continuous monitoring frames.

[0025] Specifically, according to the configured monitoring components, they are deployed and installed in appropriate positions. For example, the visual monitoring component needs to be installed inside the vehicle, facing the driver, to ensure that the facial expressions and eye states of the driver can be clearly captured; the heart rate monitor or other physiological signal monitoring devices are fixed on the driver, the steering wheel, or integrated into the seat. Provide necessary power supply and data processing units for the monitoring components, connect the monitoring components to the data processing unit of the vehicle through wired or wireless networks, and ensure that the vehicle data processing unit can establish a stable communication connection with the vehicle control center through methods such as 4G / 5G networks and satellite communications.

[0026] The vehicle control center sends control instructions to the monitoring component to start sampling the driver's posture. The visual monitoring component begins to capture consecutive images of the driver at a set frame rate (e.g., 15 FPS), while the physiological monitoring component records physiological data at a predetermined sampling interval (such as once every 10 seconds). The continuously acquired image frames and physiological signal data are aligned by timestamp to form a series of consecutive monitoring frames. Posture data of the driver's head, face, etc. are obtained through the monitoring component, such as the head Euler angles (pitch, yaw, roll angles), eye states, etc. Consecutive monitoring frames refer to a data sequence formed by consecutive sampling (such as consecutive video frames or sensor data) for subsequent analysis of the driver's behavior changes and fatigue trends. By deploying the monitoring component and establishing a stable channel interaction with the vehicle control center, real-time and continuous sampling of the driver's posture is achieved to promptly detect signs of driver fatigue.

[0027] Furthermore, S100 of this application includes:

[0028] S110: The monitoring component includes a visual monitoring component and a physiological monitoring component, where the visual monitoring is the main and the physiological monitoring is the auxiliary; S120: Using the consecutive monitoring frames obtained by the visual monitoring component and combining with the physiological signals obtained by the physiological monitoring component to make a decision on the fatigue state and give an alarm reminder.

[0029] Specifically, the monitoring component consists of a visual monitoring component and a physiological monitoring component, where the visual monitoring is the main and the physiological monitoring is the auxiliary. The visual monitoring component makes a decision on the driver's fatigue state and gives an alarm reminder by obtaining consecutive monitoring frames and combining with the physiological signals collected by the physiological monitoring component. The visual monitoring component is usually installed in the cab to capture the driver's facial images in real time and generate consecutive monitoring frames, which contain the driver's facial expressions and eye movement information. The consecutive monitoring frames are pre-processed, such as grayscale conversion, noise removal, etc., to improve the image quality.

[0030] Use a face detection algorithm (such as Haar features + AdaBoost classifier) to locate the face region, and further detect key parts such as eyes and mouth. Usually, a pre-trained model is used to analyze the image and return the coordinates of the detected face region. Within the detected face region, eye and mouth detectors are respectively used for further detection to obtain the positions of the eyes and mouth. Rectangular frames are drawn on the original image to mark the detected face, eye, and mouth regions.

[0031] The physiological signals of the driver are collected in real time through physiological sensors (such as electrocardiogram sensors) to obtain physiological signals such as the driver's heart rate and breathing rate. Since the driver's physiological states at different stages are different, combined with the physiological sensors for auxiliary judgment to determine whether the driver is in a drowsy state.

[0032] Combine visual features (such as eye closure time, blink frequency) with physiological features (such as heart rate, heart rate variability) to determine the fatigue state of the driver. When it is detected that the driver is in a fatigued state, trigger an alarm mechanism, such as a sound prompt, seat vibration, etc., to remind the driver to pay attention to safe driving. By combining visual monitoring and physiological monitoring, accurately identify the fatigue state of the driver. Visual monitoring provides intuitive behavioral information, while physiological monitoring provides internal physiological state information, and the combination of the two improves the reliability and sensitivity of monitoring.

[0033] Furthermore, this application also includes the following steps:

[0034] S130: According to the driving time, mine the periodic sampling frequency based on the fatigue trend; S140: Configure the monitoring component according to the periodic sampling frequency.

[0035] Specifically, according to the driving time, that is, the duration of continuous driving of the driver, which is used to measure the time dimension of fatigue accumulation during driving. Usually obtained from vehicle driving data (such as odometer, GPS data) or driver operation records (such as ignition time, braking frequency). The fatigue trend refers to the change trend of the driver's fatigue state as the driving time increases. Usually, the driver's attention will decline after long-term driving and show fatigue characteristics (such as increased blink frequency, heart rate changes, etc.). According to the driving time data, combined with the relationship between the fatigue state and the driving time, determine the general fatigue trend of the driver. According to the fatigue trend. For example, assume that in a continuous driving data collection of 50 drivers, it is found that: the fatigue risk of drivers is higher in the afternoon, followed by the evening, and the lowest in the morning. Therefore, the sampling frequency in the afternoon is set to once every 5 minutes, the sampling frequency in the evening is set to once every 8 minutes, and in the morning it may be set to once every 10 minutes.

[0036] According to the fatigue trend curve, set different monitoring frequencies in different time periods. For example, different driving durations will also result in different fatigue risks. The fatigue indicators for driving duration are shown in Table 1:

[0037] Table 1 Fatigue indicators for driving duration

[0038]

[0039] Adjust the sampling frequency, sampling interval, sensor parameters, etc. of the monitoring component according to the calculated periodic sampling frequency. For example, for the visual monitoring component, adjust the frame rate of the camera or the image processing frequency to avoid the sampling state affecting fatigue judgment; for the physiological monitoring component, adjust the frequency of physiological signal acquisition, such as the data reading interval of the heart rate monitor. Configure the monitoring system in the actual driving environment and test it to ensure that the adjustment of the sampling frequency can effectively capture the fatigue state. Further optimize the sampling frequency strategy according to the test results to improve the accuracy and efficiency of the monitoring system. By periodically adjusting the sampling frequency, accurately capture the fatigue state of the driver, avoid data redundancy and waste of computing resources caused by too high sampling frequency, improve the sensitivity and real-time performance of fatigue monitoring, especially in long-term driving scenarios, which helps to warn the driver of fatigue in advance and improve driving safety.

[0040] S200: Embeddedly deploy the first decision maker and the second decision maker, where the first decision maker is deployed in the scanning component and the second decision maker is deployed in the vehicle control center.

[0041] Specifically, the first decision maker and the second decision maker are embeddedly deployed in different parts of the vehicle. The first decision maker is located inside the scanning component and is responsible for preliminary processing and judgment of the original data. Fuzzy logic or fast rule algorithms are often used to screen possible abnormal states. The second decision maker is deployed in the vehicle control center and is responsible for further fusing information from multiple data sources (such as visual data, head pose data, and physiological signals), and using more complex decision algorithms for fine-grained judgment.

[0042] The first decision maker uses a fuzzy decision algorithm to make real-time judgments on the driver's head pose data (such as pitch angle, yaw angle). For example, if abnormal head angles of the driver are detected (such as lowering the head, tilting the head), it is preliminarily judged that fatigue or distraction may occur. The second decision maker is usually located in the vehicle control center and needs to have sufficient computing power and storage space. Combining the output results of the first decision maker, conduct a more complex analysis of the fatigue state, improve the accuracy rate, and reduce misjudgment. Combine visual data (head pose, eye opening and closing state) + physiological data (heart rate, skin electrical activity, etc.) for more advanced fatigue detection. When the second decision maker finally determines that the driver is in a fatigue state, trigger the alarm system. Through the multi-level decision-making structure of the first decision maker and the second decision maker, accurate detection of the driver's fatigue state is achieved, and effective reminders are combined with the fatigue alarm, improving the accuracy and real-time performance of the detection, and providing intelligent guarantee for driving safety.

[0043] S300: Connect the fatigue alarm, and according to the continuous monitoring frames, cooperate with the first-order fuzzy decision of the first decision maker and the fatigue state alarm, and the second-order decision of the second decision maker and the fatigue state alarm to give state reminders to the driver.

[0044] S400: Among them, the first decision maker makes spatio-temporal dimension decisions through the Euler angles of the head pose, and the second decision maker makes spatio-temporal dimension decisions by fusing multi-modal biometrics determined based on the correction of the Euler angles of the head pose and the aiming frame vector.

[0045] Specifically, the vehicle control center is connected to the scanning component and the fatigue alarm, sends the continuous detection frames obtained by the scanning component to the first decision maker, and makes a first-order fuzzy decision based on the Euler angles of the driver's head pose (pitch, yaw, roll) to quickly screen out suspected fatigue states. For example, a pitch angle greater than 20 degrees and lasting for 5 seconds may indicate fatigue; an eye closure time ratio greater than 30% is suspected of fatigue; an increase in the number of mouth openings (yawn frequency) is suspected of fatigue. According to the preliminary judgment result of the first decision maker, a fatigue state alarm is triggered. The first-order fuzzy decision is a decision-making method that uses fuzzy logic to process uncertain and imprecise information to obtain a preliminary decision result. For example, at the 15th minute of continuous driving, the average pitch angle of the driver reaches 22° and lasts for 3 seconds. At this time, the abnormal frame ratio is recorded as 7%, and a fatigue state alarm is triggered.

[0046] When the first decision maker does not detect that the Euler angles of the head pose meet the pose threshold, the preliminary decision result and the original monitoring data are sent to the second decision maker of the vehicle control center. The second decision maker corrects the received Euler angles of the head pose according to the determination result of the first decision maker to eliminate the influence caused by the camera angle deviation or environmental light change. Biometrics of key areas such as eyes and mouth are extracted from the driver's facial image using the aiming frame vector, and combined with physiological signals such as heart rate, and a second-order spatio-temporal dimension decision is made through weighted fusion, such as 0.7 for visual features and 0.3 for physiological features. That is to say, the multi-modal biometrics are spatio-temporally aligned, and the corrected head pose data is integrated with the facial features and physiological signals extracted based on the aiming frame vector, considering both the instantaneous performance of the features and their change trends over time.

[0047] Use the multimodal data fusion algorithm to integrate information from different data sources, which can not only compensate for the deficiencies of a single data source, but also provide more comprehensive driver state information. Perform decision-making on the continuously monitored frame data using a spatio-temporal analysis model on the fused data to determine whether the driver shows signs of fatigue. According to the decision result in the spatio-temporal dimension, output the final fatigue state determination. Synchronize the feature data from different sensors in time, and use timestamps to align the visual and physiological data to ensure that the two sets of data can reflect the driver state at the same moment. Adopt the feature-level fusion method to integrate visual features (such as head pose, PERCLOS) and physiological features (such as heart rate, skin conductance change) into a multi-dimensional feature vector. Use weighted average to adjust the importance of each modal data to obtain a more comprehensive description of the driver state. Usually, the weight of visual features is 0.7 and the weight of physiological features is 0.3.

[0048] On the fused feature sequence, adopt a spatio-temporal analysis model, such as a long short-term memory network, to learn the correlation relationship between each component in the feature vector, such as the complementary information between the head pose and the eye closure state. The long short-term memory network uses its internal memory units to gradually capture the dynamic change trend of the sequence data, and predicts that the trend will lead to a fatigue state. After being processed by the spatio-temporal analysis model, output a fatigue probability value to determine whether the driver is in a fatigue state. Connect a fatigue alarm to quickly convert this judgment result into a reminder signal, such as a sound alarm, seat vibration, or dashboard display warning, to prompt the driver to take a rest immediately.

[0049] Feed the real-time monitoring results back to the vehicle control center for subsequent model adjustment and system optimization. Through continuous iteration, the accuracy and response speed of the model are further improved. Through spatio-temporal dimension decision-making with multimodal biometric fusion, accurately capture the subtle changes in the driver state over time, improving the accuracy and real-time performance of fatigue detection.

[0050] Furthermore, the present application further includes the following steps:

[0051] S410: The Euler angles of the head pose include the pitch angle, yaw angle, and roll angle, which are determined with reference to the baseline three-dimensional coordinate system in the standard pose; S420: For the continuous monitoring frames, measure the angular vector in the spatio-temporal dimension based on the Euler angles of the head pose, where the angular vector is determined by the frame angle value and the angle change trend feature based on adjacent frames; S430: Make a fatigue state decision and alarm according to the angular vector.

[0052] Specifically, the Euler angles of the head pose are three rotation angles that describe the driver's head relative to the baseline coordinate system in three-dimensional space, including the pitch angle, yaw angle, and roll angle. The pitch angle is the angle at which the head tilts forward or backward, the yaw angle is the angle at which the head turns left or right, and the roll angle is the angle at which the head rolls left or right. They are all determined with reference to the baseline three-dimensional coordinate system in the standard pose to ensure that the data has a unified standard. The detected head features are compared with the baseline coordinate system to determine the deviation of each angle; for example, if the pitch angle of the driver in the standard pose is 0°, and the actual detected value is 20°, it can be considered that the driver's head tilts forward by 20°.

[0053] Record the Euler angles of the head pose calculated for each frame of the continuously monitored frames, and compare the angle changes between adjacent frames to extract the angle change features. For example, the Euler angles detected in one frame may be a pitch angle of 18°, a yaw angle of 4°, and a roll angle of 2°. Record the specific values of the Euler angles of the head pose for each frame, that is, the frame angle values. In order to capture the dynamic changes of the driver's state, compare the angle data between adjacent frames and calculate the change amount of the angle, that is, the angle change trend feature. For example, if the pitch angle of the 10th frame is 18° and the 11th frame is 20°, then the change trend of the pitch angle is +2°.

[0054] Combine the static angle values of each frame with the angle change trend features of adjacent frames to form a complete angle vector, which reflects the instantaneous state and dynamic change trend of the driver's head pose. The angle vector is a multi-dimensional feature vector used to measure the changes in the driver's head pose in the spatio-temporal dimension, considering both the static angle information of each frame and the change trend between consecutive frames, thus comprehensively reflecting the changes in the driver's state.

[0055] Compare the frame angle values and angle change trend features of the angle vector with the dynamic pose threshold and static pose threshold of the pose threshold to determine whether there is a fatigue state. If so, send it to the fatigue alarm for alarm. By analyzing the subtle changes in the head pose, accurately identifying the signs of fatigue helps prevent traffic accidents caused by fatigue driving.

[0056] Furthermore, the present application further includes the following steps:

[0057] S431: Set the pose threshold, where the pose threshold is a dynamic pose threshold and a static pose threshold set based on the Euler angles of the head pose; S432: Determine whether the angle vector meets the pose threshold. If it meets, control the fatigue alarm to execute an alarm based on the fatigue state.

[0058] Specifically, the posture threshold is a reference standard value used to determine whether the driver's head posture is abnormal, including a dynamic posture threshold and a static posture threshold. Among them, the dynamic posture threshold is a threshold set based on the trend of the Euler angles of the head posture changing over time, and is used to capture the dynamic fluctuations of the driver's head posture; the static posture threshold is a threshold set according to the absolute value of the Euler angles of the head posture at a certain moment, and is used to reflect whether the driver is in a certain fixed abnormal posture state.

[0059] The angular vectors of consecutive monitoring frames are judged using the set posture thresholds. That is to say, the angle change characteristics of adjacent frames are compared with the dynamic posture threshold, and at the same time, the angle value of each frame is compared with the static posture threshold to determine whether the driver's head posture exceeds the set range. If it is determined that the angular vector meets the set posture threshold, the fatigue alarm is controlled to execute an alarm based on the fatigue state. That is to say, only when the angular vector shows that the driver's head posture is abnormal in space or time, an alarm signal is triggered, such as a sound prompt, a steering wheel vibration, or a dashboard warning, so as to remind the driver to pay attention to safe driving. By setting the dynamic and static posture thresholds, the fatigue state can be accurately identified, reducing unnecessary alarms and missing real fatigue signs.

[0060] Furthermore, the present application further includes the following steps:

[0061] S433: If not satisfied, the consecutive monitoring frames are transmitted back, and in combination with the second decision maker, a posture aiming frame is determined, where the posture aiming frame is determined based on the key posture parts of fatigue and corresponds one-to-one with the consecutive monitoring frames; S434: According to the angular vector, the posture aiming frame is corrected to determine a corrected posture aiming frame, where the correction requirement is in the standard posture; S435: Based on the corrected posture aiming frame, a aiming frame vector is determined; S436: Based on the aiming frame vector, multi-modal biometric features are extracted.

[0062] S4341: Call historical posture data and extract posture data. Taking the Euler angles of the head posture as independent variables and the changes of the key posture parts as dependent variables, a linear change relationship is mined; S4342: According to the linear change relationship, the posture aiming frame is corrected based on the angular vector to determine the corrected posture aiming frame.

[0063] S4351: Traverse the corrected posture aiming frame to determine the aiming frame change characteristics based on the mapped aiming frame point cloud change of adjacent frames; S4352: According to the aiming frame change characteristics, the aiming frame vector in the space-time dimension is measured.

[0064] Specifically, when the angular vector does not meet the posture threshold, that is, it is preliminarily determined that there is no fatigue driving state, the continuous monitoring frames are transmitted back to the second decision maker. The second decision maker combines the received continuous monitoring frames and the driver's facial region information therein, and uses a pre-trained face detection algorithm to determine the posture aiming frame, that is, the target region focusing on the key fatigue posture parts (such as the face and eyes). Each aiming frame corresponds to the corresponding frame one by one. For example: Monitoring frame 1: Posture aiming frame position (x = 100, y = 50, width = 200, height = 220), Monitoring frame 2: Posture aiming frame position (x = 102, y = 52, width = 198, height = 218). During the determination of the aiming frame, a filtering algorithm is used to smooth the position change of the aiming frame and reduce the error caused by camera jitter or light change. The key fatigue posture parts mainly include regions closely related to the driver's fatigue state, such as the face and eyes. For example, long-term eye closure indicates fatigue, and expression changes such as muscle relaxation and yawning may indicate fatigue. In each frame of the image, a posture aiming frame matching the current driver state is generated. The posture aiming frame is a target region box defined based on the driver's key fatigue posture parts (such as the face and eyes), which is used to focus on the detection object and improve the accuracy of feature extraction.

[0065] Call the historical posture data of the driver, that is, call the stored head posture information of the driver in the past, and extract the head posture Euler angles and the movement trajectories of the key posture parts within a period of time, which can be used to judge whether the driver's state is fatigued. According to the posture data in the historical posture data, determine the trend changes of the head posture Euler angles and the corresponding key posture parts. Take the head posture Euler angles as independent variables and the trend changes of the key posture parts as dependent variables to establish a linear regression model and explore the linear change relationship between them. At different posture angles, the same feature cannot be effectively compared. For example, the eye movement trajectory needs to determine the dynamic trajectories of the upper and lower parts of the eyes. If the head posture angle changes at this time, the state of the upper and lower parts of the eyes cannot be effectively measured. Here, the head posture is used to compensate it to improve the information accuracy.

[0066] According to the excavated linear variation relationship, point cloud correction is performed on the pose aiming frame in real-time monitoring. Taking the reference coordinate system in the standard state as a reference, the positions of key points (such as eyes, nose tip, mouth corners, etc.) within the aiming frame are precisely adjusted to match the changes in the head pose. According to the changes in the head pose, the pose aiming frame of the angular vector is corrected. The original key point cloud is obtained, the deviation between the current frame and the standard aiming frame is calculated, the key points are iteratively adjusted to align them with the corrected positions, the corrected aiming frame is updated, and the corrected pose aiming frame is determined. The aiming frame point cloud correction means that due to the change in the head angle, the positions of facial key points will shift, so it is necessary to use point cloud data (the three-dimensional coordinate set of key parts) for correction, so that the aiming frame can still accurately capture the target part at different angles.

[0067] Check the corrected pose aiming frames frame by frame. These aiming frames have been precisely adjusted through point cloud correction technology to reflect the actual pose of the driver. For each corrected pose aiming frame, analyze the trend change of the mapped aiming frame point cloud between adjacent frames, that is, observe the dynamic changes of key pose parts (such as eyes, face) in consecutive frames. For example, if the point cloud in the eye area moves continuously between multiple frames, it may indicate that the driver is making eye movements (such as blinking, saccading). If the facial expression changes from focused to relaxed, it may indicate that the driver is in a distracted state. The aiming frame trend change feature refers to the dynamic features extracted from the time series changes of the aiming frame point cloud, such as the displacement trend of the eye area, the expansion amplitude of the mouth area, etc., which are used to measure the dynamic changes of key pose parts, and then extract information such as facial expressions, eye movement trajectories, and blinking frequencies.

[0068] Based on the aiming frame trend change features, an aiming frame vector is constructed in the spatio-temporal dimension, which describes the dynamic features such as the position, speed, and acceleration of key pose parts, and measures the dynamic change state of key pose parts. According to the aiming frame vector, relevant features such as facial expressions, eye movement trajectories, and blinking frequencies are directly identified and extracted to form multi-modal biometric features. Multi-modal biometric features are multiple types of information extracted from the key areas of the driver's face, including facial expressions, eye movement trajectories, blinking frequencies, etc., which are used to comprehensively reflect the fatigue and state of the driver.

[0069] By setting dynamic and static posture thresholds based on the Euler angles of the head posture, first determine whether the angular vector meets the preset requirements; if it meets, directly trigger a fatigue alarm to quickly respond to the driver's fatigue risk; if it does not meet, transmit the continuous monitoring frames back. The second decision-maker determines the posture aiming frame corresponding to the key posture parts of the driver's fatigue and corrects it using the angular vector to ensure that the aiming frame is aligned with the state in the standard posture. Based on the corrected aiming frame, construct an aiming frame vector, and further extract multi-modal biometric features such as facial expressions, eye movement trajectories, and blink frequencies to provide comprehensive and accurate data support for fatigue detection. Finally, this process not only improves the accuracy and real-time performance of fatigue detection, but also effectively reduces the risks of misjudgment and missed alarms, providing a solid technical guarantee for driving safety.

[0070] Furthermore, as shown in the appendix Figure 2 This application also includes the following steps:

[0071] S310: Introduce a multi-level alarm mode and build a fatigue alarm, where the multi-level alarm mode is defined based on the alarm method and alarm intensity; S320: Deploy the fatigue alarm device and establish connections with the first decision-maker and the second decision-maker; S330: The fatigue alarm responds to the outputs of the first decision-maker and the second decision-maker to give an alarm reminder to the driver in a fatigued state.

[0072] Specifically, according to the different degrees of the driver's fatigue state, use different alarm methods and alarm intensities for hierarchical prompts. For example, mild fatigue may only give a low-intensity sound or vibration warning, while severe fatigue may trigger strong sound, visual, and vibration alarms, and even send them to the vehicle management center. Set different alarm levels according to the alarm method (sound, vibration, vision) and alarm intensity (low, medium, high). For example, low-level alarm: when the driver's fatigue probability is between 30% and 50%, only give a low-intensity vibration or soft sound reminder; medium-level alarm: when the fatigue probability is between 50% and 80%, the alarm outputs relatively strong sound and visual warnings; high-level alarm: when the fatigue probability exceeds 80%, trigger a high-intensity, multi-method (strong sound, frequent vibration, dashboard flashing) alarm reminder to ensure that the driver pays attention in time and takes measures.

[0073] Build a fatigue alarm, embed the alarm hardware (such as a buzzer, vibration motor, display module) into the vehicle control center, and establish connections with the first decision-maker and the second decision-maker through an internal bus or wireless communication. After the scanning components (cameras, sensors) collect data, the first decision-maker quickly judges the preliminary fatigue signal; at the same time, the second decision-maker comprehensively analyzes the multi-modal data and outputs a more refined fatigue state. The outputs of these two modules are transmitted to the fatigue alarm through a preset protocol to ensure that the alarm can automatically adjust the alarm level and alarm method according to the received data.

[0074] The fatigue alarm monitors the driver's state in real time based on the output data from the first and second decision-making units. When a fatigue state is detected reaching a predetermined threshold, the alarm responds according to a multi-level alarm mode. When the first decision-making unit detects that the angular vector does not meet the attitude threshold, it sends a signal to the fatigue alarm, and the fatigue alarm triggers the corresponding alarm method according to the signal strength and the preset alarm level. If the judgment of the first decision-making unit needs further confirmation, the second decision-making unit will conduct a more detailed analysis, such as correcting the attitude aiming frame and extracting multi-modal biometric features. The output of the second decision-making unit will determine whether to increase the alarm level and whether to trigger a stronger alarm.

[0075] Continuously monitoring and feeding back the driver's state, the alarm remains in real-time response throughout the driving process and further optimizes the alarm strategy and intensity setting according to the feedback data. Through continuous data collection and real-time processing, a closed-loop control is formed to ensure that fatigue detection and alarm are always accurate and timely. By introducing a multi-level alarm mode, building and deploying the fatigue alarm, and establishing an effective connection with the first and second decision-making units, the system can adjust the alarm method and intensity in real time according to the different degrees of the driver's fatigue state. It can not only output a low-intensity early warning when the fatigue risk is low to avoid excessive interference with the driver; but when the fatigue risk is high, it can quickly issue a strong alarm to ensure that the driver pays attention in time and takes necessary measures, improving the response speed of driver fatigue detection and the accuracy of fatigue alarm.

[0076] Furthermore, the present application further includes the following steps:

[0077] S340: Tracking and monitoring the driver's response state; S350: Adjusting the alarm method according to the response state, including increasing-direction continuous alarm in the non-response state, continuous alarm in the weak-response state, and alarm termination in the expected-response state.

[0078] Specifically, continuously tracking and monitoring the driver's response state, the driver's behavioral responses to external cues such as fatigue alarms, such as head movements, eye changes, or body adjustments. The response state can be divided into non-response, weak-response, and expected-response, which respectively indicate that the driver does not make an obvious reaction, only makes a weak reaction, and achieves an expected effective reaction. Through in-vehicle sensors, such as cameras, accelerometers, and heart rate sensors, the driver's state data is collected in real time, and the above-mentioned content is used to monitor the driver's reaction to the initial fatigue alarm. For example, after the fatigue alarm issues an initial alarm, it will detect whether the driver has obvious head movements, facial expression changes, or fluctuations in physiological signals.

[0079] Compare the collected response data with a preset response status threshold to determine the driver's response status. For example, if the driver does not make any obvious movements within a specified time (e.g., within 3 seconds), it is determined as non-responsive; if the driver only makes weak and slow reactions (such as only slight head movements or facial muscle changes), it is determined as a weak response; if the driver makes obvious and sufficient adjustments within a specified time (e.g., within 1 second), it is determined as an expected response.

[0080] Adjust the alarm mode according to the response status. When the driver is in a non-responsive state, start an increasing continuous alarm. For example, if the driver does not show any obvious reaction within 3 seconds after the initial alarm, the alarm will start with a primary low-intensity alarm and gradually increase the alarm intensity at regular intervals (e.g., every 1 second) until the driver responds. When the driver is in a weak response state, maintain a continuous alarm with a medium intensity to continuously remind the driver to pay attention to adjusting the driving state. When the driver is in an expected response state, automatically terminate the alarm to avoid repeated interference. Feed back the alarm status and the driver's response data to the vehicle control center and record the data for subsequent model optimization and adaptive adjustment to form a closed-loop control, ensuring that the alarm mode can dynamically adapt to changes in the driver's state and maximizing the warning effect.

[0081] By continuously tracking and monitoring the driver's response status and dynamically adjusting the alarm mode according to the driver's reaction to the initial alarm, it effectively improves the driver's alertness and the system's response accuracy, issues warnings in a timely manner when signs of fatigue are detected, and thus improves driving safety.

[0082] In summary, the real-time monitoring method for the driver's fatigue state based on multi-modal biometric fusion provided by this application has the following beneficial effects:

[0083] By deploying monitoring components, establishing channel interaction between the monitoring components and the vehicle control center, controlling the monitoring components to sample the driver's posture to determine continuous monitoring frames; embedding and deploying a first decision maker and a second decision maker, where the first decision maker is deployed in the scanning component and the second decision maker is deployed in the vehicle control center; connecting a fatigue alarm, and according to the continuous monitoring frames, coordinating the first-order fuzzy decision of the first decision maker and the fatigue state alarm, and the second-order decision of the second decision maker and the fatigue state alarm to remind the driver of the state; where the first decision maker makes a spatio-temporal dimension decision through the Euler angles of the head posture, and the second decision maker makes a spatio-temporal dimension decision based on the correction of the Euler angles of the head posture and the multi-modal biometric fusion determined based on the aiming frame vector. That is to say, by fusing multiple biometric features, sampling the driver's posture, comprehensively evaluating the driver's fatigue state, deploying the first decision maker and the second decision maker, and evaluating the fatigue state at different levels, false alarms and missed alarms are reduced, and the accuracy and efficiency of fatigue warning are improved.

[0084] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0085] Obviously, those skilled in the art can make several improvements and modifications to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A real-time monitoring method for driver fatigue status based on multi-modal biometric fusion, characterized in that: include: Deploy a monitoring component, establish a channel interaction between the monitoring component and the vehicle control center, control the monitoring component to perform driver posture sampling, and determine continuous monitoring frames; Embedded deployment of a first decision maker and a second decision maker, wherein the first decision maker is deployed in a scanning component and the second decision maker is deployed in a vehicle control center; Connecting to a fatigue alarm, and reminding the driver of the driver's status according to the continuous monitoring frame, in coordination with the first-order fuzzy decision and fatigue state alarm of the first decision maker, and the second-order decision and fatigue state alarm based on the second decision maker; The first decision maker makes a decision in the spatiotemporal dimension by using the Euler angle of the head posture, and the second decision maker makes a decision in the spatiotemporal dimension by fusing the correction based on the Euler angle of the head posture and the multimodal biometric feature determined based on the aiming frame vector; The head posture Euler angles include pitch angle, yaw angle and roll angle, which are determined with reference to the baseline three-dimensional coordinate system in the standard posture; For the continuous monitoring frames, measuring the angular vector in the spatiotemporal dimension based on the Euler angle of the head posture, wherein the angular vector is determined by the frame angle value and the angle trend characteristic based on adjacent frames; According to the angle vector, fatigue state decision and alarm are made; Wherein, fatigue state decision and alarm are performed according to the angle vector, including: Setting a posture threshold, wherein the posture threshold is a dynamic posture threshold and a static posture threshold set based on the Euler angle of the head posture; determining whether the angle vector satisfies the attitude threshold, and if so, controlling the fatigue alarm to execute an alarm based on the fatigue state; The second-order decision and fatigue status alarm based on the second decision maker include: If not satisfied, the continuous monitoring frame is transmitted back, and the attitude aiming frame is determined in combination with the second decision maker, wherein the attitude aiming frame is determined based on the fatigue key attitude part and corresponds to the continuous monitoring frame one by one; According to the angle vector, the posture aiming frame is corrected to determine a corrected posture aiming frame, wherein the correction requirement is a standard posture; Determining an aiming frame vector based on the corrected posture aiming frame; Extracting multimodal biometric features based on the aiming frame vector; Wherein, correcting the posture aiming frame includes: Calling historical posture data and extracting posture data, taking the head posture Euler angle as an independent variable and the trend of the key posture part as a dependent variable, mining the linear progressive relationship; According to the linear variation relationship, performing aiming frame point cloud correction based on the angle vector on the attitude aiming frame to determine the corrected attitude aiming frame; Wherein, determining the aiming frame vector includes: Traversing the correction posture aiming frame to determine the aiming frame trend change characteristics based on the mapping aiming frame point cloud trend change of adjacent frames; According to the aiming frame trend variation characteristics, the aiming frame vector in the time and space dimension is measured.

2. The method for real-time monitoring of driver fatigue status based on multi-modal biometric fusion according to claim 1, characterized in that: Fatigue status alarm, including: Introduce a multi-level alarm mode and build a fatigue alarm, where the multi-level alarm mode is limited based on the alarm mode and alarm intensity; deploying the fatigue alarm and establishing connection with the first decision maker and the second decision maker; The fatigue alarm responds to the outputs of the first decision maker and the second decision maker to warn the driver when he is in a fatigue state.

3. The method for real-time monitoring of driver fatigue status based on multi-modal biometric fusion according to claim 1, characterized in that: Deploy monitoring components, including: The monitoring component includes a visual monitoring component and a physiological monitoring component, wherein the visual monitoring is primary and the physiological monitoring is secondary; The continuous monitoring frames obtained by the visual monitoring component are combined with the physiological signals obtained by the physiological monitoring component to make fatigue status decisions and alarm reminders.

4. The method for real-time monitoring of driver fatigue status based on multi-modal biometric fusion according to claim 1, characterized in that: Before sampling the driver's posture, include: Mining the periodic sampling frequency based on fatigue trends according to driving time; The monitoring component is configured according to the periodic sampling frequency.

5. The method for real-time monitoring of driver fatigue status based on multi-modal biometric fusion according to claim 1, characterized in that: After the fatigue status alarm is issued, it includes: Track and monitor the driver's response status; According to the response state, the alarm mode is adjusted, including increasing continuous alarm in the no response state, continuous alarm in the weak response state and alarm termination in the expected response state.

Images