Driver distraction monitoring method

Abstract

The application discloses a driver distraction monitoring method, aiming at solving the problems of false alarm or missed alarm of the existing driver monitoring system caused by individual differences of drivers, environmental changes, and single sensor failure. The application obtains the face image collected by the vehicle-mounted DMS camera and the eye movement data collected by the AR glasses worn by the driver in parallel, respectively calculates the first state parameter and its confidence, the second state parameter and its confidence. Based on the confidence evaluation result, the fusion strategy is dynamically selected: when both channels have high confidence, the or logical judgment is adopted; when the judgment results conflict, the high confidence channel result is used as the criterion; when the single channel confidence is lower than the threshold, the channel judgment is suspended and the other channel is completely relied on. In addition, the change of the pupil diameter based on the AR glasses can distinguish visual distraction and cognitive distraction. The application realizes intelligent collaboration of multi-source heterogeneous sensors, and significantly improves the accuracy and robustness of distraction monitoring in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of active safety technology for automobiles, and in particular to a method for monitoring driver distraction. Background Technology

[0002] Driver monitoring systems (DMS) are a crucial component of advanced driver assistance systems (ADAS), preventing traffic accidents caused by distraction and fatigue by monitoring driver status in real time. Current DMS systems primarily rely on in-vehicle cameras to capture facial images of the driver, using computer vision algorithms to identify key facial features and analyze head posture, eyelid opening, and other characteristics to determine driver distraction. However, this technology has inherent limitations: when the driver is wearing sunglasses, has small eyes, single eyelids, or is in backlit or low-light environments, eye features are difficult to extract accurately, leading to inaccurate eyelid opening calculations; changes in driver posture and facial obstruction can also cause errors in head posture estimation, resulting in functional degradation or false alarms.

[0003] In recent years, methods have emerged that use eye-tracking technology to detect driver attention. For example, a single camera can be used to capture images of the driver's face and the external environment, a three-dimensional eye model can be built to calculate the gaze point, and the PNP algorithm can be used to fuse the gaze into the external environment to analyze the distribution of attention.

[0004] However, this existing solution still relies on a single sensor source, and it faces the risk of data loss when the camera is obstructed, lighting conditions change abruptly, or the driver wears equipment that affects eye imaging. Furthermore, this existing solution fails to address the issues of multi-source data fusion and robustness under sensor failure, and it also struggles to distinguish between different types of distraction. Therefore, achieving highly reliable and accurate driver distraction monitoring in complex and ever-changing driving environments remains a pressing technical challenge in this field. Summary of the Invention

[0005] In view of the above, the present invention aims to provide a driver distraction monitoring method that integrates vehicle-mounted DMS and AR glasses. By constructing an intelligent collaborative mechanism of multi-source heterogeneous sensors, the accuracy and robustness of distraction monitoring are significantly improved, and the refined identification of distraction types is achieved.

[0006] The technical solution adopted in this invention is as follows:

[0007] This invention provides a method for monitoring driver distraction, comprising:

[0008] The driver's facial image is captured by the camera of the vehicle driver monitoring system, and facial key points are identified to obtain the first state parameters, which include the head posture angle and eyelid opening degree.

[0009] Eye images are captured by an eye-tracking camera integrated into the augmented reality glasses worn by the driver, and eye-tracking analysis is performed to obtain a second state parameter, which includes the three-dimensional gaze direction and pupil diameter.

[0010] Evaluate the first confidence level of the first state parameter and the second confidence level of the second state parameter respectively;

[0011] The first state parameter and the second state parameter are dynamically fused to collaboratively determine the driver's distraction state and obtain the final distraction determination result.

[0012] If the final distraction determination result is characterized as distraction, then the distraction type is determined and the corresponding distraction alarm signal is output, wherein the distraction type includes at least visual distraction or cognitive distraction.

[0013] In at least one possible implementation, the evaluation of the first confidence level of the first state parameter and the second confidence level of the second state parameter specifically includes:

[0014] The first confidence level is evaluated based on the completeness of facial key point detection and the stability of head pose tracking;

[0015] The second confidence level is evaluated based on the success rate of pupil capture and the continuity of gaze tracking.

[0016] In at least one possible implementation, the coordinated determination of the driver's distraction state includes:

[0017] When the first confidence level is higher than the first preset high threshold and the second confidence level is higher than the second preset high threshold, a first distraction determination is obtained based on the first state parameter, and a second distraction determination is obtained based on the second state parameter. If the results of the first distraction determination or the second distraction determination are consistent, the final distraction determination result is determined by OR logic.

[0018] In at least one possible implementation, the collaborative determination of the driver's distraction state further includes:

[0019] When the results of the first distraction determination and the second distraction determination are inconsistent, the first confidence level and the second confidence level are compared, and the distraction determination result with the higher confidence level is taken as the final distraction determination result.

[0020] In at least one possible implementation, the coordinated determination of the driver's distraction state includes:

[0021] When the first confidence level is lower than the first preset low threshold, the distraction determination based on the first state parameter is paused, and the distraction determination is performed only based on the second state parameter, and the second distraction determination result is used as the final distraction determination result.

[0022] When the second confidence level is lower than the second preset low threshold, the distraction determination based on the second state parameter is paused, and the distraction determination is performed only based on the first state parameter, and the first distraction determination result is used as the final distraction determination result.

[0023] In at least one possible implementation, determining the distraction type specifically includes:

[0024] Based on the deviation of the three-dimensional gaze direction from the preset driving attention area, and the change pattern of the pupil diameter, the driver's distraction type is determined:

[0025] If the three-dimensional line of sight continuously deviates from the driver's attention area and the pupil diameter does not show obvious periodic changes, it is determined to be visual distraction;

[0026] If the three-dimensional line of sight is located within the driver's attention area and the pupil diameter exhibits periodic contraction and expansion, it is determined to be cognitive distraction.

[0027] In at least one of the possible implementations:

[0028] The process of determining distraction based on the first state parameter includes: calculating the duration of head deviation from the front based on the head posture angle, and calculating the PERCLOS value based on the eyelid opening and closing degree; if the duration exceeds a first duration threshold or the PERCLOS value exceeds a first percentage threshold, the result of the first distraction determination is distraction.

[0029] The process of determining distraction based on the second state parameter includes: comparing the three-dimensional line of sight with a preset driving attention area, counting the duration of continuous deviation of the line of sight from the area, and if the duration exceeds the second duration threshold, the result of the first distraction determination is distraction.

[0030] Compared to existing technologies, the main design concept of this invention lies in the deep fusion of macroscopic facial features captured by an in-vehicle DMS camera and microscopic eye-movement features captured by AR glasses. This not only utilizes two independent data sources to create redundant monitoring but also establishes a dynamic fusion and arbitration mechanism based on data confidence. This mechanism can evaluate the reliability of the data from both channels in real time and adaptively adjust the fusion strategy according to the confidence level: when both channels are reliable, an "OR" logic is used to ensure high sensitivity; when a conflict is detected, the channel with higher confidence is used to avoid false alarms; and when a single channel fails, it seamlessly switches to the other channel to ensure monitoring continuity. Furthermore, this invention utilizes the high-precision pupil diameter changes provided by AR glasses, combined with the direction of gaze, to distinguish between visual distraction and cognitive distraction, providing a deeper understanding of the driver's state.

[0031] The overall solution of this invention has at least the following outstanding effects:

[0032] Compared to existing solutions that use only a single camera or binocular cameras for eye tracking, this invention introduces physically separate AR glasses as an independent eye tracking device, forming heterogeneous redundancy with the vehicle-mounted DMS, fundamentally solving the problem of single sensor failure.

[0033] The dual-channel confidence assessment and dynamic fusion arbitration logic proposed in the specific embodiment can intelligently cope with various complex scenarios. For example, it can automatically rely on AR glasses when strong backlight causes DMS to fail, and automatically rely on DMS when AR glasses lose pupils due to severe shaking. When the two judgments are contradictory, the higher confidence result shall prevail, thereby greatly improving the robustness and accuracy of the overall system.

[0034] Existing methods that only count the distribution of fixation points cannot distinguish between visual distraction and cognitive distraction. This invention utilizes pupil diameter changes provided by AR glasses as a physiological indicator of cognitive load, combined with gaze deviation, to achieve a preliminary distinction between the two types of distraction in the field of driver distraction monitoring, providing richer information for personalized alarms and human-computer interaction. Attached Figure Description

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0036] Figure 1 This is a flowchart illustrating the driver distraction monitoring method provided in an embodiment of the present invention. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0038] This invention proposes an embodiment of a driver distraction monitoring method, specifically, as follows: Figure 1 As shown, it includes:

[0039] Step S1: Collect the driver's facial image using the driver monitoring system and collect the driver's eye movement image using AR glasses;

[0040] Specifically, the DMS function is activated after the vehicle is started. The vehicle-mounted near-infrared camera continuously captures images of the driver's face at a frame rate of 30fps. Simultaneously, it can establish a communication connection with the AR glasses worn by the driver via Bluetooth or Wi-Fi. The eye-tracking camera built into the AR glasses synchronously captures high-resolution images of both eyes. Those skilled in the art will understand that, in the context of this invention, the AR glasses worn will not affect the driver's observation of the real-world scene or their field of vision, and therefore will not interfere with normal driving operations. Next, a preliminary quality assessment is performed on the DMS images and AR eye-tracking images respectively. The assessment indicators include image brightness, contrast, sharpness, and the presence of occlusion. In actual operation, for example, if the DMS image quality is lower than a preset threshold, such as due to strong backlight causing the face to be too dark, the initial confidence value of the first state parameter is marked as low; if the AR eye-tracking image quality is lower than a preset threshold, such as due to the pupils being blurred because the glasses are slipping, the initial confidence value of the second state parameter is marked as low.

[0041] Step S2: Extract facial key points from the facial image and calculate the corresponding first state parameters, and extract pupil features from the eye movement image and calculate the corresponding second state parameters;

[0042] On the one hand, for DMS image frames, 106 facial key points can be extracted using a deep learning-based facial key point detection model, and the first state parameters can be calculated based on these key points:

[0043] (1) Head attitude angles: The pitch angle, yaw angle and roll angle of the head are calculated by using the PnP algorithm through the 2D-3D correspondence; among them, the yaw angle is used to determine whether the head has deviated from the front for a long time.

[0044] (2) Eyelid opening and closing: The aspect ratio of the eyes is calculated based on the key points of the upper and lower eyelids, and then the PERCLOS value is calculated. In other words, it is the percentage of frames in which the eye closure degree exceeds 80% per unit time.

[0045] On the other hand, for the AR eye-tracking channel, tracking technology based on pupil-corneal reflection can be used, but is not limited to, to reconstruct the three-dimensional gaze direction and pupil diameter of both eyes in real time as a second state parameter. It should be noted that the three-dimensional gaze direction is represented in the AR glasses' own coordinate system, and subsequent processing can transform it to the vehicle coordinate system, for example, through a pre-calibrated transformation matrix between the AR glasses and the vehicle coordinate system.

[0046] Step S3: Evaluate the first confidence level of the first state parameter and the second confidence level of the second state parameter respectively;

[0047] Here are some implementation methods for reference:

[0048] For the first confidence level C1, the value range is [0,1]. It can be calculated based on the following factors: the completeness of facial key point detection, that is, the number of key points successfully detected / the total number of points, and the stability of head pose tracking, such as the smoothness of pose angle changes between adjacent frames. It can also be combined with the image quality evaluation results in the aforementioned embodiments.

[0049] For the second confidence level C2, the value range is [0,1]. It can be calculated based on the following factors: the success rate of pupil capture, that is, the proportion of time that the pupil center can be stably extracted, and the continuity of gaze tracking, such as whether frames are frequently dropped. Similarly, it can also be combined with the image quality evaluation results in the aforementioned embodiments.

[0050] Step S4: Dynamically fuse two different confidence levels to collaboratively determine whether the driver is distracted;

[0051] Continuing from the previous example, different fusion strategies can be dynamically selected based on the values ​​of C1 and C2 to obtain the final result of whether or not the user is distracted. The following implementation methods for different scenarios are provided for reference:

[0052] Strategy 1: Dual-channel high-confidence mode.

[0053] If C1 > Th1_high (e.g., a preset signal height value of 0.8) and C2 > Th2_high (e.g., a preset signal height value of 0.8), then distraction determination is performed simultaneously based on the first state parameter and the second state parameter. If the conclusions of the two parameters corresponding to the aforementioned DMS and AR glasses are consistent, then one of them is selected as the final determination result using "OR" logic.

[0054] Here, the results of the two determinations are recorded as R1 and R2. The rules for determining whether R1 and R2 are distractions can be referred to as follows (the same applies below):

[0055] R1 represents the condition of distraction: it can be that when and only when the duration that the absolute value of the head yaw angle in the first state parameter exceeds the preset degree of 15° exceeds the predetermined duration of 2 seconds, or the PERCLOS value in the first state parameter exceeds the preset standard value of 0.15 within the predetermined duration of 30 seconds.

[0056] R2 represents the condition of distraction: it can be that when and only when the duration that the three-dimensional line-of-sight direction in the second state parameter continuously deviates from the preset driving attention area (including the conical space lights of the front windshield, instrument panel, and left and right rearview mirrors) exceeds the predetermined duration of 2 seconds.

[0057] The final determination result R = R1 or R2, and this mode realizes redundant monitoring with maximum sensitivity.

[0058] Strategy 2: Decision conflict arbitration mode.

[0059] If C1 > Th1_high and C2 > Th2_high, but the results of R1 and R2 are inconsistent (for example, R1 is distracted and R2 is not distracted), then enter arbitration: compare C1 and C2. If C1 > C2, then the final determination is R = R1; otherwise, R = R2. This mode ensures that in case of contradictions, the more reliable data source is used as the standard, effectively suppressing false alarms.

[0060] Strategy 3: Single-channel failure mode.

[0061] If C1 < Th1_low (such as the preset low confidence value of 0.3), it is considered that the DMS channel fails temporarily, and the determination based on the first state parameter is suspended. At this time, the determination is completely dependent on the second state parameter, that is, R = R2, and at the same time, the driver can be prompted that the DMS function is limited. Understandably, vice versa, if C2 < Th2_low, then it is completely dependent on the first state parameter, that is, R = R1. This mode ensures that the monitoring function does not interrupt when a single sensor fails.

[0062] Strategy 4: Weighted fusion mode.

[0063] For the situation where both C1 and C2 are between the intermediate thresholds (the confidence levels are both between 0.3 and 0.8), a weighted fusion method can be used to obtain a continuous distraction probability value, and then compare it with the threshold. For example, the distraction probability = (C1 * R1_score + C2 * R2_score) / (C1 + C2), where R1_score and R2_score are distraction quantization weight values between 0 and 1, and can be pre-calibrated based on experience in actual operation.

[0064] Continuing from the previous text, in step S5, if the distraction determination result indicates distraction, then determine the distraction type and output the corresponding distraction alarm signal.

[0065] In practice, determining the type of distraction primarily relies on data from the AR glasses, specifically using pupil diameter changes provided by the AR glasses to refine the distraction type. This is because research indicates that pupil diameter dilates with increased cognitive load, and under certain circumstances, pupil size exhibits periodic fluctuations. Therefore:

[0066] If R is distraction, the direction of the gaze continuously deviates from the driver's focus area, and the pupil diameter does not show obvious periodic changes (standard deviation is less than the set threshold), then it is judged as "visual distraction".

[0067] If R is characterized as not being distracted (the driver's gaze is within the driving area) due to the adoption of the result of R2, but the pupil diameter in the second state parameter shows a periodic contraction and expansion state (period such as 0.5~2Hz), then R will be determined as distracted and the specific type will be "cognitive distraction". This means that although the gaze is not deviated, the driver's mental state may be wandering, which can still trigger the distraction alarm.

[0068] Alarm prompts can be output through the vehicle's sound system, visual system, or the display function of the AR glasses themselves. This invention does not limit or elaborate on this, but it can be added that different reminder modes can be triggered for different types of distraction. For example, visual distraction includes a voice prompt "Please look at the road", while cognitive distraction includes a prompt voice "Please concentrate" or a gentler prompt measure than buzzers, flashing lights, etc.

[0069] In summary, by integrating in-vehicle DMS and AR glasses and introducing a confidence-based dynamic collaborative mechanism, we have achieved highly robust and accurate driver distraction monitoring, and for the first time in the industry, we have achieved preliminary differentiation of distraction types.

[0070] In this invention, when directional terms are mentioned, they are relative concepts based on the embodiments. Furthermore, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0071] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for monitoring driver distraction, characterized in that, include: The driver's facial image is captured by the camera of the vehicle driver monitoring system, and facial key points are identified to obtain the first state parameters, which include the head posture angle and eyelid opening degree. Eye images are captured by an eye-tracking camera integrated into the augmented reality glasses worn by the driver, and eye-tracking analysis is performed to obtain a second state parameter, which includes the three-dimensional gaze direction and pupil diameter. Evaluate the first confidence level of the first state parameter and the second confidence level of the second state parameter respectively; The first state parameter and the second state parameter are dynamically fused to collaboratively determine the driver's distraction state and obtain the final distraction determination result. If the final distraction determination result is characterized as distraction, then the distraction type is determined and the corresponding distraction alarm signal is output, wherein the distraction type includes at least visual distraction or cognitive distraction.

2. The driver distraction monitoring method according to claim 1, characterized in that, The specific steps of evaluating the first confidence level of the first state parameter and the second confidence level of the second state parameter include: The first confidence level is evaluated based on the completeness of facial key point detection and the stability of head pose tracking; The second confidence level is evaluated based on the success rate of pupil capture and the continuity of gaze tracking.

3. The driver distraction monitoring method according to claim 1, characterized in that, The coordinated determination of the driver's distracted state includes: When the first confidence level is higher than the first preset high threshold and the second confidence level is higher than the second preset high threshold, a first distraction determination is obtained based on the first state parameter, and a second distraction determination is obtained based on the second state parameter. If the results of the first distraction determination or the second distraction determination are consistent, the final distraction determination result is determined by OR logic.

4. The driver distraction monitoring method according to claim 3, characterized in that, The collaborative determination of the driver's distracted state also includes: When the results of the first distraction determination and the second distraction determination are inconsistent, the first confidence level and the second confidence level are compared, and the distraction determination result with the higher confidence level is taken as the final distraction determination result.

5. The driver distraction monitoring method according to claim 1, characterized in that, The coordinated determination of the driver's distracted state includes: When the first confidence level is lower than the first preset low threshold, the distraction determination based on the first state parameter is paused, and the distraction determination is performed only based on the second state parameter, and the second distraction determination result is used as the final distraction determination result. When the second confidence level is lower than the second preset low threshold, the distraction determination based on the second state parameter is paused, and the distraction determination is performed only based on the first state parameter, and the first distraction determination result is used as the final distraction determination result.

6. The driver distraction monitoring method according to claim 1, characterized in that, The determination of the distraction type specifically includes: Based on the deviation of the three-dimensional gaze direction from the preset driving attention area, and the change pattern of the pupil diameter, the driver's distraction type is determined: If the three-dimensional line of sight continuously deviates from the driver's attention area and the pupil diameter does not show obvious periodic changes, it is determined to be visual distraction; If the three-dimensional line of sight is located within the driver's attention area and the pupil diameter exhibits periodic contraction and expansion, it is determined to be cognitive distraction.

7. The driver distraction monitoring method according to any one of claims 1 to 6, characterized in that: The process of determining distraction based on the first state parameter includes: calculating the duration of head deviation from the front based on the head posture angle, and calculating the PERCLOS value based on the eyelid opening and closing degree; if the duration exceeds a first duration threshold or the PERCLOS value exceeds a first percentage threshold, the result of the first distraction determination is distraction. The process of determining distraction based on the second state parameter includes: comparing the three-dimensional line of sight with a preset driving attention area, counting the duration of continuous deviation of the line of sight from the area, and if the duration exceeds the second duration threshold, the result of the first distraction determination is distraction.

Images