Method and device for detecting the fatigue of a driver of a vehicle

Abstract

A device (101) for detecting driver fatigue in a vehicle (100) is described. The device (101) is configured to acquire driver sensor data relating to the driver's current state and to obtain driver feedback regarding the driver's perception of their current state. The device (101) is further configured to determine, and in particular adapt, a detection algorithm for detecting driver fatigue based on the driver sensor data and the driver feedback.

Description

[0001] The invention relates to a method and a corresponding device designed to reliably detect whether the driver of a vehicle is tired or not.

[0002] A vehicle can be trained to collect sensor data related to the driver during operation in order to detect whether the driver is tired. If the vehicle determines, using a detection algorithm, that the driver is tired, a warning message can be issued to prompt the driver to take a break.

[0003] The (potentially repeated) erroneous display of warning messages can be perceived as inconvenient by a vehicle driver. This document addresses the technical challenge of detecting driver fatigue in a particularly reliable manner.

[0004] The problem is solved by each of the independent claims. Advantageous embodiments are described, inter alia, in the dependent claims. It should be noted that additional features of a claim dependent on an independent claim, without the features of the independent claim itself or only in combination with a subset of the features of the independent claim, can constitute a separate invention independent of the combination of all features of the independent claim, which can be made the subject of an independent claim, a divisional application, or a subsequent application. This applies equally to technical teachings described in the description, which can constitute an invention independent of the features of the independent claims.

[0005] According to one aspect, a device for detecting driver fatigue in a (motor) vehicle is described. The device is designed to acquire driver sensor data relating to the driver's current (fatigue) state. The driver sensor data can be acquired, for example, using an interior camera and / or a steering wheel sensor of the vehicle.

[0006] The device is further configured to obtain driver feedback regarding their (personal) perception of their current (fatigue) state. In this context, the device can be configured to issue a (possibly voice-based) status query to the driver via a vehicle user interface, asking how the driver perceives their current state. This status query can, for example, include a question about the driver's perceived degree of fatigue from a range of possible values ​​(e.g., from 2 to 20, particularly 5 to 10, different levels). The driver feedback can then be reliably recorded in response to the status query.

[0007] The device is further configured to determine, and in particular adapt, a detection algorithm for recognizing driver fatigue based on driver sensor data and driver feedback. The device may, in particular, be configured to adapt one or more parameters of the detection algorithm, especially a threshold for triggering the output of a warning message regarding detected driver fatigue, depending on the driver feedback.

[0008] The adapted detection algorithm can then be used (possibly based solely on driver sensor data) to detect driver fatigue in a particularly reliable and precise manner.

[0009] The driver feedback can include, in particular, voice-based feedback. Alternatively or additionally, the driver feedback can be determined based on voice-based feedback from the driver. The device can be configured to determine one or more indications of driver fatigue based on the voice-based feedback, in particular based on reaction time, speech rate, word choice, and / or intonation, and to take these into account within the driver fatigue detection algorithm.

[0010] The device can be configured, for example, to determine one or more signal characteristics of the speech signal from the speech-based feedback. Examples of signal characteristics include: a frequency, in particular the fundamental frequency, of the speech signal; the energy of the speech signal; and / or the frequency of zero crossings in the speech signal.

[0011] One or more properties of the speech-based feedback, in particular one or more (signal) properties of the speech signal of the speech-based feedback, can thus be used to detect driver fatigue with increased reliability. It can be checked whether the (semantic) content of the speech-based feedback matches the one or more properties of the speech-based feedback, in particular the one or more (signal) properties of the speech signal of the speech-based feedback.

[0012] For example, driver feedback can be determined based on the (semantic) content of the speech-based feedback (e.g., that the driver is not tired). Furthermore, based on one or more properties of the speech-based feedback, in particular one or more (signal) properties of the speech signal of the speech-based feedback, it can be determined whether the detection algorithm for detecting driver fatigue is determined, and in particular adapted, based on the driver feedback or not. For example, the driver feedback can be ignored when determining, and in particular adapting, the detection algorithm if the one or more properties of the speech-based feedback, in particular one or more (signal) properties of the speech signal of the speech-based feedback, do not match the (semantic) content of the speech-based feedback (e.g.,because one or more properties of the speech-based feedback indicate that the driver is tired. On the other hand, driver feedback can (possibly only) be taken into account in determining, and especially in adapting, the recognition algorithm if one or more properties of the speech-based feedback, in particular one or more (signal) properties of the speech signal of the speech-based feedback, match the (semantic) content of the speech-based feedback.

[0013] In other words, the device can be configured to determine driver feedback based on the semantic content of the speech signal from the driver. The semantic content can, for example, indicate an initial degree of driver fatigue.

[0014] The device can further be configured to determine one or more characteristics of the speech signal used for speech-based feedback. Based on these characteristics, it is possible, for example, to infer a second degree of driver fatigue. This second degree of fatigue may be consistent with the first degree of fatigue (e.g., if the first and second degrees differ by less than a threshold value) or inconsistent (e.g., if the first and second degrees differ by more than the threshold value).

[0015] The recognition algorithm can then be determined, and in particular adapted, based on one or more properties of the speech signal and driver feedback. Specifically, the device can be configured to ignore driver feedback when determining, and especially adapting, the recognition algorithm if one or more properties of the speech signal do not match the semantic content of the speech signal (especially if the second degree of fatigue does not match the first). Alternatively or additionally, the device can be configured to ignore driver feedback (if applicable).only then) to be taken into account in the determination, in particular in the adaptation, of the recognition algorithm if one or more properties of the speech signal of the speech-based feedback match the semantic content of the speech signal of the speech-based feedback (in particular if the second degree of fatigue matches the first degree).

[0016] This allows for a particularly reliable adaptation of the recognition algorithm.

[0017] The device can be configured to repeatedly output the status query during a vehicle journey. Furthermore, driver feedback in response to each output status query can be repeatedly recorded. The recognition algorithm can then be adapted with particular precision based on the repeatedly recorded driver feedback.

[0018] The device can be configured to determine, based on the detection algorithm (before any adaptation of the detection algorithm) and driver sensor data, whether to issue a warning regarding detected driver fatigue. For example, it can detect that the driver's level of fatigue has reached or exceeded a certain threshold. This can then trigger the display of a warning message. The user interface can subsequently be instructed to display the warning message.

[0019] The device can further be configured to prompt the user interface to issue a status query to the driver following the warning message. In particular, the device can be configured to detect user input via the user interface, thereby canceling the issued warning message (by the driver). This can be interpreted as an indication of an erroneous warning message output. The user interface can then (optionally selectively) be prompted to issue the status query to the driver in response to the detected user input. This allows for a particularly convenient and targeted driver-specific adaptation of the detection algorithm.

[0020] According to another aspect, a (road) motor vehicle (in particular a passenger car or a truck or a bus or a motorcycle) is described that includes the device described in this document.

[0021] According to one aspect, a method for detecting driver fatigue in a (motor) vehicle is described. The method includes acquiring driver sensor data regarding the driver's current state, as well as obtaining driver feedback regarding their perception of their current state. Furthermore, the method includes determining, and in particular adapting, a detection algorithm for recognizing driver fatigue based on the driver sensor data and the driver feedback.

[0022] It should be noted that the aspects described in connection with the device, in particular the claims described in connection with the device, are also applicable to the method as corresponding process features.

[0023] Another aspect described is a software (SW) program. The SW program can be configured to run on a processor (e.g., on a vehicle's control unit) and thereby execute the procedure described in this document.

[0024] Another aspect describes a storage medium. This storage medium can include a software program configured to run on a processor and thereby execute the procedure described in this document.

[0025] It should be noted that the methods, devices, and systems described in this document can be used both alone and in combination with other methods, devices, and systems described in this document. Furthermore, any aspect of the methods, devices, and systems described in this document can be combined with one another in a variety of ways. In particular, the features of the claims can be combined with one another in a variety of ways. Features listed in parentheses are to be understood as optional features.

[0026] The invention will now be described in more detail using exemplary embodiments. Fig. 1. Exemplary components of a vehicle; and Fig. 2. A flowchart of an exemplary procedure for determining an algorithm for detecting the fatigue of a driver of a vehicle.

[0027] As stated at the outset, this document deals with the reliable detection of driver fatigue in a vehicle. In this context, it shows Fig. 1 An exemplary vehicle 100 with one or more environmental sensors 102, each configured to acquire sensor data (also referred to as environmental data) relating to the environment of the vehicle 100. Examples of environmental sensors 102 are a camera, a radar sensor, a lidar sensor, an ultrasonic sensor, etc.

[0028] A (control) device 101 of the vehicle 100 can be configured to evaluate environmental data, e.g., to detect one or more objects (e.g., other vehicles) in the vicinity of the vehicle 100. The device 101 can further be configured to effect automated longitudinal and / or lateral control of the vehicle 100 based on the environmental data, in particular based on the one or more detected objects. For this purpose, one or more longitudinal and / or lateral control actuators 103 (e.g., a drive motor, a braking device, and / or a steering device) of the vehicle can be controlled.

[0029] The vehicle 100 can include one or more driver sensors 105, 106, each configured to acquire driver sensor data relating to the driver, in particular regarding driver fatigue. Examples of driver sensors 105, 106 are a camera 106 configured to acquire camera data relating to the driver, and / or a steering wheel sensor 105 configured to acquire steering data relating to the driver's steering behavior.

[0030] The (control) device 101 of the vehicle 100 can be configured to determine, based on driver sensor data from one or more driver sensors 105, 106, whether the driver is tired or not. In particular, the degree of driver fatigue can be determined based on the driver sensor data from one or more driver sensors 105, 106 and using a detection algorithm, and optionally compared with a degree threshold value to determine whether the driver is tired or not. If it is determined that the driver is tired, a warning message can be issued to the driver of the vehicle 100 via the user interface 104 of the vehicle 100, e.g., to prompt the driver to take a break.

[0031] The erroneous issuance of warning messages can be perceived as inconvenient by the driver of vehicle 100. In particular, it may happen that the detection algorithm determines that the driver is tired, even though this does not correspond to the driver's perception.

[0032] The device 101 can be configured to prompt the driver via the user interface 104 as to whether the driver currently feels tired or not. If necessary, the driver can also be asked to enter their current level of tiredness (e.g., on a scale between 0% and 100% or between 1 and 10).

[0033] The driver can then provide feedback regarding their current perceived fatigue via the user interface 104. Furthermore, the device 101 can be configured to adjust the detection algorithm based on this feedback. For example, the degree threshold can be adjusted based on the driver's feedback. Alternatively or additionally, the method used by the detection algorithm to determine the driver's degree of fatigue can be adjusted.

[0034] Based on one or more driver feedback signals, the fatigue detection system can be adapted to the individual driver. This prevents erroneous warnings from being issued, thus increasing vehicle comfort without compromising driving safety.

[0035] Driver self-annotation is thus incorporated to increase the reliability of the individual fatigue calculation. This can be achieved through a multi-stage process. In a relatively early stage of fatigue (e.g., when fatigue is relatively low), one or more questions about the driver's actual perception of fatigue can be posed via the user interface 104. Based on the driver's feedback, one or more fatigue thresholds can be defined and / or adjusted more precisely. This allows for more accurate and timely warnings before the driver experiences signs of fatigue and / or sleepiness. By asking about the driver's actual perceived fatigue, their speech can be used as an additional channel for fatigue detection (in the case of voice-based feedback).

[0036] For example, based on camera data from an interior camera 106, the driver's eyelid opening, head pose, and / or movement can be evaluated. Based on this camera data, a detection algorithm (e.g., using PERCLOS, eyelid closure rate, etc.) can calculate the driver's fatigue (e.g., the degree of fatigue).

[0037] Furthermore, a question can be posed to the driver (e.g., "On a scale of 1-10: How fit do you feel? / How tired do you feel today?"). This question can be repeated. Alternatively or additionally, the question can be posed when a certain fatigue threshold is reached. The driver can then provide feedback (possibly voice-based) in response to each question. This feedback can, for example, indicate the actual perceived level of fatigue. The voice-based feedback can be analyzed for fatigue detection (e.g., by evaluating reaction time and / or tone of voice, word choice, signal characteristics, etc.). The content of the feedback can also be considered. Based on this, the detection algorithm, particularly the fatigue threshold, can be adjusted.

[0038] If necessary, a query may be issued to the driver after a warning regarding the detected driver fatigue has been issued, and after the driver has canceled (e.g., dismissed) the warning via user interface 104.

[0039] Fig. Figure 2 shows an exemplary (possibly computer-implemented) method 200 for detecting the fatigue of the driver of a (motor) vehicle 100. The method 200 can be carried out by a (control) device 101 of the vehicle 100.

[0040] Procedure 200 comprises the acquisition 201 of driver sensor data (e.g., camera data and / or steering data) relating to the driver's current (fatigue) state. Furthermore, procedure 200 comprises the determination 202 of driver feedback regarding the driver's perception of their current (fatigue) state.

[0041] The procedure 200 further includes determining 304, in particular adapting, the detection algorithm for detecting driver fatigue based on driver sensor data and driver feedback.

[0042] The measures described in this document can efficiently increase the reliability of driver fatigue detection in a vehicle.

[0043] The present invention is not limited to the embodiments shown. In particular, it should be noted that the description and the figures are intended only to illustrate the principle of the proposed methods, devices, and systems by way of example.

Claims

[1] Device (101) for detecting the fatigue of the driver of a vehicle (100); wherein the device (101) is configured, - To collect driver sensor data relating to the driver's current state; - to obtain driver feedback regarding the driver's perception of their current condition; and - to determine and, in particular, adapt a detection algorithm for detecting driver fatigue based on driver sensor data and driver feedback. [2] Device (101) according to claim 1, wherein the device (101) is configured to adapt one or more parameters of the detection algorithm, in particular a threshold for triggering the output of a warning message in respect of detected driver fatigue, depending on the driver feedback. [3] Device (101) according to one of the preceding claims, wherein - the driver feedback includes voice-based feedback; and - the device (101) is configured to identify one or more indications of driver fatigue based on speech-based feedback, in particular based on reaction speed, speech speed, word choice, signal properties of a speech signal and / or intonation, and to take these into account within the framework of the detection algorithm for detecting driver fatigue. [4] Device (101) according to one of the preceding claims, wherein the device (101) is configured, - to determine the driver feedback based on the semantic content of a speech signal or a speech-based feedback from the driver; - to determine one or more properties of the speech signal of the speech-based feedback; and - to determine, and in particular adapt, the recognition algorithm depending on one or more properties of the speech signal based on driver feedback. [5] Device (101) according to claim 4, wherein the device (101) is configured, - to ignore driver feedback during the determination, particularly during the adaptation, of the recognition algorithm if one or more properties of the speech signal of the speech-based feedback do not match the semantic content of the speech signal of the speech-based feedback; and / or - to take into account driver feedback when determining, especially when adapting, the recognition algorithm, if one or more properties of the speech signal of the speech-based feedback match the semantic content of the speech signal of the speech-based feedback. [6] Device (101) according to one of the preceding claims, wherein the device (101) is configured, - to issue a status query to the driver via a user interface (104) of the vehicle (100) regarding how the driver perceives his current state; and - to capture driver feedback in response to the issued status request. [7] Device (101) according to claim 6, wherein the device (101) is configured, - to repeatedly output the status query during a journey of the vehicle (100); - repeatedly record driver feedback in response to each status request; and - to adapt the recognition algorithm based on repeatedly recorded driver feedback. [8] Device (101) according to one of claims 6 to 7, wherein the device (101) is set up, - to determine, based on the detection algorithm and the driver sensor data, that a warning should be issued in relation to detected driver fatigue, in particular because a degree of driver fatigue has reached or exceeded a certain degree threshold; - to cause the user interface (104) to display the warning message; and - following the issuance of the warning message, to cause the user interface (104) to issue a status query to the driver. [9] Device (101) according to claim 8, wherein the device (101) is configured, - to capture user input via the user interface (104) that cancels the issued warning message; and - in response to the captured user input, to cause the user interface (104) to output the status query to the driver. [10] Device (101) according to one of claims 6 to 9, wherein the state query includes a query for a perceived degree of fatigue of the driver from a plurality of possible degree values. [11] Device (101) according to one of the preceding claims, wherein the driver sensor data are acquired using an interior camera (106) and / or a steering wheel sensor (105). [12] Method (200) for detecting the fatigue of the driver of a vehicle (100); wherein the method (200) comprises, - Acquisition (201) of driver sensor data relating to a current state of the driver; - Determining (202) driver feedback regarding a perception of his current condition; and - Determine (304), in particular adapt, a detection algorithm for detecting driver fatigue based on driver sensor data and driver feedback.

Images