Method for training an individual fatigue detection model for a driver of a motor vehicle, fatigue detection method and electronic computing device

Abstract

The invention relates to a method for training an individual fatigue detection model for a driver of a motor vehicle, in which the driver is identified (VA1), driver fatigue is investigated using a global fatigue detection model based on input data which serve as indicators of driver fatigue (VA2), a fatigue event is detected when driver fatigue is detected, and at least part of the input data, which were determined for a defined period before the onset of fatigue of the fatigue event, for which the onset of driver fatigue was detected, are used as training data for the driver-specific fatigue detection model, which is trained to recognize individual early signs of fatigue of the identified driver (VA3).Furthermore, the invention relates to a method for detecting the fatigue of a driver of a motor vehicle by means of an individual fatigue detection model for the driver.

Description

[0001] The invention relates to a method for training an individual fatigue detection model for a driver of a motor vehicle, a method for fatigue detection of a driver of a motor vehicle and an electronic computing device.

[0002] Especially on long car journeys, at night, or when not in optimal physical condition, many drivers suffer from fatigue, which can have dangerous consequences. Methods based on indicators such as steering behavior, driving behavior, lane changes caused by steering movements, and data from a camera aimed at the driver can be used to detect fatigue in motor vehicles. However, fatigue manifests itself differently in each person, and early warning signals, in particular, can vary greatly from person to person.

[0003] The object of the present invention is therefore to create a solution by means of which a driver's fatigue can be determined particularly reliably.

[0004] This object is achieved according to the invention by the subject matter of the independent claims. Further possible embodiments of the invention are disclosed in the subclaims, the description, and the figures. Features, advantages, and possible embodiments presented in the description for one of the subject matter of the independent claims are to be regarded at least analogously as features, advantages, and possible embodiments of the respective subject matter of the other independent claims, as well as any possible combination of the subject matter of the independent claims, optionally in conjunction with one or more of the subclaims.

[0005] The invention relates to a method for training an individual fatigue detection model for a driver of a motor vehicle, in particular a motor vehicle, in particular a passenger car. In the method, the driver of the motor vehicle is identified. The driver can be identified, for example, using images depicting the driver recorded by a camera device. These images can be analyzed, in particular, using an image evaluation device. The method further provides for the driver's fatigue to be examined using a global fatigue detection model based on input data that serve as indicators of the driver's fatigue. This global fatigue detection model is, in particular, not specifically trained for the driver, but can be used for several different drivers.Input data can include data characterizing steering behavior and / or driving behavior, in particular data characterizing steering and / or braking behavior, and / or data characterizing a completed lane change and / or data from the camera device with its field of view directed toward the driver. The global fatigue detection model can be trained, for example, using machine learning. This global fatigue detection model can then be used to determine, based on the input data provided to the global fatigue detection model, whether the driver is fatigued and thus a fatigue event has occurred, or whether the driver is not yet fatigued and thus no fatigue event has occurred.If the global fatigue detection model detects the driver's fatigue based on the input data, thus determining the fatigue event, then at least a portion of the input data provided to the fatigue detection model is used as training data for the driver-specific fatigue detection model. This portion of the input data relates to a defined period before the onset of the fatigue event for which the onset of the driver's fatigue was determined. The onset of the fatigue event is understood to mean that the driver's degree of fatigue exceeds a specified fatigue threshold.This means that the fatigue event detected by the global fatigue detection model is assigned a fatigue onset time, and the input data characterizing the driver at a time prior to this fatigue onset time is used to train the driver-specific fatigue detection model. For example, the time period extends from the fatigue onset time into the past. Alternatively, the time period can be a time lag from the fatigue onset time.

[0006] For example, input data from the last x to y minutes before the onset of fatigue is used as training data, where y is greater than zero. For example, x can stand for five minutes and y for 20 minutes. In particular, the time period refers to the same journey of the motor vehicle for which the driver fatigue event was detected. Thus, based on the input data characterizing the driver's behavior or appearance before the onset of the fatigue event, early warning signs of fatigue in the driver can be identified. The driver-specific fatigue detection model is trained using the training data to detect individual early warning signs of fatigue of the identified driver. In other words, the global fatigue detection model is used to reliably detect a driver fatigue event.Once this driver fatigue event has been detected, the input data is used to look back into the past, starting from the time of occurrence of this fatigue event. This input data from the past with respect to the time of occurrence of the fatigue event is used to train the driver-specific fatigue detection model to reliably detect early signs of fatigue in the driver for which the driver-specific fatigue detection model is designed. By training the driver-specific fatigue detection model, it can be achieved that the driver-specific fatigue detection model is capable of reliably predicting a near-future driver fatigue event based on early signs of fatigue in the driver.As a result, this highly likely impending driver fatigue event can be counteracted particularly well.

[0007] In a possible further development of the invention, it is provided that a quality measure for the trained driver-specific fatigue detection model is determined with regard to early fatigue detection. Input data from another fatigue event is used as training data for training the driver-specific fatigue detection model if the determined quality measure falls below a predetermined threshold. This means that additional input data is used as additional training data for training the driver-specific fatigue detection model if, during analysis of the already trained driver-specific fatigue detection model, it is determined that the predetermined quality measure for the driver-specific fatigue detection model is below the predetermined threshold and thus the quality of the already trained driver-specific fatigue detection model is not yet sufficient.By further training the driver-specific fatigue detection model with additional input data from another fatigue event detected by the global fatigue detection model as training data, the quality of the trained driver-specific fatigue detection model can be improved by continuously training the driver-specific fatigue detection model. This allows the probability of incorrectly detecting early signs of fatigue in the driver or of failing to detect early signs of fatigue to be kept particularly low.

[0008] In this context, it can be provided, in particular, that the trained driver-specific fatigue detection model is released for use in a motor vehicle if the quality measure reaches or exceeds the specified threshold. In other words, the trained driver-specific fatigue detection model is not released for use in the motor vehicle, especially if the quality measure falls below the specified threshold. Thus, the driver-specific fatigue detection model is only used in a motor vehicle for the early detection of a driver's fatigue if the motor vehicle has a sufficiently high quality, which is characterized by the determined quality measure exceeding or reaching the specified threshold.

[0009] In this context, it may also be intended that accuracy be used as a quality measure. It is also quite common to refer to this accuracy as so-called accuracy. Accuracy describes the ratio of fatigue events correctly predicted by the driver-specific fatigue detection model based on the input data, for which respective fatigue events have been identified by the global fatigue detection model, to all fatigue events predicted by the driver-specific fatigue detection model based on the input data. The respective fatigue events can be predicted by the driver-specific fatigue detection model in particular if early signs of fatigue in the driver have been identified by the driver-specific fatigue detection model based on the input data.If the driver-specific fatigue detection model determines that the driver exhibits at least one of the predefined early warning signs of fatigue, the driver-specific fatigue detection model predicts the occurrence of a fatigue event. This predicted occurrence of the fatigue event can either be confirmed by the global fatigue detection model also detecting a fatigue event, or it can be refuted as false if the global fatigue detection model has not detected a fatigue event based on the input data for a given period.

[0010] As an alternative to accuracy, precision can be used as a measure of quality. Accuracy and precision are two terms that refer to measurement errors. Accuracy indicates how close a given set of measurement results is to a true value, while precision indicates how close the measured values ​​are to each other. In other words, precision describes the random error, a measure of statistical variability. For a statistical sample or set of data points from repeated measurements of the same quantity, the sample or set can be said to be accurate if its average is close to the true value of the measured quantity, while the set can be said to be precise if the standard deviation is relatively small.

[0011] In a further possible embodiment of the invention, the trained driver-specific fatigue detection model is provided as the basis for a global early fatigue detection model. This means that the already trained driver-specific fatigue detection model can in turn be provided to a server device superordinate to the motor vehicle, in which the global early fatigue detection model is created based on the driver-specific fatigue detection model. To create this global early fatigue detection model, the trained driver-specific fatigue detection model can be trained using input data from a large number of different drivers, in particular from drivers of a large number of different motor vehicles.This allows the global early fatigue detection model to be created based on the driver-specific fatigue detection model, generalized by training with the input data of the different drivers or the different vehicles, and used to reliably detect fatigue events and, if necessary, additional early fatigue detection signs.

[0012] The invention further relates to a method for detecting fatigue in a driver of a motor vehicle. The method provides for the driver to be examined for the presence of early signs of fatigue using an individual fatigue detection model for this driver. The individual fatigue detection model is, in particular, an individual fatigue detection model trained within the framework of the method according to the invention already described. The individual fatigue detection model is thus particularly well adapted to the driver of the motor vehicle and, as a result, particularly well-equipped to reliably detect early signs of fatigue in the driver.Because the driver is examined for early signs of fatigue using the individual fatigue detection model as part of the fatigue detection process, an impending driver fatigue event can be detected particularly reliably based on the detected early signs of fatigue while the driver is driving the vehicle. As a result, the impending driver fatigue event can be counteracted very early, for example, by triggering an action.

[0013] In a possible further development of the invention, it is provided that as soon as at least one individual early detection sign of fatigue of the driver has been detected by means of the driver-specific fatigue detection model, in particular at least one predetermined individual early detection sign of fatigue, an information signal is output via an output device of the motor vehicle. This information signal characterizes the detection of the early detection sign of fatigue. This means that by outputting the information signal via the output device, the driver can be informed that at least one early detection sign of fatigue has been detected. This informs the driver of the impending fatigue event. As a result, it is possible for the driver to actively counteract the development of fatigue.This can delay the fatigue event and / or the driver can ensure that the driver stops driving the motor vehicle before the fatigue event occurs, for example by ending the journey or by the driver handing over control of the motor vehicle to another person.

[0014] In this context, it can be provided, in particular, that at least one fatigue prevention measure is suggested by means of the output device, at least one fatigue prevention measure is selected as a function of a user input, and the at least one selected fatigue prevention measure is triggered, wherein the activation of any fatigue prevention measures is omitted if no user input is determined which characterizes the selection of a suggested fatigue prevention measure. In particular, several different fatigue prevention measures can be suggested to the driver by means of the output device, of which the driver can select at least one via user input. The at least one fatigue prevention measure selected by the driver via the user input is subsequently triggered.This makes it particularly effective to counteract the driver's increased fatigue, particularly by selecting at least one fatigue prevention measure preferred by the driver. If the driver does not select a suggested fatigue prevention measure, no fatigue prevention measures will be activated. This allows the driver to avoid initiating any fatigue prevention measures by not selecting a fatigue prevention measure—for example, in the event of incorrect detection of early warning signs of fatigue.

[0015] In a further possible embodiment of the invention, it is provided that, as a fatigue prevention measure, a temperature control of an air conditioning system of the motor vehicle is adjusted and / or a volume and / or a genre of music played in the motor vehicle is adjusted and / or a color of ambient lighting is adjusted and / or a massage seat of the motor vehicle is activated. Within the scope of the temperature control of the air conditioning system, for example, a target temperature of the air conditioning system can be lowered, whereby the driver can be awakened by cooling the vehicle interior. Cool interior temperatures can counteract fatigue particularly easily and effectively.When adjusting the genre of music played in the vehicle interior, for example, you can select music with a heavy bass or short beats, which can stimulate the driver. Increasing the volume of the music played in the vehicle can also help wake the driver up. A color considered stimulating can be selected for the ambient lighting. By activating the vehicle's massage seat, the driver can be made more alert through tactile contact with the massage seat.

[0016] The invention further relates to an electronic computing device for a motor vehicle, which is configured to carry out a method as already described in connection with the inventive method for fatigue detection, or to carry out a method for training an individual fatigue detection model, as already described in connection with the inventive method for training an individual fatigue detection model. The electronic computing device can thus be configured to train the individual fatigue detection model and / or to detect the driver's fatigue in the motor vehicle. In this case, the electronic computing device is configured, in particular, to carry out the respective method steps of the respective methods.

[0017] The invention may further relate to a motor vehicle comprising the described electronic computing device according to the invention.

[0018] Further features of the invention can be derived from the following description of the figures and from the drawings. The features and combinations of features mentioned above in the description, as well as the features and combinations of features shown below in the description of the figures and / or in the figures alone, can be used not only in the respective combinations specified, but also in other combinations or on their own, without departing from the scope of the invention.

[0019] The drawing shows: Fig. 1 shows a method diagram for a method for training an individual fatigue detection model for a driver of a motor vehicle; and Fig. 2 a process diagram for a method for detecting fatigue of a driver of a motor vehicle.

[0020] In Fig. Figure 1 shows a process diagram for a method for training an individual fatigue detection model for a driver of a motor vehicle. The method comprises three method steps VA1 to VA3. In the first method step VA1 of the method, the driver is identified. In the second method step VA2 of the method, the driver's fatigue is examined using a global fatigue detection model based on input data that serves as an indicator of the driver's fatigue.In the third method step VA3 of the method, if driver fatigue is detected, a fatigue event is detected, and at least a portion of the input data determined for a defined period prior to the onset time of the fatigue event for which the onset of driver fatigue was detected is used as training data for the driver-specific fatigue detection model. The driver-specific fatigue detection model is then trained to detect individual early signs of fatigue of the identified driver.

[0021] Furthermore, it can be provided that a quality measure is determined for the trained driver-specific fatigue detection model with regard to early fatigue detection. Within the scope of early fatigue detection, the occurrence of a fatigue event, particularly for the near future, is predicted based on at least one identified early fatigue detection sign of the driver. If the determined quality measure falls below a predefined threshold, input data from another fatigue event is used as training data for training the driver-specific fatigue detection model. If it is determined that the quality measure reaches or exceeds the predefined threshold, the trained driver-specific fatigue detection model can be released for use in a motor vehicle.

[0022] Accuracy is used as a quality measure. This accuracy describes the ratio of fatigue events correctly predicted by the driver-specific fatigue detection model based on at least one detected individual early warning sign of fatigue of the driver, for which the respective fatigue events have been confirmed by the global fatigue detection model, to all fatigue events predicted by the driver-specific fatigue detection model based on the input data.

[0023] It is possible that the trained driver-specific fatigue detection model could be used as the basis for a global early fatigue detection model designed to detect early signs of fatigue. Thus, with the consent of the respective driver, the individualized data for early fatigue detection could be used to develop the global early fatigue detection model.

[0024] The basis for the described method is therefore the already trained global fatigue detection model, which is capable of detecting general fatigue indicators. The global fatigue detection model can be based on camera data from a driver observation camera and other vehicle data, such as steering and braking data, as input data. Within the framework of the method, the global fatigue detection model can be customized for each driver using transfer learning to detect driver-specific early warning signals of fatigue. Transfer learning is a machine learning technique in which a model already trained for one task is used as a basis and applied to a different task.For example, the global fatigue detection model can be used as the basis for the individual early fatigue detection model and the training data can be used to train the detection of early fatigue detection signs as a new task.

[0025] The method can use driver recognition to identify different drivers, which makes it possible to assign sensor data to the respective driver-specific fatigue detection models depending on the identified driver. The method collects sensor data. In particular, all the sensor data required for fatigue detection can be collected over an entire driving period. As soon as the global fatigue detection model detects the onset of a fatigue event, the sensor data from the last few minutes, particularly the last five to 20 minutes, is stored as training data for early fatigue detection. Transfer learning of the driver-specific fatigue detection model then takes place, in which parameters of the individual fatigue detection model are retrained using the stored training data as part of a fine-tuning process.

[0026] Subsequently, the accuracy of the individual fatigue detection model can be calculated as a quality measure for the period for which the input data was used as training data, specifically the period from five to 20 minutes before the onset of the fatigue event. If the accuracy is less than the specified threshold, which could be 90 percent, for example, additional training data is generated to train the driver-specific fatigue detection model. If the desired specified accuracy is reached or exceeded, it can be assumed that the individual fatigue detection model has learned individual early fatigue detection signs for the driver.This trained model can then be used for driver-specific fatigue detection for that driver in the vehicle, and if early signs of fatigue are detected, at least one fatigue prevention measure can be initiated. Further training of the driver-specific fatigue detection model with additional generated training data can be continued for further improvement even after the specified threshold has been reached by the accuracy.

[0027] In Fig.Figure 2 shows a method diagram for a method for detecting fatigue in a driver of a motor vehicle. The method comprises a first method step VB1 and a second method step VB2. In the first method step VB1, the driver is examined for the presence of early detection signs of fatigue using the individual fatigue detection model trained in the first method. As soon as at least one individual early detection sign of fatigue in the driver has been detected in the first method step VB1 using the driver-specific fatigue detection model, the second method step VB2 triggers the output of an information signal by an output device of the motor vehicle. This information signal characterizes the detection of the early detection sign of fatigue.The information signal can thus be used by the output device to particularly effectively inform the driver about the detection of the presence of the early detection sign of fatigue. Additionally, in the second method step VB2, at least one fatigue prevention measure can be suggested to the driver via the output device. Depending on a user input detected in response to the suggestion of the at least one fatigue prevention measure, at least one of the suggested fatigue prevention measures can be selected. In other words, the driver can select at least one of the suggested fatigue prevention measures via the user input. This selected at least one fatigue prevention measure is then triggered.As a fatigue prevention measure, the temperature control of an air conditioning system in the motor vehicle can be adjusted and / or the volume and / or genre of music played in the motor vehicle can be adjusted and / or the color of ambient lighting can be adjusted and / or a massage seat in the motor vehicle can be activated. The driver can thus be offered various fatigue prevention measures from which they can select the desired measures that are implemented upon early fatigue detection. If an early fatigue detection sign is detected by the individual fatigue detection model, this can be shown to the driver, in particular on a display as an output device, whereby the driver can be actively asked via the display whether at least one of the fatigue prevention measures suggested by the output device should be initiated.

[0028] Both the described method for training the individual fatigue detection model and the method for fatigue detection can be carried out by means of an electronic computing device of the motor vehicle.

[0029] The described methods enable early fatigue detection, in which early signals of driver fatigue can be identified by collecting input data preceding a fatigue event and making it available to the individual fatigue detection model as training data. Using transfer learning, the existing global fatigue detection model can be individually retrained for each driver, thus obtaining the individual fatigue detection model. By training this transferred fatigue detection model on individual early warning signals of driver fatigue, the global fatigue detection model can be expanded into a local and individualized model for early fatigue detection. Furthermore, the individual fatigue detection model can be continuously improved through training within the method.

[0030] Overall, the invention shows how individualized detection of early warning signals for fatigue can be achieved. List of reference symbols VA1 to VB2 respective procedural steps

Claims

[1] Method for training an individual fatigue detection model for a driver of a motor vehicle, in which - the driver is identified (VA1), - using a global fatigue detection model, driver fatigue is investigated using input data that serve as indicators of driver fatigue (VA2), - when the driver's fatigue is detected, a fatigue event is detected and at least part of the input data which has been determined for a defined period of time before the onset of fatigue of the fatigue event for which the onset of the driver's fatigue has been detected is used as training data for the driver-specific fatigue detection model which is trained to detect individual early signs of fatigue of the identified driver (VA3). [2] Method according to claim 1, wherein a quality measure for the trained driver-specific fatigue detection model is determined with regard to early fatigue detection, wherein input data of a further fatigue event are used as training data for training the driver-specific fatigue detection model if the determined quality measure falls below a predetermined threshold value. [3] Method according to claim 2, wherein the trained driver-specific fatigue detection model is released for use in a motor vehicle when the quality measure reaches or exceeds the predetermined threshold value. [4] Method according to claim 3, wherein an accuracy is used as a quality measure, wherein the accuracy describes the ratio of fatigue events correctly predicted by the driver-specific fatigue detection model on the basis of the input data, for which respective fatigue events have been determined by the global fatigue detection model, to all fatigue events predicted by the driver-specific fatigue detection model on the basis of the input data. [5] Method according to one of the preceding claims, wherein the trained driver-specific fatigue detection model is provided as a basis for a global fatigue early detection model. [6] Method for detecting fatigue of a driver of a motor vehicle, in which the driver is examined for the presence of early detection signs of fatigue by means of an individual fatigue detection model for the driver, which has been trained in a method according to one of the preceding claims (VB1). [7] Method according to claim 6, wherein, as soon as at least one individual early detection sign of fatigue of the driver has been detected by means of the driver-specific fatigue detection model, an information signal is triggered by means of an output device of the motor vehicle (VB2), which information signal characterizes the detection of the early detection sign of fatigue. [8] Method according to claim 7, wherein it is triggered that at least one fatigue prevention measure is proposed by means of the output device (VB2), at least one fatigue prevention measure is selected as a function of a user input and the at least one selected fatigue prevention measure is triggered, wherein the activation of any fatigue prevention measures is omitted if no user input is determined which characterizes the selection of a proposed fatigue prevention measure. [9] Method according to claim 8, wherein at least one of the following measures is carried out as a fatigue prevention measure: - a temperature control of an air conditioning system of the motor vehicle is adjusted, - the volume and / or genre of music played in the vehicle is adjusted, - a color of an ambient light is adjusted, - a massage seat in the vehicle is activated. [10] Electronic computing device for a motor vehicle, which is designed to carry out a method according to one of the preceding claims.

Images