Driver fatigue detection from differential inertial data of a smartwatch

Abstract

The invention relates to a system for determining a physical condition of an occupant of a vehicle (1), comprising a mobile terminal (3) that can be fixed to the body and has a first inertial sensor (5), as well as a second inertial sensor (7) for attachment to the vehicle (1), wherein the first inertial sensor (5) and the second inertial sensor (7) are used to respectively detect a kinematic variable, and further comprising a computing unit (9) that is designed to relate sensor data from the first inertial sensor (5) and the second inertial sensor (7) to a common measurement time for synchronization and to subtract the synchronized sensor data from the second inertial sensor (7) from the sensor data from the first inertial sensor (5) in order to obtain kinematic data about movements of a body part with the fixed terminal (3) relative to the vehicle (1), and to determine a physical condition of the occupant therefrom.

Description

The invention relates to a system for determining a physical state of an occupant of a vehicle, to a vehicle having such a system, and to a method for determining a physical state of an occupant of a vehicle.A large part of accidents in road traffic is caused by the inattentiveness and tiredness of drivers of motor vehicles. There are therefore many efforts in the prior art to detect tiredness and to warn the driver of the vehicle. Approaches to this include, for example, the machine image evaluation of camera images which record the face of the driver. Also, portable sensors are an option for monitoring an occupant of a motor vehicle according to the prior art.DE 10 2020 212 965 A1 relates to a method for monitoring an occupant of a motor vehicle for existing tiredness in which, at least when units of the motor vehicle are put into operation: a) first measured values are continuously recorded by at least one first sensor unit of the motor vehicle, b) second measured values are continuously recorded by at least one sensor unit of a device carried on the body of the occupant, c) the first measured values are checked according to a first criterion to determine whether a tiredness exists, d) the second measured values are checked according to a second criterion to determine whether a tiredness exists, and e) the third measured values recorded by a second sensor unit of the motor vehicle are continuously checked according to a third criterion to determine whether a tiredness exists, and f) if the result is according to at least one of the first, second and third criterion, that there is tiredness, an output of a warning and / or blocking of a function of the motor vehicle and / or activation of a function of the motor vehicle takes place.Furthermore, smart watches which are increasingly emerging are known from practice, which can record vital data by their natural permanent contact with the skin surface of the wearer of the smart watch. Whereas, as is known from smartphones, kinematic data can typically also be detected with such smart watches, which are determined, for example, by position sensors and / or rotation rate sensors and by acceleration sensors, fatigue detection for a driver of a vehicle is practically not possible with this raw data. The reason for this is that such inertial sensors, among which the position sensors, rotation rate sensors or acceleration sensors mentioned fall, measure absolute values.If a person wearing the smart watch is located in a moving vehicle, an inertial sensor of the smart watch measures not only the movement of the watch on the arm relative to the vehicle seat, but also the kinematic data of the vehicle, for example during accelerations, decelerations, cornering, unevenness in the road surface and the like.From the raw sensor data of a smart watch alone, it is in principle not possible to determine whether the movement of the vehicle with respect to the ground or the movement of the wrist with the smart watch with respect to the interior of the vehicle is causal for measured movement data. In principle, however, the kinematic data from inertial sensors of a smart watch would be suitable for determining a physiological state, such as, for example, for determining fatigue, in isolation from the movement of the vehicle.It is therefore an object of the invention to use kinematic sensor data from at least one inertial sensor worn close to the body, in particular a smart watch of a driver of a vehicle, to carry out the determination of a physiological state of the driver during the journey.The invention results from the features of the independent claims. Advantageous refinements and refinements are the subject matter of the dependent claims.A first aspect of the invention relates to a system for determining a physical state of an occupant of a vehicle, comprising a mobile terminal device fixable to the body having a first inertial sensor, and comprising a second inertial sensor for attachment to the vehicle, wherein the first inertial sensor and the second inertial sensor are used for respectively detecting a kinematic variable, in particular of the same category, and further comprising a computing unit which is designed to relate sensor data of the first inertial sensor and of the second inertial sensor to a common measurement time for synchronization and to subtract the synchronized sensor data of the second inertial sensor from the synchronized sensor data of the first inertial sensor in order to obtain kinematic data about movements of a body part having the terminal device fixed thereto relative to the vehicle and to determine a physical state of the occupant therefrom.The system according to the invention can be applied to a passenger car, a truck, trains, airplanes, ships and other vehicles. Particularly advantageous applications result in manually guided passenger cars in which a fatigue warning is relevant for preventing traffic accidents; however, also in automatically driving passenger cars which require neither manual vehicle guidance nor monitoring of the journey carried out by a driver. In the latter case, a corresponding reaction can be triggered by the determination of a physical state, in particular whether the occupant is sleeping or awake, in order to be able to adapt entertainment systems or other subsystems of the vehicle, for example. For example, when a sleeping occupant is detected, the entertainment in the interior may be suspended, the air conditioning system may be adjusted accordingly, and / or a driving style may be switched to a gentle driving style.In order to be able to accurately detect the physical state of the occupant, in particular of the driver of the vehicle, kinematic data is required which the occupant causes by a movement with respect to the interior of the vehicle. Kinematic data that the vehicle exerts with respect to the environment of the vehicle is subtracted from that acquired at the first inertial sensor of the mobile terminal. This procedure is based on the known relationship that the absolute kinematic data measured at the first inertial sensor are composed of the movements of the vehicle with respect to the environment and the movements of that body part of the occupant on which the mobile terminal is located with respect to the vehicle. For example, accelerations and rotation rates that the occupant exerts with his body part relative to the vehicle and such accelerations and rotation rates that the vehicle has relative to the environment add up.Both the first inertial sensor and the second inertial sensor are designed to ascertain respective kinematic sensor data. The determined sensor data are in particular from the same category, i.e. they describe information about the same size. In the simplest case, only coordinate system transformations are to be used, for example, in order to break down the directions of measured translatory accelerations into their components in order to be able to indicate these in a harmonized coordinate system. Only accelerations and rotational modes referenced to the same coordinate system and synchronized to each other can be meaningfully subtracted from each other. The "same category" thus excludes, for example, a detection of once a rotational variable and once a translational variable, since these cannot be subtracted from one another in a meaningful manner. However, integration over time is allowed to fall in the same category. For example, a rotational acceleration can be determined by one of the sensors, which is calculated in the computing unit in a temporally integrated manner to form a rotation rate, while the other of the inertial sensors measures a rotation rate at the same time. As a result, rotation rates of rotation rates and accelerations of accelerations, which are described in particular in the same coordinate system, preferably a vehicle-mounted coordinate system, are to be subtracted.In order that the sensor data of the second inertial sensor can be correctly subtracted from those of the first inertial sensor, a respective data pair which is subtracted from one another must also be related to one and the same point in time. Since sensors in general and the transmission of their sensor data, such as an inertial sensor in a mobile terminal, whose sensor data are transmitted wirelessly to the computing unit of the vehicle, typically have a certain time offset at the measurement time, and this time offset can be different for each transmission and each inertial sensor, the sensor data must be synchronized.This means effectively that the sensor data arriving first at the computing unit are retained and temporarily stored until the captured sensor data of the other sensor are likewise available at the computing unit. This can be done in various ways:If both inertial sensors acquire and transmit their sensor data to the computing unit in real time (i.e. not without any time offset, also called latency, but rather to a known, predictable time offset), the sensor data of the inertial sensor transmitting more quickly can be held by the difference of latencies and delayed. However, if these latencies are variable and / or unknown for some other reason, a synchronization can be applied by automatic methods, for example by significant events being determined as references for the synchronization by sensor analysis. While the second inertial sensor is preferably installed in a fixed manner in the vehicle, it can already be determined at the design time by corresponding latency measurements or predictions, and corresponding protocols such as UDP can be used to enable transmission of known and constant latency. If the mobile terminal is also already prepared for the application according to the invention in its design phase, corresponding mechanisms can also be provided therein. Alternatively or in addition to these possibilities, further possibilities known in the prior art for signal synchronization can be used, for example via the emission of time stamps of the first inertial sensor and of the second inertial sensor from respective reference clocks, which in turn are synchronized. The respective time stamp can then be transmitted together with the associated sensor data in a data packet, and the sensor data can be assigned to one another with reference to the same time by the time stamps in the computing unit. As an alternative to the time stamps of the synchronized clocks, further known methods can be used, for example the method of the rolling counter as is known from vehicle CAN data buses.The first inertial sensor preferably transmits only raw sensor data from the mobile terminal without modifying the latter. The advantage is that a predictable time delay, i.e. latency, can thus be achieved more easily in the transmission to the computing unit, in particular because a real-time sensor processing is more difficult to implement in the mobile terminal than in the computing unit and in the computing unit provided for determining the physical state of the occupant, corresponding programs and algorithms can be adapted more easily to this purpose. In addition, mobile terminals typically require a mobile energy supply, for example by means of a battery, while only a very small energy current is necessary in the computing unit for calculations necessary for sensor processing in comparison with the required drive power.The mobile terminal is preferably a smart watch. Other so-called "wearables" can also be used, including a smartphone with a holder, for example, on the upper arm of the occupant.According to an advantageous embodiment, the computing unit is designed to determine at least one of the following physical states of the occupant: awake, tired, sleeping.According to a further advantageous embodiment, the computing unit is arranged in the vehicle.According to a further advantageous embodiment, the second inertial sensor is an acceleration sensor which is arranged on or under a vehicle seat.According to a further advantageous embodiment, the mobile terminal fixable to the body is a smart watch.According to a further advantageous embodiment, the first inertial sensor of the mobile terminal fixable to the body is designed to transmit raw sensor data to the computing unit in near real time without modification.According to a further advantageous embodiment, the computing unit is designed to determine and execute a reaction depending on the determined physical state of the occupant.According to a further advantageous embodiment, the reaction is one of the following: warning the occupant in the role of a manually vehicle-guiding driver of inattentivity due to fatigue or sleep; in each case in the case of fully automatic driving: lowering the sound level of acoustic entertainment when detecting sleep, obscuring windows of the vehicle when detecting sleep, changing a parameter in a digital driving control system, or changing a chassis parameter.According to a further advantageous embodiment, the computing unit is designed to additionally use at least one further sensor value of a non-inertial sensor for determining the physical state of the occupant for the purpose of ascertaining physical data of the occupant.Such a further sensor value of a non-inertial sensor preferably comprises at least one of: pulse, electrical conductivity of the skin, muscle activity, brain activity, movement of another body part, in particular detected by a camera. All sensor data, from inertial and non-inertial sensors, may be fed into a common model to determine the physical condition of the occupant. The more sensor data of different sources and categories are acquired, the more reliable the physical state of the occupant can be determined in principle, but the more challenging the calculation, in particular the complexity of the model used (i.e. the rule for mapping the sensor data into a physical state), also becomes.A further aspect of the invention relates to a vehicle having a system as described above and below.Advantages and preferred refinements of the proposed vehicle result from an analogous and analogous transfer of the explanations given above in connection with the proposed system.A further aspect of the invention relates to a method for determining a physical state of an occupant of a vehicle, wherein a first inertial sensor of a terminal device fixable to the body of the occupant and a second inertial sensor of the vehicle each record a kinematic variable, in particular of the same category, in particular one of rotational and translational, and transmit it to a computing unit, which relates the sensor data of the first inertial sensor and of the second inertial sensor to a common measurement time for synchronization and subtracts the synchronized sensor data of the second inertial sensor from the synchronized sensor data of the first inertial sensor in order to obtain kinematic data about movements of a body part with the fixed terminal device relative to the vehicle, wherein a physical state of the occupant is determined from this by the computing unit.Advantages and preferred refinements of the proposed method result from an analogous and analogous transfer of the statements made above in connection with the proposed system.Further advantages, features and details are evident from the following description, in which--possibly with reference to the drawing--at least one exemplary embodiment is described in detail. Identical, similar and / or functionally identical parts are provided with the same reference numerals.The following are shown: FIG. 1 : A vehicle having a system for determining a physical state of an occupant according to an exemplary embodiment of the invention. FIG. 2 : shows an exemplary computing logic in the system of FIG. 1.FIG. 1 shows a vehicle 1 with an occupant who is simultaneously a driver of the vehicle 1. Below a vehicle seat of the vehicle 1, a second inertial sensor 7 is arranged. This measures a kinematic variable of the vehicle 1. The driver of the vehicle 1 wears a mobile terminal 3 in the form of a smart watch (a miniature wrist computer to extend the functions of a digital clock). The mobile terminal 3 has a first inertial sensor 5, with the aid of which sensor data are determined in a terminal coordinate system. The sensor data of the first inertial sensor 5 and of the second inertial sensor 7 are of the same category, for example translatory accelerations which are related to a coordinate axis of the reference coordinate system. This coordinate system transformation is made possible with the aid of a position angle sensor in mobile terminal 3, which outputs, for example, Euler angles of mobile terminal 3 with respect to a fixed-earth coordinate system, while the orientation of the reference coordinate system with respect to this fixed-earth coordinate system is likewise known by computing unit 9 of vehicle 1, so that sensor data can be subtracted from one another. Furthermore, for the purpose of meaningful subtraction, a temporal synchronization of the sensor data from the mobile terminal 3 and from the second inertial sensor 7 is necessary, which is effected, for example, by means of time stamps of synchronized clocks of the vehicle 1 and of the mobile terminal 3. The first inertial sensor 5 and the second inertial sensor 7 are representative here of a multiplicity of respective inertial sensors 5, 7 both in the vehicle 1 and in the mobile terminal 3, the sensor data of which can be compared in pairs. Preferably, three translatory acceleration sensors are provided for measuring accelerations in three mutually perpendicular coordinate axes. Possible further inertial sensors for obtaining the sensor data are, for example, gyroscopes for determining angles with respect to the earth and for determining rotation rates, which can also be obtained by integration over time of rotation accelerometers.FIG. 2 shows a respective possible signal path from the first inertial sensor 5 and from the second inertial sensor 7 for determining the state of the occupant in the computing unit 9. In comparison with this time scale, the inertial sensors 5, 7 transmit their sensor data to the computing unit 9 at a significantly higher frequency, so that artificial delay by temporary temporary buffering of the data of a data stream does not mean a relevant loss of time in information acquisition. The data of the first sensor 5 of the mobile terminal 3 are transformed into T onto a common reference coordinate system, and since these are measured rotational accelerations, are integrated in time in the signal path. This takes place in the computing unit 9. the transmission of the sensor data from the mobile terminal 3, which originate from the first inertial sensor 5, is transmitted wirelessly to the computing unit 9. The sensor data of the second sensor 7 arranged under the vehicle seat are transformed to the same reference coordinate system in T and, since these are speeds such as rotation rates, are not integrated. Thus, there are rotational speed information of the vehicle 1 with respect to the environment related to a common reference coordinate system and rotational speed information of the wrist with the terminal 3 with respect to the vehicle 1. After the two sensor data streams have been synchronized in time S, they are subtracted from one another, so that kinematic data only with respect to the vehicle 1, exerted by movements of the wrist of the occupant, remain from the first inertial sensor 5 of the mobile terminal 3. These are fed into a model M and a current physiological state of the occupant is determined therefrom.Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a large number of possible variations exist. It is also clear that embodiments mentioned by way of example represent only examples which are not to be understood in any way as limiting, for example, the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, wherein the person skilled in the art, knowing the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.List of reference characters1 Vehicle 3 Mobile terminal 5 First inertial sensor 7 Second inertial sensor 9 Computing unit M Model T Coordinate system transformation S SynchronizationReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedDE 10 2020 212 965 A1

[0003]

Claims

System for determining a physical state of an occupant of a vehicle (1), comprising a mobile terminal (3) fixable to the body with a first inertial sensor (5), and comprising a second inertial sensor (7) for attachment to the vehicle (1), wherein the first inertial sensor (5) and the second inertial sensor (7) are used for the respective detection of a kinematic variable, and further comprising a computing unit (9) which is designed to relate sensor data of the first inertial sensor (5) and of the second inertial sensor (7) to a common measurement time for synchronization and to subtract the synchronized sensor data of the second inertial sensor (7) from the synchronized sensor data of the first inertial sensor (5) in order to obtain kinematic data about movements of a body part with the terminal (3) fixed thereto relative to the vehicle (1), and to determine a physical state of the occupant therefrom.The system according to claim 1, wherein the computing unit (9) is configured to determine at least one of the following physical conditions of the occupant: awake, tired, sleeping.System according to one of the preceding claims, wherein the computing unit (9) is arranged in the vehicle (1).The system of claim 3, wherein the second inertial sensor (7) is an accelerometer disposed on or below a vehicle seat.The system according to any one of the preceding claims, wherein the mobile body fixable terminal (3) is a smart watch.The system according to any one of the preceding claims, wherein the first inertial sensor (5) of the mobile terminal (3) fixable to the body is configured to transmit raw sensor data to the computing unit (9) in near real time without modification.The system according to any one of the preceding claims, wherein the computing unit (9) is configured to determine and execute a reaction depending on the determined physical state of the occupant.The system of claim 7, wherein the response is one of: alerting the occupant in the role of a manually-riding driver of inattentiveness from fatigue or sleep; each in case of fully automatic driving: lowering the sound level of acoustic entertainment upon detection of sleep, obscuring windows of the vehicle (1) upon detection of sleep, changing a parameter in a digital driving control system, or changing a chassis parameter.System according to one of the preceding claims, wherein the computing unit (9) is designed to additionally use at least one further sensor value of a non-inertial sensor for determining the physical state of the occupant for the purpose of ascertaining physical data of the occupant.Vehicle (1) comprising a system according to one of the preceding claims.

Images