Driver physical condition determination for vehicle safety

Abstract

The invention relates to driver physical condition determination for vehicle safety. A computerized method of controlling one or more vehicle safety measures by determining a height of a person seated on a seat in a vehicle cabin is provided. The method includes determining a height of the person based on one or more images showing the person on the vehicle seat by determining a set of body features of the person, comparing the set of determined body features to a corresponding set of reference features, and determining the height of the person based on the comparison. The method further includes outputting a control signal to the one or more vehicle safety measures based at least in part on the determined height.

Description

Technical Field

[0001] The present disclosure relates generally to safety improvements for vehicles, and more particularly to methods and systems for controlling vehicle safety measures based on determining a physical condition of a vehicle driver. Background Art

[0002] Smart vehicles such as smart cars, smart buses, etc. are significantly improving the safety of passengers. Such smart vehicles may be equipped with onboard cameras and are able to capture images of the interior of the vehicle. These images may then be used, sometimes in conjunction with other sensors, for different safety-related tasks.

[0003] Safety tasks in vehicles involve the control of safety measures, such as airbag or seatbelt deployment, door or window locks, etc. In modern intelligent vehicles, these safety measures can be based on person detection and classification. Therefore, in future vehicles, person detection and classification can play an important role, for example, in seat occupancy detection. Seat occupancy detection is used to determine whether a person is located in a seat. However, this basic detection of whether a person is located in a seat is not sufficient to reliably control all safety measures in a reliable manner.

[0004] Therefore, there is a need for an improved system for controlling safety measures. Summary of the invention

[0005] One aspect of the present invention provides a computer-implemented method for determining a physical condition of a driver of a vehicle. The method includes: receiving at least one image captured by a vehicle camera, the image showing at least a portion of the driver and a sensor to be operated by the driver to determine the physical condition; and determining whether the sensor is operated by the driver based on the image.

[0006] Optionally, determining that the sensor is operated by the driver includes: identifying the driver among multiple persons in the vehicle, including: classifying the activities of the multiple persons; and assigning the detected body parts to corresponding persons among the multiple persons on the vehicle seats; and providing a unique identifier for the corresponding persons among the multiple persons.

[0007] Optionally, determining that the sensor is operated by the driver includes: locating and tracking at least one hand of the driver; and determining that the sensor is operated by the driver's hand.

[0008] Optionally, locating and tracking at least one hand of the driver based on the at least one image includes: tracking a body shown in the image, tracking a hand shown in the image, and assigning the at least one finger to the driver based on the tracked body and hand.

[0009] Optionally, determining that the sensor is operated by the driver's hand based on the at least one image includes: positioning the sensor, and determining the proximity of the driver's hand to the sensor.

[0010] Optionally, the method further comprises determining that the sensor is operated by a person other than the driver.

[0011] Optionally, the method also includes: generating a warning signal, and / or triggering a control action for preventing or stopping the operation of the vehicle, and the control action includes achieving a safe stop of the vehicle, and / or prompting the driver to confirm current physical driving ability.

[0012] Optionally, the method further comprises determining the physical condition of the driver via a sensor operated by the driver.

[0013] Optionally, determining the driver's physical condition also includes determining at least one of the driver's eye state, the driver's body movement, the driver's hand operation, object detection near the driver, and the driver's facial expression based on the at least one image.

[0014] Optionally, the driver's body movement includes drinking movement and / or smoking movement; and the object detection near the driver includes detecting a container that may contain liquid and / or detecting smoking products.

[0015] Optionally, determining the physical condition of the driver further comprises analyzing driving behavior, wherein the driving behavior comprises at least one of lane position, roundabout driving, steering wheel dynamics and acceleration / deceleration dynamics.

[0016] Optionally, the method is performed prior to vehicle start-up and includes preventing the vehicle from being operated until use of a sensor by the driver has been confirmed and the sensor output has confirmed that the driver's physical condition satisfies given conditions for operating the vehicle.

[0017] Optionally, the method is performed during operation of the vehicle and includes outputting a warning signal in response to determining that the sensor is operated by the driver and the sensor output indicates that the driver's physical condition may not be suitable for driving,

[0018] Optionally, the sensor includes at least one of a breath analyzer, a finger contact sensor, a pulse measurement device, a blood pressure device, an air analyzer, and a blood analyzer.

[0019] Another aspect of the invention relates to a system for controlling one or more vehicle safety measures, said system implementing a method as described herein.

[0020] Another aspect of the invention relates to a vehicle comprising: a camera for capturing one or more images; and a system for controlling one or more vehicle safety measures as described herein.

[0021] A final aspect of the invention relates to a computer program product comprising instructions which, when executed on a computer, cause the computer to carry out a method as described herein.

[0022] These and other objects, embodiments and advantages will become apparent to those skilled in the art from the following detailed description of the embodiments with reference to the accompanying drawings, the present disclosure not being limited to any particular embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The foregoing and further objects, features and advantages of the present subject matter will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings, in which like reference numerals are used to represent like elements, and in which:

[0024] Figure 1 An advanced version of this approach is shown.

[0025] Figure 2 An exemplary vehicle cabin is shown.

[0026] Figure 3 An exemplary image of a vehicle cabin captured by a vehicle camera is shown.

[0027] Figures 4 to 7 More detailed aspects of the functionality of the present invention are provided.

[0028] Figure 8 and Fig. 9 Functions used before the vehicle is started are shown.

[0029] Fig.10 and Fig.11 Implementation aspects of hand and face authentication are provided.

[0030] Fig.12 and Fig.13 This refers to functions used while the vehicle is in motion.

[0031] Fig.14 An exemplary camera system is shown.

[0032] Fig.15 is a diagram of a computing system that implements the functionality described herein. DETAILED DESCRIPTION

[0033] The present disclosure relates to methods and systems for controlling vehicle safety measures by determining the physical condition of the vehicle driver. Vehicles driven in whole or in part by human drivers are a safety concern if the human driver is in a physical condition that may impair driving ability. For example, driving under the influence is a common source of serious accidents. There are several active measures to prevent drunk drivers from operating vehicles, i.e., to enhance safety by not allowing the operation of the car when the driver is drunk, using marijuana, or otherwise intoxicated. Therefore, determining the physical condition of the driver plays an important role in ensuring driving and vehicle safety.

[0034] In order to solve these problems, the vehicle can be equipped with one or more sensing devices (referred to as sensors herein) that can determine one or more factors of the driver's physical condition. For example, in the United States, a person who has been convicted of drunk driving may be required to install an ignition interlock device, which only allows the car to be started after the alcohol level is shown to be below the legal threshold (e.g., using a breathalyzer). Although such sensors can output relatively reliable indications of the driver's current physical condition, these sensors do not ensure that they are actually operated by the driver, rather than by another person such as a passenger. Therefore, a positive indication from the sensor does not necessarily mean that the driver's current physical condition allows driving.

[0035] Additionally, the use of sensors such as an alcohol breathalyzer connected to the vehicle's ignition can prevent the vehicle from being operated before it is driven, but will not necessarily monitor the driver's intoxication during the ride.

[0036] Other methods use, for example, odor sensors that analyze the air in the cabin for specific gases associated with intoxication. However, these methods cannot distinguish between the intoxication state of the driver and passengers. For example, a drunk passenger may also trigger an alarm when the driver is sober.

[0037] Against this background, the present disclosure proposes a solution for safety measures control based on generally existing information available in the vehicle, which allows an improved determination of the driver's physical condition 10. Figure 1 The present invention utilizes image data provided by a vehicle's onboard camera. Although the following description relates to a camera, this also includes multiple cameras and utilizes images captured and provided by multiple cameras of a vehicle.

[0038] The method of the present invention is performed by a computerized platform (see Fig.15), which is typically an electronic control unit (ECU) in a vehicle. The method can be used for any type of vehicle that is at least to some extent operated by a human driver, such as a car (including a taxi), a truck, a bus, a train, a ship, an airplane, etc.

[0039] At a general level, a computerized platform receives image data representing an image captured by an onboard camera, such as a cabin camera. Cameras are typically already built into vehicles for other control purposes. For example, modern cars are typically equipped with a collection of multiple cameras that form an onboard camera system. Cameras that may include one or more of a color or monochrome camera, an infrared camera, a depth camera, and a thermal imaging camera provide images of the interior of the vehicle to the interior components of the vehicle. Images may be captured based on defined periodic or non-periodic time intervals and / or in response to another triggering event (e.g., opening a door, interacting with the vehicle's multimedia system, using voice commands, etc.). The captured images may then be used by multiple internal systems to control the safety devices described herein, as well as for entertainment and comfort use cases. The computerized platform that implements the present mechanism may be hosted by the camera itself, or may be communicatively connected to the camera.

[0040] The image of the vehicle interior may include a complete image showing the entire interior. Alternatively, the image may depict only partial areas of the interior, such as one front seat, one or more rear seats, door areas, etc. If the camera captures the entire interior, the image may be an image crop extracted from one or more original images of one or more areas in the vehicle. The image crop may be pre-processed, for example, by removing the effects of the sensor CFA (color filter array) pattern or by performing additional enhancement / pixel processing.

[0041] The image received at box 11 shows at least a portion of the driver and at least a portion of the sensor. The computerized platform processes the received image to determine whether the sensor is operated by the driver and not by any other non-driver. To facilitate determination 12, the system is able to locate and track hands, mouths, eyes, and objects. The system can also classify the activities of the passengers, assign detected body parts to people and seats, and provide a unique identifier for a given passenger, particularly the driver of the vehicle. Typically, the vehicle can be started before the vehicle is started (see below, for example Figures 8 to 10 ) and during operation of the vehicle (e.g. Figure 12 to Figure 13 ) Execute determination 12, Figure 1 Including these two stages.

[0042] Figure 2A non-limiting exemplary vehicle cabin 1 is shown. The vehicle cabin 1 features various standard controls and devices that enable the driver to operate the vehicle, such as a steering wheel, a gear lever, a smartphone holder, a display, a vehicle ignition button 4, etc., and a sensor 2. The sensor 2 can be any analyzing device that checks one or more factors indicative of the driver's physical driving condition. Non-limiting examples of the sensor 2 are a breath analyzer, a finger contact sensor, a pulse measuring device, a blood pressure device, an air analyzer, a blood analyzer (see also Figure 7 ).

[0043] Typically, the sensor 2 is manually operated by the driver. Figure 3 As shown in the exemplary case of FIG. 1 , the driver can grab the sensor 2 with one of his / her hands and operate the sensor 2 (e.g., by blowing a breathing air sample into the blowing port). The sensor then performs a corresponding analysis (e.g., analyzing the ethanol concentration in the breathing air) and outputs a result signal. The result signal is fed into the vehicle control system. If the result signal indicates a positive result (e.g., there is no ethanol concentration in the breathing air, or the ethanol concentration is below a given threshold), the driver is allowed to operate the vehicle. Otherwise, the operation of the vehicle may be prevented.

[0044] In some embodiments, the function for determining whether the sensor 2 is operated by the driver includes multiple sub-functions (e.g., Figure 4 , Figure 5 , Figure 6 ). Some of these sub-functions involve identifying a driver among multiple persons in or around a vehicle. For example, the driver of the vehicle may be sitting in a driver's seat, while a non-driver passenger of the vehicle may be sitting in another seat next to or behind the driver's seat, or may be standing next to the vehicle at the driver's door. The image received by the computerized platform may show the driver and the non-driver passenger or person. The sub-function for identifying the driver uses image processing and analysis techniques to identify which of the two persons in the image is the driver who should operate sensor 2.

[0045] In this regard, reference Figure 4, determining 11 whether the sensor is operated by the driver includes a sub-function of identifying 13 the driver among multiple people based on the image. Furthermore, identifying 13 the driver among multiple people in or at the vehicle may include classifying 14 the activities of multiple people shown in the image received from the camera, and assigning 15 the detected body part to a corresponding one of multiple people on the vehicle seat, and providing a unique identifier for the corresponding one of these people. Identifying 13 the driver can reuse the seat occupancy mechanism that already exists in the vehicle and has been used for other control purposes (e.g., determining the seat occupancy state of the seat in the vehicle to control the seat belt state and airbag pressure). The seat-based classifier module can use a fixed bounding box around the seat as input. The seat occupancy classification may also include one or more object-based modules, such as an object detector using a YOLO-based object detection algorithm, a body key point classifier, and / or a face detector. The classifier may also be arranged to detect people outside the vehicle, such as people standing next to the driver's door and still visible in the image.

[0046] Continue to refer to Figure 4 , another sub-function involves locating and tracking 16 the driver's hands. In some embodiments, locating and tracking 16 the driver's hands may include tracking 17 a body shown in the image, tracking 18 a hand shown in the image, and assigning 19 at least one finger in the hand tracked in the image to the driver. For example, body tracking 17 and hand tracking 18 may utilize bounding boxes and key points detected in the image and representing given hand and body points or elements. Assigning 19 the fingers to the driver may utilize proximity analysis between hand and body points.

[0047] For example, the distance between the wrist part key point and the bounding box of the hand is determined. For example, the minimum distance between the wrist part key point and the nearest edge of the bounding box of the hand or the distance from the wrist part key point to the center of the bounding box of the hand can be determined. If the distance between the wrist part key point and the bounding box of the hand is less than a threshold, the finger is assigned to the person associated with the wrist part key point. In order to handle the situation where the bounding box of the hand is close to, for example, multiple wrist part key points of multiple people or close to the right wrist and left wrist key points of a person, the hand can be assigned when another criterion is met. For example, if the distance between the bounding box of the hand and the corresponding wrist part key point is the minimum distance from the bounding box of the hand to all wrist part key points, the finger can be assigned to the person. Alternatively, other criteria can be applied. For example, a finger can be assigned to the wrist key point of the person, which has an average nearest distance to the bounding box of the hand over a period of time (e.g., on multiple images taken by the on-board cabin camera 3).

[0048] In some embodiments, locating and tracking 18 one or more hands may also utilize key points on the hands, such as hand joints or fingertips (hand skeleton). Based on the detected key points of the hands, the posture of the hands may be classified as, for example, "open hand", "hand grasping something", "hand holding an object", "hand operating a sensor", "hand performing a gesture", etc. This classification may be performed on an image crop derived from a bounding box of the hand, or on a skeleton of the hand.

[0049] In some embodiments, the above sub-functions can then be further utilized. Figure 5 As shown, in some embodiments, once at least one of the driver's hands is located and tracked 16 in the image, another sub-function can locate 20 the sensor 2 in the image and then compare the position of the driver's hand and the sensor to determine 21 the proximity of the driver's hand to the sensor. The determined proximity of the driver's hand to the sensor can indicate whether the driver is operating the sensor 2. Fig.10 Additional information is further given. In some embodiments, additionally or alternatively, the position of the sensor may be determined outside the system, for example, because the position of the sensor is known from another source. For example, the sensor may be integrated in the vehicle cabin (e.g., forming part of the ignition button), and therefore, the fixed position of the sensor may be known and may be retrieved from a vehicle-specific configuration file stored in the vehicle's memory or in an external memory. Therefore, locating the sensor 20 may also be achieved by retrieving the sensor position from an external source.

[0050] The outcome of the determination 12 as to whether the driver operated the sensor may be twofold: either it is confirmed that the probability that the sensor was operated by the driver is sufficiently high, or the determination 12 concludes that the sensor was operated by another person. Figure 6 The latter case is concerned when it is determined 22 that the sensor 2 is actually operated by someone other than the driver. In this case, the system can trigger appropriate actions, such as outputting 23 a warning signal and / or a control signal.

[0051] The warning signal may be an audio signal, a visual signal, or an audio-visual signal. For example, a control light may be activated using the vehicle's display, and / or a warning tone may be played through a speaker in the vehicle's cabin. If the driver's smartphone is communicatively coupled to the vehicle's control system, the warning signal may also be sent to the smartphone and output to the driver by the smartphone.

[0052] The control signal can generally prevent the vehicle from driving (before driving) or increase the safety level of the vehicle operation during driving. For example, the control signal can prevent the vehicle's engine from igniting, activate the vehicle's parking brake, prevent the vehicle's gear lever from remaining in the parking state, force the vehicle to brake (e.g., if the vehicle is moving, an automatic safety stop is performed, etc.). In some embodiments, the control signal can enable or adjust certain vehicle control functions performed by an advanced driver assistance system (ADAS), for example. For example, the control signal can activate lane keeping assistance, increase the safe distance to the vehicle in front, etc. In addition to the warning signal, the control signal can also be output. For example, when the warning function is activated to prompt the driver to stop manually safely, the control function can activate additional driving assistance to increase the safety level during vehicle driving until a manual safety stop is achieved. The warning signal and / or the control signal can be issued before the start / driving of the vehicle and during the driving of the vehicle, as discussed further below.

[0053] Figure 7 A positive situation is shown, i.e., it is determined 24 that the sensor is actually operated by the driver. In this case, the output of sensor 2 is processed to determine whether the driver's physical condition allows driving. The sensor output can be one, but not the only data for evaluating the driver's physical condition. Depending on the type of sensor 2, the sensor can measure or determine different physical factors. For example, sensor 2 can perform at least one of breathing analysis, finger contact analysis, pulse measurement, blood pressure measurement, air analysis, and blood analysis. Any specific type and function of sensor 2 as well as performance, reliability and output characteristics are included, because the method described here is universal for all sensor types, as long as the sensor is operated by at least one hand of the driver.

[0054] In addition to the analysis feedback from the sensor 2, in some embodiments, other data can also be used to evaluate the physical condition of the identified driver. For example, one or more images received and having been subjected to the sensor operation determination 12 can also be processed and analyzed to supplement the determination of the physical condition of the driver. For example, the image analysis can include at least one of the eye state analysis, body movement analysis, hand operation analysis, object analysis near the driver, and facial expression analysis. This analysis can utilize a neural network trained to detect certain objects, hand gestures, body movements, etc. in the image.

[0055] In some embodiments, additional data may also be used to again use the image data captured by the vehicle camera system and / or beyond the image data to determine the identified physical condition of the driver. For example, data available from monitoring the current operating state of the vehicle and / or the driving process of the vehicle may be analyzed. Such analysis may include, for example, at least one of a driving lane position analysis, a circuitous driving analysis, a steering wheel dynamics analysis, and an acceleration / deceleration analysis.

[0056] Figure 8 An exemplary system for implementing the present method at a functional abstraction level is shown. The system includes a sensing element (sensor 2) to determine one or more factors of the driver's physical condition, such as a level of intoxication operated by the driver through at least some interaction with the driver's hands (e.g., a breathalyzer). The system also includes an imaging system such as a cabin camera 3 that captures images of the vehicle interior. Figure 8 The activities and functions shown can be performed by a computerized platform ( Fig.15 The computerized platform receives the image and has image processing capabilities to detect and track 16 a hand shown in the image, detect an object, in particular locate 20 a sensor 2 (being one of the detectable objects) in the image data, and combine the results of the hand activity classification 16 and the sensor location 20, in particular determine 26 that the tracked hand actually operates the sensor 2.

[0057] The image processing function also includes a seat occupancy classification module 27 and a manual seat assignment module 28. The occupancy classification module 27 is configured, for example, to determine whether the driver's seat is occupied by a person, classify the driver's age group to distinguish from children, and detect changes in seat occupancy status, such as seat changes between people or people in the vehicle. The manual seat assignment 28 can utilize the above reference Figure 4 The functions described, for example, assign 15 a detected body part to a corresponding person on a vehicle seat and assign 19 at least one finger to the driver.

[0058] The results of the seat occupancy classification 27 and the assignment of hands to seats 28 are supplied to a determination logic unit 29 which ultimately determines whether the sensor is operated by the driver and at the same time whether the seat occupancy has not changed.

[0059] Figure 8 The logic unit may also consider a signal from sensor 2 indicating that sensor 2 is actually (and correctly) used. If such a "sensor operation" signal is noticed, for example by a sensor operation monitoring module (which may be part of the sensor 2 itself) (block 30, "yes"), it is propagated to the decision logic unit 31, which generates a fusion signal to the determination logic unit 29 to indicate that the sensor 2 is now actually operated. Otherwise, the logic loop returns (block 30). Figure 8until a sensor operation signal is detected.

[0060] If applied in a vehicle start scenario, the determination logic unit 29 determines whether the vehicle start will be allowed. This determination combines the information from the modules 26, 27, 28 and the trigger of the actual operation of the sensor (box 30, "yes"). In other words, if it is determined that the hand using the sensor originates from a person sitting in the driver's seat, and at the same time the driver's seat occupancy has not changed since the last seat occupancy classification was determined (meaning that the person occupying the driver's seat is still the driver of the vehicle), then the vehicle 32 will generally be allowed to start (unless the sensor results indicate that the driver's physical condition is not suitable for driving, which is in the case of Figure 8 Otherwise, if it is determined that the sensor is operated by a hand of a person who does not belong to the person in the driver's seat and / or a person who is sitting in the driver's seat and has changed seats or has left the vehicle, vehicle start-up is prevented 33.

[0061] Figures 9 to 11 A more detailed implementation example of a system for determining whether the driver has operated the sensor 2 before the vehicle is started is shown. Fig. 9 The implementation example utilizes hand verification, that is, detecting the driver's hand in the received image and determining whether the driver's hand operates sensor 2 (aspects of which have been described above). As an optional addition, the implementation example can also use facial verification technology and determine whether the driver operates sensor 2 by analyzing the driver's face. Fig. 9 As shown, the two authentication techniques of hand authentication and face authentication can be combined. Fig. 9 In a variation of the example of , hand verification is utilized instead of facial verification because the sensor operation determination based on hand verification solves the potential privacy issues of capturing biometrics of a person such as a driver. Hand tracking is one way to overcome privacy issues because sensitive information does not need to be acquired and processed.

[0062] Fig. 9 The system may be triggered by a signal 40 indicating that the vehicle is to be started, for example, the ignition button 4 of the vehicle has been pressed. Another trigger may be given by a signal 30 indicating that the use of the sensor has begun, as described above with reference to Figure 8 Already described.

[0063] generally, Fig. 9 The function of the system processes one or more images captured by the cabin camera 3. In some embodiments, multiple images of the vehicle cabin are processed. In some embodiments, the system periodically (for example, every 100 milliseconds) captures a continuous image frame stream for at least a period of time for the cabin camera 3.

[0064] In some embodiments, the image processing activity for the purposes described herein is triggered by a vehicle start signal, such as a signal 40 indicating that the vehicle ignition button 4 has been pressed. Then, the image processing activity is performed within a period of time after the ignition button press signal 40. Fig. 9 A further trigger may be given by a signal 30 indicating that the sensor 2 has begun to be operated or used, as described above with reference to Figure 8 As already described, as long as the signal 30 and / or the signal 40 are not received, the hand authentication 50 and / or the face authentication 60 can be kept in a standby or power saving state.

[0065] After being triggered by the ignition button press signal 40, the hand verification 50 and / or the face verification 60 input one or more images from the cabin camera 3. Typically, the hand verification 50 is determined by analyzing one or more images for the position of the driver's hand relative to the sensor 2 in order to determine whether the sensor is actually operated by the driver. The hand verification 50 then outputs a signal indicating whether the sensor has been used by the driver and whether it has been noticed that the seat occupancy has not changed. In addition, in some embodiments, the face verification 60 can analyze one or more images for the face belonging to the driver and the face operating the sensor 2. Both faces are given recognition. If the two facial recognitions are determined to correspond to each other, the facial verification 60 determines that the sensor 2 is operated by the driver.

[0066] The outputs from hand verification 50 and (optionally) face verification 60 are then combined with the actual sensor results 41, which may indicate adequate or inadequate physical condition of the driver. In response to determining at 42 that all outputs are positive, i.e.,

[0067] - sensors indicating 41 a suitable physical condition,

[0068] - hand verification 50 indicates that the sensor 2 is operated by the driver,

[0069] - Facial verification 60 indicates that the sensor 2 is operated by the driver (optional), the system determines that the vehicle start is allowed 32. Otherwise, in the event that one of the outputs is negative, the system prevents the vehicle start 33.

[0070] Fig.10 Provides implementation Fig. 9 Close-up of hands verifying the sub-function of 50.

[0071] The hand detection and tracking module 16 generally corresponds to the Figure 4The described localization and tracking hand functionality 16. This module includes an image processing algorithm that is configured to detect (locate) a hand in an image and track the detected hand in consecutive image frames (e.g., subsequent images captured by the cabin camera 3). In some embodiments, the hand detection and tracking module 16 utilizes a neural network model that is configured to detect a bounding box of a hand using, for example, an object detection model architecture. In some embodiments, the hand detection and tracking module 16 is configured to classify the posture of the hand detected in the image (e.g., open palm, grasping posture / holding an object, pointing with a finger, etc.).

[0072] The body tracking module 52 also uses image processing techniques to detect (locate) characteristic body key points of the human body, such as shoulders, hips, elbows, wrists, corners of the mouth, eye positions, ears, noses, fingertips, bone joints, points of the hands or facial feature points. A neural network can be used to determine the body key points. The body key point determination neural network can be an independent neural network, or in some embodiments, it can be part of a body recognition network used by an existing seat occupancy classification system of the vehicle. In addition, the body tracking module 52 groups the individual detections of the body key points into (corresponding) single persons. The body tracking module 52 can also assign people to predetermined seat areas in the vehicle cabin. Therefore, it can be determined which hand and body key points belong to (for example and in particular) the person in the driver's seat of the vehicle. In some embodiments, the body tracking module 52 is configured to output 2D (x, y) image coordinates or 3D (X, Y, Z) coordinates of the body key points in camera coordinates or vehicle coordinates.

[0073] The outputs of modules 16 and 52 are further processed by a hand assignment module 53. Module 53 utilizes, for example, a rule-based mechanism that combines information from the hand detection and tracking module 16 with the body tracking module 52. In some embodiments, the hand assignment module 53 is implemented by calculating a distance metric between a hand detection (e.g., a bounding box) and a key point of the body (e.g., a point of the wrist). In response to determining that the distance metric is within a (configurable) threshold, a bounding box can be assigned to the same person and seat position as the key body point. For example, the distance metric can be two-dimensional or multi-dimensional, taking into account the depth information and / or scale or hand posture information used to assign the logical unit. The output of the hand assignment module 53 can be 2D or 3D position information of one or both hands of the driver.

[0074] The sensor positioning module 20 generally corresponds to the reference Figure 5The sensor positioning module 20 detects the sensor 2 in the cabin camera image using image processing techniques. In some embodiments, the sensor positioning module 20 includes an object detection module configured to locate the sensor 2 according to the object bounding box in the image. The object detection module can be implemented by any suitable method that can output a bounding box and classify the object, such as a machine learning classifier. If the sensor 2 cannot move (its position in the vehicle cabin is fixed), the object detection module can be limited to a specific area of ​​the vehicle geometry of interest, which reduces processing resources and time. In some embodiments, the position of the sensor 2 can be configured based on the known cabin geometry and can be projected into the image using camera calibration parameters. These parameters can include non-intrinsic calibration (i.e., the translation and rotation of the camera in a given vehicle reference coordinate system) and camera intrinsics that capture imaging characteristics of the optical path (e.g., focal length, optical center, distortion model parameters). The output of the sensor positioning module 20 is 2D or 3D position information of the sensor 2.

[0075] The hand sensor proximity determination 21 generally corresponds to the referenced Figure 5 Functionality described. The hand sensor proximity determination 21 combines information from the finger assignment module 53 and the sensor positioning 20 using, for example, a rule-based mechanism. In some embodiments, a distance metric is calculated between the driver's hand position information (output of module 53) and the sensor position information (output of module 20). In response to determining that the distance metric is below a (configurable) threshold, it is determined that the hand is near the sensor 2. In some embodiments, tracking information (which may be obtained from the hand tracking function 16 across multiple image frames) is used to classify whether the sensor 2 and the tracked hand movements are consistent during operation of the sensor 2. The output of the hand sensor proximity determination 21 can be a binary output (yes or no) indicating close enough or far enough. In some embodiments, the output of the hand sensor proximity determination 21 can (additionally or alternatively) be a normalized value indicating the proximity between the hand and the sensor 2.

[0076] Hand to driver face proximity determination 55 utilizes, for example, a rule-based mechanism to combine information from finger assignment module 53 and driver face body key points of body tracking module 52. In some embodiments, when sensor 2 is (also) operated by the driver's mouth (e.g., a breathalyzer), a distance metric between the hand position information and the mouth position information indicating the position of the driver's mouth is calculated. In response to determining that the distance metric is below a (configurable) threshold, it is considered that the hand is close to the driver's mouth. If, for example, sensor 2 is operated by blowing into sensor 2 (i.e., moving sensor 2 close to the driver's mouth, or vice versa, moving the driver's mouth close to sensor 2), the use of hand to driver face proximity determination 55 is optional and exists / activated.

[0077] The seat occupancy classification 54 utilizes an image processing mechanism to classify the seat occupancy status of different seats in the cabin, wherein the status of the driver's seat is particularly relevant. The seat occupancy classification 54 is configured to, for example, distinguish empty seats from seats occupied by people. The seat occupancy classification 54 is also capable of distinguishing adults and older children from younger children (e.g., 6 years old or younger). For example, the seat occupancy classification 54 can determine objects in an image and assign objects to seats of a vehicle, which can then include analyzing images to detect objects, classifying object types, and outputting bounding boxes of objects detected over time and confidence values ​​for object type classification. Object types can include adults, child seats, children, animals, inanimate objects (e.g., bags, parcels, backpacks, child seats, clothes, etc.), empty seats, etc.

[0078] The seat occupancy classification 54 may also include a tracking mechanism for detecting changes in the seat occupancy state, such as detecting whether the seat occupancy state changes from empty to occupied or from occupied to empty. This facilitates determining whether the person in the driver's seat has changed.

[0079] The hand verification function 50 also includes a sensor use fusion module 56 that utilizes, for example, a rule-based mechanism to detect whether the sensor 2 is operated by the driver based on the information of the hand-to-sensor proximity 21 and the hand-to-face proximity 55 (as described above, which is optional). The sensor use fusion module 56 also inputs information from the sensor 2 itself, which indicates that the sensor 2 is being used or that sensor operation has been initiated (from Fig. 9 The sensor start function 30 shown in FIG. 1 may be used to receive the sensor start function 30, that is, the arrow from the start of the sensor use 30 may be used to Fig.10 The higher granularity representation input sensor usage fusion module 56). In some embodiments, the sensor usage indication may also be provided as an indirect signal by triggering the sensor location 20 and the hand-to-sensor proximity determination 21 at the start of sensor usage, i.e., the output of the hand-to-sensor proximity 21 triggers the sensor fusion 56. In an embodiment, the sensor usage signal is provided directly as an input to the sensor fusion module 56, and the sensor location 20 and the hand-to-sensor proximity determination 21 may be processed in the background and provide their inputs independently.

[0080] In some embodiments, the sensor fusion module 56 is configured to maintain the seat occupancy state as long as the seat occupancy classification module 54 does not detect a change in state. When the vehicle ignition button 4 is pressed to start the vehicle, the output of the sensor fusion module 56 is checked and combined with the detection sensor output to determine whether to allow the vehicle to be started.

[0081] Fig.11 Provides close-ups of sub-functions to achieve Fig. 9 Optional facial verification 60 of the embodiment of the present invention. Facial verification 60 is based on determining whether the same face operates the sensor 2 and is associated with the driver. To this end, facial verification 60 utilizes image processing techniques to detect (locate) and track 61 one or more faces in one or more images captured by the cabin camera 3. The located one or more faces can be tracked in consecutive image frames (e.g., subsequent images captured by the cabin camera 3). In some embodiments, the facial detection and tracking module 61 utilizes a neural network model configured to detect a bounding box of a face using, for example, an object detection model architecture.

[0082] The facial verification 60 also utilizes a facial recognition mechanism to identify one or more faces detected at 61. Two facial recognition processes 62a, 62b can run in parallel. The first facial recognition instance (driver facial recognition 62a) can identify a face associated with the driver, that is, the face of a person occupying the driver's seat and having pressed the vehicle ignition button 4. As a result of the driver facial recognition 62a, a driver facial identifier (face_id) can be determined at 63. The second facial recognition instance (sensor facial recognition 62b) can identify a face associated with a person operating the sensor 2 independently of the driver facial recognition 62a. As a result of the sensor facial recognition 62b, a sensor facial identifier can be determined at 64. The facial comparison 65 then inputs the two facial identifiers, the driver facial identifier and the sensor facial identifier, and compares the two facial identifiers. In response to determining that the two facial identifiers are the same, the facial verification 60 determines that the sensor 2 is operated by the driver. In response to determining that the two facial identifiers are different, the facial verification 60 determines that the sensor 2 is operated by another person other than the driver.

[0083] The facial identifier may already be defined in the vehicle's memory for other purposes. Facial recognition uses the biometric recognition of the face of a person in the vehicle. With the help of facial recognition, a unique identifier can be associated with the person and the biometric features of this person. This makes it possible, for example, to adapt the features of the vehicle to the person individually, such as moving the driver's seat to the correct position, loading the driver's profile onto a display in the vehicle, i.e. onto the vehicle's infotainment system, etc.

[0084] Due to different execution triggers, the two facial recognition instances (driver facial recognition 62a, sensor facial recognition 62b) may not necessarily run at the same time. In some embodiments, the first facial recognition instance (driver facial recognition 62a) is triggered by the ignition button press signal 40 (see Fig. 9 ,Right now, Fig. 9 The arrow from box 40 to box 60 can be entered into Fig.11In some embodiments, the second facial recognition instance (sensor facial recognition 62b) is triggered by the sensor usage start signal 30 (see Fig. 9 ,Right now, Fig. 9 The arrow from box 30 to box 60 can be entered into Fig.11 Since the two events of operation of the ignition button 4 and operation of the sensor usually occur at different times (for example, the driver may first operate the sensor 2 and then press the ignition button 4), the facial recognition instances (driver facial recognition 62a, sensor facial recognition 62b) may be activated at different (or overlapping) times and may also be based on different images or image frame sequences received from the cabin camera 3.

[0085] Fig.12 and Fig.13 An implementation example of a system for determining whether the driver operates the sensor 2 during vehicle travel is shown. The entire system described here can be implemented in different variations. In some embodiments, the driver's physical condition verification is performed when the vehicle is started, for example in response to detecting a vehicle start signal such as the operation of the ignition button 4. In some embodiments, the driver's physical condition verification is performed when the vehicle is started and also during vehicle travel (e.g., in periodic time intervals).

[0086] In order to implement an optional second stage of driver physical condition determination during vehicle travel, the interior sensing camera system can be configured to periodically provide cabin images for image-based analysis. In general, the driver can also be prompted to operate sensor 2 from time to time during vehicle travel, for example at given periodic time intervals (e.g. every hour). Sensor operation is possible without the vehicle being stopped, for example by a finger sensor. In some embodiments, sensor operation during vehicle travel may also require a temporary vehicle stop, however without stopping the vehicle's engine. The verification mechanism for determining whether sensor 2 is operated by the driver during vehicle travel can generally correspond to the mechanism used before the vehicle is started, as described above with reference to Figures 1 to 11 The descriptions include, for example, assigning at least one hand to the driver, positioning the sensor, checking the proximity between the driver's hand and the sensor 2 , etc. Therefore, for the sake of brevity, these details will not be repeated.

[0087] Image-based analysis during vehicle travel can detect signs of a driver's deteriorating physical condition, such as intoxication, including eye state (blinking, gaze dynamics, pupil diameter, etc.), body movement, driver activity during vehicle ride (e.g., drinking or smoking detection), object detection of objects located near the driver (e.g., bottles, joints), and / or facial expressions. In some embodiments, alternatively or additionally, vehicle driving behavior is analyzed to assess the driver's physical condition. Driving behavior includes, for example, lane position and related dynamics (e.g., zigzag driving), steering wheel dynamics, acceleration / deceleration dynamics, etc.

[0088] During vehicle travel, both analyses (driver characteristics and driving characteristics) can be performed independently at different time intervals or in time synchronization. Fig.12 In , both analyses are groups in the behavior-based intoxication detection 70. The driving characteristics analysis may be based on current vehicle data 6 (speed, acceleration, etc.) and data provided by the vehicle's advanced driver assistance system (ADAS) 5, such as lane position data. Fig.13 Further details are given showing the three main components of behavior-based intoxication detection 70 .

[0089] Based on the images provided by the cabin camera 3, the driver's condition and the cabin environment around the driver are analyzed 71 using image processing techniques. On a stream of multiple images, the analysis 71 can, for example, determine that the driver regularly drinks from a bottle next to him / her. Image recognition mechanisms can determine indications of drunkenness based on the driver's eye state or facial expressions.

[0090] ADAS information analysis 72 may input data from the vehicle's ADAS 5 to analyze current vehicle control information such as driving lane accuracy. Regular failure to maintain driving lane may be an indication of unqualified driver physical condition. Vehicle data analysis 74 may input current vehicle data 6 such as speed or acceleration. Sudden speed changes may be an indication of unqualified driver physical condition.

[0091] The outputs of the three components 71, 72, 74 may be combined (fused) 75 to provide an aggregate output of behavior-based intoxication detection 70. The aggregate output may indicate possible intoxication or indicate non-intoxication ( Fig.12 ).

[0092] Back to Fig.12 , such as a trigger event of a periodic time trigger 43, which can prompt the driver to operate the sensor 2. The start of the sensor operation can output a signal 30 (as described above, for example, with reference to Figure 8 and Fig. 9The sensor use start signal 30 can trigger the hand verification function 50, which corresponds to the hand verification function used before the vehicle is started ( Fig. 9 and Fig.10 ). Hand verification 50 outputs whether sensor 2 was actually operated by the driver before. The result of sensor operation 41 and optional behavior-based intoxication detection 70 can indicate that the driver's physical condition is suitable for driving ("not intoxicated"). If all three inputs to the combination / fusion module 42 are positive ("true"), the vehicle 32 is allowed to continue driving. Otherwise, if the sensor result 41 or behavior-based intoxication detection 70 indicates a lack of driver physical condition or hand verification 50 indicates that sensor 2 was not operated by the driver ("false"), the control signal is implemented.

[0093] In some embodiments, the control signal implements a safe stop 45 of the vehicle. In the case where the sensor operation is performed during a temporary vehicle stop, the safe stop 45 can be implemented by shutting down the vehicle engine. In the case where the vehicle is moving, the safe stop 45 can be implemented by gradually slowing the vehicle and notifying the driver about the vehicle stop, for example, by an audio signal and / or a visual signal. In some embodiments, instead of a safe stop, the control signal can be a (visual or audio) warning signal output to the driver. In some embodiments, the safe stop can be a cascade of warning messages to notify the driver that a stop is required. If the warning message is ignored, the system can gradually reduce the vehicle speed, or in a highly automated vehicle completely take over control of the vehicle operation and stop on the side of the road. In some embodiments, the control signal may also include a signal to an entity external to the vehicle, such as a police officer or a road monitoring system.

[0094] In addition, the method may be characterized by the following additional devices and improvements: the cabin sensing system can be used to detect how many people are in the vehicle cabin. Fig.14 ) may include multiple cameras and optionally sensors (e.g., weight sensors, seat belt monitoring) for determining, for example, whether a seat is occupied. In response to determining that only one person is in the vehicle cabin, the results of the sensor analysis may be directly linked to the driver. Then, if the in-cabin sensing system determines that there are multiple people in or near the vehicle cabin (e.g., standing at the driver's door and therefore visible in the image of the cabin camera 3), the hand estimation function of the present system is utilized.

[0095] A similar process may be performed if the in-cabin sensing system determines that there are multiple people in the vehicle cabin, but only one is an adult and the rest are children. In some embodiments, additional child-related configurations may be applied depending on whether a child is detected in the vehicle. For example, the threshold of sensor 2 may be adjusted based on the number and age of people detected in the vehicle. For example, if a driver is detected alone in the vehicle cabin, a higher sensor threshold may be applied, i.e., the requirement for the driver's physical condition to operate the vehicle is more relaxed (e.g., a blood alcohol level threshold of 0.05%). However, if a child is detected in the vehicle cabin, a lower sensor threshold may be applied to increase the level of safety, i.e., the requirement for the driver's physical condition to operate the vehicle is set more stringent (e.g., a blood alcohol level threshold of 0.01%) to allow vehicle operation. In some embodiments, the presence of a child in the vehicle cabin may also affect the warning / control strategy ( Figure 6 , box 23; Fig.12 , box 45), for example, if one or more children are detected in the vehicle cabin, a stronger warning or control action is applied, or the driver is not given the option to manually override the sensor results.

[0096] In addition, the system may feature an override module or confirmation module that allows manual input. For example, if the in-cabin sensing system determines that there are multiple adults in the vehicle cabin, and the aforementioned function determines that sensor 2 is not operated by the driver and / or the driver is not in a physical condition suitable for driving, then a person in the vehicle may be enabled to manually confirm that the sensor is in fact operated by the driver and / or the driver is in an appropriate physical condition, such as not intoxicated. Similarly, during vehicle travel as described above, blocks of the vehicle startup or control system may be overridden by such manual confirmation. The system may record such manual confirmation for documentation purposes and / or neural network learning purposes.

[0097] Fig.14 A vehicle interior is shown with possible locations of a camera and a primary display according to embodiments described herein, the camera may be used to capture one or more images for controlling one or more vehicle safety measures.

[0098] The vehicle may include one camera, but may also include multiple cameras located at different locations. The camera may be a color or monochrome camera, an infrared camera, a depth camera, a thermal imaging camera, or a combination thereof, which may be arranged, for example, in the middle of the front windshield, or on the rearview mirror, as shown in position 91. Additionally or alternatively, the camera may be located below the rearview mirror, as shown in position 92. If one camera is located at position 93, the other camera is typically also located at position 94, but this is not mandatory. With two cameras, a depth image or a 3D image may be generated. Additionally or alternatively, the camera may be located in the dashboard or center console, as shown in position 95. Each of positions 91 to 95 may also include two co-located cameras for achieving a 3D view of the interior of the vehicle.

[0099] The camera may capture images, for example at regular time intervals or triggered by an application that needs to determine the height or age of a passenger as described herein. The application using the images may be executed on the vehicle computing system, or at least partially remotely, for example in the cloud. The results of the application may not only output control signals to safety measures, but may also trigger a display on the vehicle's main display 96 in the center console. The vehicle's main display may also be located in another location, such as on the dashboard behind the steering wheel.

[0100] As described above, the mechanisms described herein such as object detection, body key point determination, finger assignment, facial analysis and other image processing techniques can utilize machine learning methods such as one or more neural networks. The machine learning method is trained on example images representing different classes such as object classes, hand gestures, body gestures, facial expressions, etc. The machine learning method can be characterized by one or more of a gradient boosting tree, a random forest, an artificial neural network, a recursive neural network, a convolutional neural network, an autoencoder, a deep learning architecture, a support vector machine, a data-driven trainable regression model, a k nearest neighbor classifier, a physical model and / or a decision tree or a combination thereof. The machine learning method can be remotely trained before the vehicle is used and applied during the use of the vehicle. The machine learning method can also be constrained during the use of the vehicle. The use of the vehicle should be understood as anything that activates the power supply in the vehicle, i.e., unlocking the door, approaching the vehicle with the key / token of the keyless entry system, driving the vehicle, etc.

[0101] Fig.151 is a diagram of the internal components of a computing system 100 that implements the functionality described herein. The computing system 100 may be located in a vehicle and includes at least one processor 101, a user interface 102, a network interface 103, and a main memory 106 that communicate with each other via a bus 105. Optionally, the computing system 100 may also include a static memory 107 and a disk drive unit (not shown), which also communicate with each other via the bus 105. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of the user interface 102.

[0102] In addition, the computing system 100 may also include a designated camera interface 104 for communicating with a vehicle's onboard camera. Alternatively, the computing system 100 may communicate with the camera via a network interface 103. The camera is used to capture images. The computing system 100 may also be connected to a database system (not shown) via a network interface, wherein the database system stores at least a subset of images for providing the functions described herein.

[0103] The main memory 106 can be a random access memory (RAM) and / or any other volatile memory. The main memory 106 can store program codes that implement the above-mentioned hand verification module 50, facial verification module 60, and behavior-based physical condition analysis 70, and the program code implements the method described herein. Other modules for other functions described herein can be stored in the main memory 106. The main memory 106 can also store additional program data for providing the functions described herein. A portion of the program code and program data can also be stored in a separate, for example, cloud storage, and at least partially executed remotely. In some embodiments, the memory 106 can store data such as configurable thresholds for proximity determination described herein in the cache 111.

[0104] According to one aspect, a vehicle is provided. The methods described herein may be stored as program code or modules 50, 60 or 70 and may be at least partially included in a vehicle. Portions of the program code 50, 60, 70 may also be stored and executed on a cloud server to reduce the computational workload on the computing system 100 of the vehicle. The vehicle may also include one or more cameras 3, for example connected via a camera interface 104, for capturing one or more images.

[0105] According to one aspect, a computer program including instructions is provided. When a computer executes the program, these instructions cause the computer to perform the methods described herein. The program code embodied in any system described herein can be distributed individually or collectively as a program product in a variety of different forms. Specifically, the program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, and the computer-readable program instructions are used to cause a processor to perform various aspects of the embodiments described herein.

[0106] Computer-readable storage media, which are non-transitory in nature, may include volatile and nonvolatile, and removable and non-removable tangible media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may also include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be read by a computer.

[0107] The computer-readable storage medium itself should not be interpreted as a transient signal (e.g., a radio wave or other propagating electromagnetic wave, an electromagnetic wave propagating through a transmission medium such as a waveguide, or an electrical signal transmitted through a wire). Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded to an external computer or external storage device via a network.

[0108] It should be understood that although specific embodiments and variations are described herein, further modifications and substitutions are obvious to those skilled in the relevant art.In particular, examples are provided by illustrating principles, and a variety of specific methods and arrangements for implementing these principles are provided.

[0109] In some embodiments, the functions and / or actions specified in the flow chart, sequence diagram and / or block diagram can be reordered, processed in series and / or processed concurrently without departing from the scope of the present disclosure. In addition, any flow chart, sequence diagram and / or block diagram may include more or less blocks than those shown in the embodiments of the present disclosure.

[0110] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present disclosure. It should also be understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of the features, elements, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, elements, steps, operations, elements, components and / or combinations thereof. In addition, to the extent that the terms "include", "have", "have", "have", "consist of" or variations thereof are used in the detailed description or claims, these terms are intended to be included in a manner similar to the term "comprise".

[0111] Although the description of various embodiments has illustrated the method, and although these embodiments have been described in considerable detail, it is not the intention of the applicant to restrict or in any way limit the scope of the appended claims to such details. Additional advantages and modifications will be apparent to those skilled in the art. Therefore, the present disclosure in its broader aspects is not limited to the specific details, representative devices and methods, and illustrative examples shown and described. Therefore, the described embodiments should be understood as provided as examples for the purpose of teaching general features and principles, and should not be understood as limiting the scope as defined in the appended claims.

Claims

1. A computer-implemented method of determining a physical condition of a driver of a vehicle, the method comprising the steps of: receiving at least one image captured by a vehicle camera, the image showing at least a portion of the driver and a sensor to be operated by the driver to determine the physical condition; as well as Based on the image, it is determined whether the sensor is operated by the driver.

2. The method according to claim 1, wherein: Determining that the sensor is operated by the driver includes: identifying the driver among a plurality of persons in the vehicle, including: categorizing activities of the plurality of persons; and The detected body part is assigned to a corresponding one of the plurality of persons on the vehicle seat and a unique identifier is provided for the corresponding one of the plurality of persons.

3. The method according to claim 1 or 2, wherein: Determining that the sensor is operated by the driver includes: locating and tracking at least one hand of the driver; and It is determined that the sensor is operated by the driver's hand.

4. The method according to claim 3, wherein: Locating and tracking at least one hand of the driver includes based on the at least one image: tracking a body shown in the image; tracking a hand shown in the image; as well as Based on the tracked body and hands, the at least one finger is assigned to the driver.

5. The method according to claim 3 or 4, wherein: Determining that the sensor is operated by the driver's hand includes based on the at least one image: positioning the sensor; and A proximity of the driver's hand to the sensor is determined.

6. The method according to any one of claims 1 to 5, further comprising the following steps: It is determined that the sensor is operated by someone other than the driver.

7. The method according to claim 6, further comprising the steps of: Generate warning signals; and / or triggering a control action to prevent or stop the operation of the vehicle, wherein the control action comprises: achieving a safe stop of the vehicle; and / or The driver is prompted to confirm current physical driving ability.

8. The method according to any one of claims 1 to 7, wherein: Determining the driver's physical condition further includes: Based on the at least one image, at least one of the driver's eye state, the driver's body movement, the driver's hand operation, object detection near the driver, and the driver's facial expression is determined.

9. The method according to claim 8, wherein: The driver's body movement includes drinking movement and / or smoking movement, and The object detection in the vicinity of the driver may include detecting containers that may contain liquids and / or detecting smoking products.

10. The method according to claim 8 or 9, wherein: Determining the physical condition of the driver further includes: Driving behavior is analyzed, the driving behavior comprising at least one of lane position, winding driving, steering wheel dynamics, and acceleration / deceleration dynamics.

11. The method according to any one of claims 1 to 10, the method being performed before the vehicle is started and comprising: The vehicle is prevented from being operated until the driver's use of the sensor has been confirmed and the sensor output has confirmed that the physical condition of the driver satisfies given conditions for operating the vehicle.

12. The method according to any one of claims 1 to 11, the method being performed during operation of the vehicle and comprising: In response to determining that the sensor is operated by the driver and the sensor output indicates that the physical condition of the driver is not suitable for driving, a warning signal is output.

13. A system for controlling one or more vehicle safety measures, the system implementing the method according to any one of claims 1 to 12.

14. A vehicle, comprising: a camera for capturing one or more images; as well as The system for controlling one or more vehicle safety measures of claim 13.

15. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Images