Vehicle security monitoring

Abstract

The invention relates to vehicle safety monitoring. A computerized method of in-vehicle safety monitoring is presented, the method comprising the steps of: determining a driver-related state based on data from one or more sensors of a vehicle, where at least one sensor captures a cabin of the vehicle; comparing the driver-related state with one or more predetermined rules; and in response to detecting a violation of a rule of the one or more rules, triggering a warning event.

Description

Technical Field

[0001] The present disclosure relates generally to vehicle safety improvements, and particularly to methods and systems for vehicle safety monitoring. Background Art

[0002] Smart vehicles such as smart cars, smart buses, etc. are on the way to significantly improve passenger safety. Such smart vehicles may be equipped with onboard cameras and are able to capture images of the vehicle interior. These images (sometimes in combination with other sensors) can then be used for different safety-related tasks.

[0003] In a shared or rented vehicle or when a young driver is using the vehicle (i.e., in situations where the driver must follow restrictions), vehicle safety monitoring is important not only for the driver, but also for other entities such as the car sharing or rental company or the young driver's parents. Current vehicle safety monitoring systems focus on monitoring vehicle operating conditions, such as speed or geographic location, i.e., general vehicle status. Although onboard cameras can identify the driver to determine the rules for the driver to follow, current systems do not monitor the entire cabin condition and cannot determine comprehensive driver-related status.

[0004] Therefore, there is a need to provide an effective vehicle safety monitoring system that focuses on considering cabin conditions to determine comprehensive driver-related states. Summary of the invention

[0005] In this context, a computerized method for vehicle safety monitoring, a vehicle safety monitoring system and a computer program product are proposed.

[0006] More specifically, a computerized method for vehicle safety monitoring is proposed. The computerized method includes: determining a driver-related state based on data from one or more sensors of the vehicle, wherein at least one sensor photographs a cabin of the vehicle; comparing the driver-related state to one or more predetermined rules; and in response to detecting a violation of a rule in the one or more rules, triggering a warning event.

[0007] In some embodiments, a sensor among the one or more sensors is a vehicle-mounted camera, and determining the driver-related state includes obtaining image data from the vehicle-mounted camera, determining at least one of a seating posture of a driver of the vehicle, interactions of the driver with people and / or objects in the vehicle, and types of objects placed in the cabin based on the image data; and determining the driver-related state based on at least one of a seating posture of the driver of the vehicle, interactions of the driver with people and / or objects in the vehicle, and types of objects placed in the cabin.

[0008] In other embodiments, determining the driver-related status also includes determining at least one of the number of people in the cabin, the age of one or more individuals, the activity of one or more individuals, the seat belt status of one or more individuals, and alcohol use and / or intoxication of one or more individuals, and determining the driver-related status based on at least one of the number of people in the cabin, the age of one or more individuals, the activity of one or more individuals, the seat belt status of one or more individuals, and alcohol use and / or intoxication of one or more individuals.

[0009] In some embodiments, determining the driver related state is further based on additional vehicle data, wherein the additional vehicle data includes input from further vehicle sensors, the state of an advanced driver assistance system and / or the state of an in-vehicle infotainment system.

[0010] In further embodiments, the one or more rules relate to at least one of the presence of a co-pilot, a maximum number of people in the vehicle, phone use while driving, a maximum speed limit, areas where driving is permitted, and time periods where driving is permitted.

[0011] In some embodiments, the warning event is at least one of an email sent to an email address associated with the vehicle, a message sent to an application connected to the vehicle, a warning sound output from an onboard speaker, a warning light at the vehicle dashboard, and a warning message displayed on the vehicle's infotainment system.

[0012] In further embodiments, the method further comprises determining a warning level associated with a violation of a rule of the one or more rules, wherein the warning level defines a warning event to be triggered.

[0013] In still further embodiments, the method further includes interacting with one or more vehicle driving control systems in response to a violation of the one or more rules.

[0014] On the other hand, a vehicle safety monitoring system is provided, comprising: one or more sensors, wherein at least one sensor is configured to photograph a cabin of a vehicle; a vehicle monitoring computing unit, wherein the vehicle monitoring computing unit is configured to determine a driver-related state based on data from one or more sensors of the vehicle and compare the driver-related state with one or more predetermined rules; and a communication unit, wherein the communication unit is configured to trigger a warning event in response to detecting a violation of a rule of the one or more rules.

[0015] In some embodiments, the cabin sensor of the one or more cabin sensors is an on-board camera, and the vehicle monitoring computing unit is further configured to obtain image data from the on-board camera in order to determine the driver-related state, determine at least one of a seating posture of the driver of the vehicle, the driver's interaction with people and / or objects in the vehicle, and the types of objects placed in the cabin based on the image data, and determine the driver-related state based on at least one of the seating posture of the driver of the vehicle, the driver's interaction with people and / or objects in the vehicle, and the types of objects placed in the cabin.

[0016] In another embodiment, the vehicle monitoring computing unit is further configured to determine the driver-related state by determining at least one of the number of persons in the cabin, the age of one or more persons, the activity of one or more persons, the seat belt status of one or more persons, and the alcohol use and / or intoxication of one or more persons, and further determine the driver-related state based on at least one of the number of persons in the cabin, the age of one or more persons, the activity of one or more persons, the seat belt status of one or more persons, and the alcohol use and / or intoxication of one or more persons.

[0017] In some embodiments, the vehicle monitoring computing unit is further configured to determine a driver-related state based on additional vehicle data, wherein the additional vehicle data includes input from further vehicle sensors, a state of an advanced driver assistance system, and / or a state of an in-vehicle infotainment system.

[0018] In further embodiments, the one or more rules relate to at least one of the presence of a co-pilot, a maximum number of people in the vehicle, phone use while driving, a maximum speed limit, areas where driving is permitted, and time periods where driving is permitted.

[0019] In some embodiments, the warning event is at least one of an email sent to an email address associated with the vehicle, a message sent to an application connected to the vehicle, a warning sound output from an onboard speaker, a warning light at the vehicle's dashboard, and a warning message displayed on the vehicle's infotainment system.

[0020] In further embodiments, the communication unit is further configured to determine a warning level associated with violating a rule of the one or more rules, wherein the warning level defines a warning event to be triggered.

[0021] In further embodiments, the system is further configured to interact with one or more vehicle driving control systems in response to a violation of one or more rules.

[0022] Another aspect relates to a vehicle comprising the vehicle safety monitoring system described herein.

[0023] A final aspect relates to a computer program product comprising instructions which, when executed on a computer, cause the computer to perform a method as described herein.

[0024] These and other objects, embodiments and advantages will become apparent to those skilled in the art from the following detailed description of the embodiments with reference to the accompanying drawings, the present disclosure not being limited to any particular embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The foregoing and further objects, features, and advantages of the present subject matter will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings, in which like reference numerals are used to represent like elements, and in which:

[0026] Figure 1 A high-level overview of the implementation of vehicle safety monitoring.

[0027] Figure 2 How the driver-related state is determined in an embodiment is further detailed.

[0028] Figure 3 and Figure 4 An image from an on-board camera is shown of circumstances relevant to determining a driver-related state.

[0029] Figure 5 It is a further improvement on how to determine the driver related state in an embodiment.

[0030] Figure 6 An automobile is shown having sensors and systems for providing additional data for determining a driver-related state.

[0031] Figure 7 Technical modules supporting the vehicle safety monitoring method according to the embodiment are depicted.

[0032] Figure 8 Another embodiment shows the calculation of the safe driving score.

[0033] Fig. 9 The interior of a vehicle is presented with possible locations of onboard cameras.

[0034] Fig.10 is a diagram of a computing system that implements the functionality described herein. DETAILED DESCRIPTION

[0035] The present disclosure relates to a method and system for vehicle safety monitoring. Vehicle safety monitoring is related to several situations.

[0036] For example, drivers who have just obtained their driver's license (especially young drivers, such as between 16 and 20 years old) may have a special need for driver monitoring systems. In some countries, there are even legal requirements for young drivers, for example, young drivers are only allowed to operate the car when there is an adult in the passenger seat, or the maximum number of passengers is restricted. Parents of young drivers may also be particularly concerned about the safety of their children and want to know how their children are driving. Cabin monitoring technology can provide some special monitoring functions for parents and support young drivers in the cabin.

[0037] As another example, car sharing or car rental services can also seek to monitor driver behavior. Typically, young drivers are not allowed to rent cars. If a rental company wants to allow young drivers to rent cars, they can define some rules that the driver must follow, such as no interstate driving, no smoking / drinking in the car, no exceeding the speed limit, no more than two passengers in the car, etc. In this approach, cabin monitoring technology can support the rental company.

[0038] Sensors in the vehicle can also be used to analyze the energy efficiency of a given driver's driving and calculate an energy efficiency score. For example, frequent acceleration / deceleration can reduce this score, and predictive driving with small speed changes can increase the score. In addition, a safe driving score can be calculated to rate the driver's driving behavior based on safety. All of these methods can support monitoring of the driver.

[0039] Before commencing the detailed description, some general definitions and explanations are given below. Vehicles herein are also referred to as smart vehicles, smart cars or cars. Vehicle safety monitoring by driver-related states according to the present disclosure involves monitoring the cabin state of the vehicle and optionally / additionally monitoring the operating state of the vehicle. These two states can form a driver-related state according to the present disclosure, which means that the driver is (usually) responsible for the state of the cabin and the operating parameters of the vehicle. In addition, in vehicle safety monitoring, each driver defines rules, that is, these rules can also be considered as driver-related rules. The driver-related state is compared with the rules to determine whether the rules are followed. If not, an event is triggered, such as notifying the owner or operator of the vehicle, the driver's parents, etc. of the violation of the rules. The details of the vehicle safety monitoring method and system are described below with reference to the accompanying drawings.

[0040] Figure 1is a high-level overview of the implementation of vehicle safety monitoring. The method starts at box 11 (performed by a vehicle monitoring computing unit in some embodiments), where a driver-related state is determined based on data from one or more sensors of the vehicle. At least one of the one or more sensors captures the cabin of the vehicle. The sensor that captures the cabin of the vehicle may be an onboard camera or other sensor, such as a radar sensor. The onboard camera may include a color, monochrome or grayscale camera, an infrared camera, a depth camera, a thermal camera, an event camera (i.e., a camera that operates at a very high frame rate and outputs only frame differences (e.g., motion)), or a combination of the above cameras. Other sensors may be velocity / speed sensors, radar sensors for detecting objects on the way of the vehicle, different engine sensors, GPS sensors, parking sensors, weight sensors on seats, seat belt sensors, door opening sensors, microphones, etc. Other cabin or non-cabin sensors may take the form of, for example, radar, lidar or ultrasonic sensors or a combination thereof, or cameras similar to onboard cameras.

[0041] Driver-related states are states that are related to the driver, but in contrast to the vehicle monitoring systems of the prior art, driver-related states not only focus on the driver and the driver's driving behavior, but also generally consider the condition of the entire cabin. For more completeness, the driver-related states include at least the state of the cabin, but may also include vehicle operating states, such as the speed of the vehicle, the geographic location of the vehicle, the distance to other traffic participants, etc. The state of the cabin or cabin state may involve many different aspects and may reflect the overall situation in the cabin. For example, the cabin state may include at least one of how many people have been determined to be in the vehicle, the seating posture of the passengers, the age of the passengers, the activities of the driver and passengers (such as the use of the phone, talking, eating, drinking, posture, hands on the steering wheel, etc.), the detection and classification of objects on the seat, the state of the seat belt, the alcohol use and / or drunkenness of one or more individuals, etc. Compared with the prior art that only focuses on the driver's situation, the use of information related to the cabin state as a whole (e.g., the number of passengers considered, the interaction of the passengers, the objects in the vehicle, the posture of the driver and other passengers, etc.) improves vehicle safety monitoring.

[0042] Determining the driver-related state can be further based on additional vehicle data, wherein the additional vehicle data includes inputs from additional vehicle sensors, the state of an advanced driver assistance system, and / or the state of an in-vehicle infotainment system. For example, the volume and type of music played by the in-vehicle infotainment system can be considered to determine the driver-related state. Additionally or alternatively, the state of the advanced driver assistance system, such as whether the car is in heavy traffic, the distance to other cars, cyclists, pedestrians, etc., can be considered. Different implementations are obvious to those skilled in the art. For example, if the speed of the vehicle is considered to determine the driver-related state, the speed can be determined directly from data from a wheel speed sensor (e.g., located in the wheel as part of an anti-lock braking system (ABS) or an electronic stability program (ESP) control system, where its data is received directly from the vehicle message bus), but can also be received from an advanced driver assistance system. The advanced driver assistance system can additionally provide information about the speed of other traffic participants around the vehicle, information about the permitted speed limit, etc. Alternatively, if the music volume is to be considered to determine the driver-related state, the volume can be determined directly from a microphone in the cabin or received from the in-vehicle infotainment system. The microphone can also sense other audio or noise sources that the infotainment system is not aware of, such as music playing from a Bluetooth speaker or smartphone in the cabin, or from audio sources outside the vehicle.

[0043] Determining driver-related states based on sensor data can be achieved by applying several methods, such as traditional algorithms and / or machine learning methods for object detection and classification, for seat posture determination, for determination and classification of interactions, for seat belt position verification, etc. Before these methods can be applied, the raw sensor signals can be preprocessed. Preprocessing of raw data can involve pixel processing, such as calculating an RGB color image or a monochrome image from a raw pixel grid with a specific color filter array (CFA) (e.g., red, green, blue and / or IR pixels). Preprocessing can also include image enhancement (e.g., denoising, sharpening, black level correction, lens shading correction, white balance, etc.), image cropping or resizing, distortion correction, motion detection (e.g., optical flow), or a combination thereof.

[0044] For example, a driver-related state determined based on, for example, preprocessed sensor signals (such as camera images) may be characterized by: one driver and three passengers in the vehicle, a large item on a seat (not secured to the seat), and 62 s of driver-distracting interactions detected in the last 300 s.

[0045] In this example, the number of passengers and the items on the seat may have been detected with a bounding box object detection algorithm. The information that the item is not fixed on the seat can be determined by an image analysis algorithm that is trained to detect the route of the seat belt and / or the movement of the object. By tracking the bounding box position of the object and (alternatively or additionally) the unique feature points on the object over time according to the displacement in the image coordinates, the movement of the object can be determined. Distracting interactions can be determined by a machine learning algorithm that is trained to classify actions based on the movement of the passenger and / or based on the state of the infotainment system (e.g., the volume and type of music played). Distracting interactions can be one or more of the following: the driver often looks at other passengers, the driver often touches the front passenger seat, the driver often turns to the back seat, leans to the side, leans forward, the driver often interacts with children (or child seats) by approaching or looking at the direction with his hands, other hand interactions, violence, chaos and / or fighting between people, loud and aggressive music, and strong negative or positive emotions (observing people's very happy / laughing, very unhappy / panic emotions).

[0046] Determining the driver-related state may also include a post-processing procedure after the current condition of the cabin / vehicle has been determined (e.g., after computer vision and machine learning algorithms have been applied to determine information about passengers in the vehicle (number, age, posture, etc.) and / or information about objects in the vehicle (type of object, location of object, etc.), or generally after sensor data has been collected (and possibly also pre-processed). Post-processing may be achieved by filtering input signals (e.g., low-pass filtering, band-pass filtering, Kalman filtering, etc.), input signal validation / authenticity checks (e.g., range checks, relationships between two or more signals, etc.), transfer functions (e.g., signal range modification, normalization, etc.), state machine updates (e.g., hidden Markov models, discrete state machines, etc.), and signal fusion (e.g., combining multiple input signals to obtain additional processed signals).

[0047] The method is carried out in frame 12 (also performed by the vehicle monitoring computing unit in some embodiments), and the driver-related state is compared with one or more predetermined rules. The predetermined rule should be understood as defining the rule before comparing the rule with the driver-associated state. The rule can be predefined before the driver enters the vehicle, or it can be predefined in real time (on the fly) by remotely accessing the computing system of the vehicle. For example, a parent can access the computing system of the vehicle by using an application on her / his phone that connects it to the vehicle. The parent can define the parameters of the preconfigured rules or define the rules from scratch. One or more rules can relate to at least one of the presence of a co-pilot, the maximum number of people in the vehicle, the use of the phone while driving, the maximum speed limit, the given geographic area where driving is allowed, and the time period where driving is allowed.

[0048] For example, pre-configured rules that can be defined with parameters and provided by the application may include rules for at least one of the following: the maximum number of passengers allowed in the car (the parameter can be a number), the presence of at least one adult in the cabin (the parameter can define which adult or where the adult needs to sit), the maximum phone usage time allowed while driving (the parameter can be a time in seconds, the default value can be 0 seconds), the maximum phone usage time allowed when stationary in traffic (the parameter can be a time in seconds, the default value can be 0 seconds), the maximum allowed time for eyes to be off the road when driving at a speed greater than a threshold (the parameters can be time and speed), eating or drinking (the parameters can be "allowed" or "forbidden"), not allowing certain objects in the vehicle, such as weapons (the parameter can be selected from a predetermined list of detectable objects), not allowing dangerous objects on the seats, for example, large items that are observed to be moved or classified as moveable (the parameter can be "allowed" or "forbidden"), not allowing distracting interactions with other passengers longer than a configurable time (the parameter can be a time), and only allowing specific people in the cabin through facial recognition (the parameter can be selected from a predetermined list of people). Additional rules may relate to the maximum allowed speed, the maximum allowable amount of exceeding the current speed limit, the maximum allowed time of exceeding the speed limit, geo-fencing (i.e., allowed driving areas, times of the day and / or week when driving is allowed, and violations of traffic signs and red lights. Additional rules may include wearing or not wearing a mask as required (the parameter may be "yes" or "no"); for example, a rule may define that taxi drivers / passengers in a taxi must wear a mask to ensure healthy and safe driving, or that the driver is not allowed to wear a mask to ensure identification of the driver.

[0049] In the above example (the driver-related state is characterized by: there is a driver and three passengers in the vehicle, there is a large item on the seat that is not secured to the seat, and there were 62 seconds of driver distraction interactions detected in the last 300 seconds), the rule may include not allowing the driver to carry more than one additional passenger while driving. Therefore, comparing the driver-related state to the rules may result in a determination of a violation of one or more rules.

[0050] In this case (i.e., when a violation of one or more rules is detected), the method continues in box 13 (performed by the vehicle's communication unit in some embodiments) with the triggering of a warning event. The warning event can have several formats. The warning event can be at least one of an email sent to an email address associated with the vehicle (e.g., a parent's email address or an email associated with a car sharing / rental car company), a message sent to an application connected to the vehicle, a warning sound output from an onboard speaker, a warning light at the vehicle's dashboard, and a warning message displayed on the vehicle's infotainment system.

[0051] The application connected to the vehicle can be a mobile App (for parents, etc.) to supervise and / or monitor the young driver. A message can be triggered if the young driver does not have an adult in the front passenger seat when driving the car, if the young driver drives a car with more passengers than allowed, if the young driver uses a phone while driving, if the young driver drives faster than the allowed speed, and many other situations as explained herein. Alternatively, the application can also be a monitoring application for a car sharing company or a car rental company. As explained herein, a message can be triggered if the vehicle leaves the country, if an unregistered driver drives the car, if a dangerous object is placed in the car, etc.

[0052] As described above, the warning event may be different depending on the rule violation. For example, if the driver is distracted for longer than the allowed time and the excess time does not exceed 10% of the allowed time (which may also be defined as a parameter of the rule), a warning signal may be output from the vehicle speaker to notify the driver to focus on driving again. If the driver is distracted for more than 10% longer than the allowed time, a message may be sent to a supervision application associated with the vehicle, thereby indicating to the user, for example in a pop-up message, that the driver has been distracted for too long.

[0053] Which type of warning event can be triggered can also depend on the warning level associated with the violation of the rule. In this case, the method also includes determining a warning level, wherein the warning level defines the warning event to be triggered. For example, a warning level can be defined for each rule, and the warning level can take a value from 0 to 5, wherein 5 represents high risk and requires immediate warning on all channels, and 0 represents that no warning is required. For example, if the driver fights with other people in the car, this may result in a warning level of 5, and may trigger warning events on all possible channels (e.g., emails, applications, phone calls, sound outputs, warning messages, etc.). Violating the eating rule may result in a warning level of 1, and may only trigger a local warning event in the vehicle, reminding the driver of the rule.

[0054] Messages sent to external devices (e.g., emails, application messages, etc.) may also include information from sensors, such as images captured by an onboard camera. Some messages may also open a live stream to the onboard camera, enabling the recipient of the message to connect to a speaker or similar action. In other embodiments, some rule violations may also result in interactions with one or more vehicle driving control systems, resulting in not allowing the vehicle to be started, stopping the vehicle (after a warning and timeout period), limiting the maximum speed to a very low value, turning on lights, etc. Those skilled in the art will be aware of other possible implementations of warning events.

[0055] In addition, the method may also include calculating several scores based on the driver-related state and the information received from the sensor. For example, the energy efficiency of a given driver's driving can be analyzed and the energy efficiency score can be calculated. For example, frequent acceleration / deceleration can reduce this score, and predictive driving with a smaller rate change can increase the score. In addition, a safe driving score can be calculated to assess the driver's driving behavior according to safety. For signals related to safe driving (such as seat belts being correctly worn, normal seating position, both hands on the steering wheel, eyes observing road conditions, scanning surroundings or observing rearview mirrors (based on eye gaze crossing) etc.), the score can be increased, and the score can be reduced by being considered unsafe prompts, such as phone use, distractions caused by other activities outside the driving task (such as eating / drinking / smoking, incorrect use of seat belts, etc.)

[0056] Now turn to Figure 2, further details of how the driver-related state can be determined are shown. In this example, at least one sensor is an onboard camera, from which an image is received in box 21. The image data may be pre-processed as explained herein, but may also be raw image data. Based on the image data, at least one of the seating posture of the driver of the vehicle, the driver's interaction with people and / or objects in the vehicle, and the type of objects placed in the cabin is determined in box 22.

[0057] For example, it may be determined that the driver is leaning toward the co-pilot and frequently interacting with the co-pilot. It may also be determined that a weapon is in the car. In this example, these determinations form a driver-related state. In block 23, a determination of a driver-related state based on at least one of the seating posture of the vehicle driver, the driver's interaction with other passengers and / or objects in the vehicle, and the type of object placed in the cabin is depicted.

[0058] Therefore, in Figure 2 In the example of , at least a plurality of images (e.g., a sequence of a plurality of images) from an onboard camera may be used to determine the driver-related state. Figure 1 13, the state is compared with the rules (such as Figure 1 12) to determine whether a warning event should be triggered.

[0059] Figure 3 and Figure 4 The figures show images from a vehicle-mounted camera in a number of situations that are relevant to determining a driver-related state. Figure 3 In the figure, based on the image data, it can be determined that 5 people are in the car. Driver 31, co-pilot 32 and three people 33, 34 and 35 at the back seat. Person 34 can be determined to distract the attention of driver 31, so that the driver's eyes are away from the road for longer than the allowed time. Therefore, a warning event can be triggered. The personnel in the vehicle can be determined based on a bounding box algorithm. The posture and interaction of people (passengers and drivers) can be determined on body key points and / or bounding boxes (e.g., bounding boxes determined around the head, torso or other body parts) generated based on image data of one or more images. Body key points (e.g., body joints (shoulders, hips, elbows, wrists, knees, ...) or other feature points of the human body (eyes, ears, noses, corners of the mouth, fingertips, ...) can be located by neural networks or computer vision algorithms. Body key points are often used to obtain 2D or 3D skeleton models of people in the vehicle cabin, and for example, sitting postures can be determined from the model. Body key points can also be tracked over time to estimate activities.

[0060] exist Figure 4In the example of FIG. 4 , the driver 41 may be determined as an unauthorized driver, for example, by facial recognition or a fingerprint sensor. An object detection algorithm may detect that a weapon 42 is in the car. Furthermore, it may be determined that the person 43 is drunk based on the motion of the person 43 detected over a sequence of images. Thus, the driver-related state may violate at least three different rules. Warning events in this case may even include messages to a central police messaging account and interactions of, for example, a vehicle monitoring computing unit or a communication unit with a driving control system to stop the car, lock / unlock the doors, turn on the lights, etc.

[0061] Figure 5 is a further improvement of how to determine the driver-related state according to some embodiments. As shown in box 51, determining the driver-related state may also include determining at least one of the number of people in the cabin, the age of one or more people, the activity of one or more people, the seat belt status of one or more people, and the alcohol use and / or intoxication of one or more people. This determination can be done based solely on on-board sensor data (e.g., image data), but inputs from other sensors may be considered in addition or alternatively. For example, the seat belt status may also or alternatively be determined based on a seat belt buckle sensor. Alternatively, intoxication may also or alternatively be determined based on a breathing air sensor.

[0062] In box 52, the method continues to determine the driver-related state based on at least one of the number of people in the cabin, the age of one or more people, the activity of one or more people, the seat belt status of one or more people, and the alcohol use and / or intoxication of one or more people. Obviously, different information of the cabin, different information of on-board sensors sensing the interior of the vehicle, other sensors or information provided by the advanced driver assistance system and / or the on-board infotainment system can be considered to determine the driver-related state. More information can lead to a better estimate of the current condition of the vehicle driven by the driver.

[0063] Figure 6 A car is shown with sensors and systems for providing additional data for determining driver-related status. An onboard camera 61 is depicted as a sensor for sensing the interior of the cabin. Such an onboard camera may have different locations, which will be referred to in detail in the following. Fig. 9 Further description. The vehicle-mounted camera 61 provides raw or pre-processed image data to the vehicle monitoring computing unit 62. Therefore, a pre-processing unit (not shown) can be provided between the vehicle-mounted camera 61 and the vehicle monitoring computing unit. The vehicle monitoring computing unit 62 can also receive other raw or pre-processed data from sensors such as a speed sensor 63, a door sensor 64, a distance sensor 65, or any other sensor located in the vehicle. The pre-processing unit can also pre-process the raw data from other sensors before providing it to the vehicle monitoring computing unit 62.

[0064] In addition, the vehicle monitoring computing unit 62 may receive additional data from an advanced driver assistance system 66 connected to a plurality of sensors such as a speed sensor 63, a front distance sensor 65, a side distance sensor 67, a parking sensor 68 (which can also function as a distance sensor), additional distance sensors (not shown), etc. Those skilled in the art will appreciate that Figure 6 These are merely illustrative, and there may be more or fewer sensors included by the vehicle.

[0065] The advanced driver assistance system 66 generally includes technology to assist the driver in safely operating the vehicle. The advanced driver assistance system 66 uses input from sensors to detect nearby obstacles or driver errors. Therefore, the advanced driver assistance system 66 already maintains a large amount of information that can be provided to the vehicle monitoring computing unit 62 to determine the relevant state of the driver.

[0066] In addition, the vehicle monitoring computing unit 62 may also receive additional data from the in-vehicle infotainment system 69. The in-vehicle infotainment system 69 is a collection of hardware and software in the vehicle that provides audio or video entertainment and additional information (e.g., navigation, etc.). The in-vehicle infotainment system 69 may provide data to the vehicle monitoring computing unit 62, such as the volume and type of music played in the car, the planned route, etc., which may be further used to determine driver-related states.

[0067] Figure 7 The technical modules supporting the vehicle safety monitoring method according to the embodiment are depicted. One or more sensors 71 may be disposed in or at the vehicle. The sensors provide sensor data (e.g., image data, other cabin sensor data, vehicle operation data, etc.) to a computing unit 72, which may be Figure 6 The vehicle monitoring computing unit 62. The computing unit 72 determines the driver-related state based on the data from the one or more sensors 71. The computing unit 72 also compares the driver-related state with one or more predetermined rules. If the computing unit 72 detects a violation of one or more rules, the communication unit 73 triggers a warning event. For example, the computing unit 72 detects a violation of rule #A, and this information can be provided to the communication unit 73. The communication unit 73 then checks which warning event the violation of rule #A should trigger, and then triggers the warning event.

[0068] The warning event may be a message or signal sent to at least one receiving unit 74, which then triggers an output on at least one output device 75. The receiving unit 74 and the output device 75 may be inside or outside the vehicle. For example, the warning event may be a signal transmission to a receiving unit 74 (e.g., an in-vehicle infotainment system) associated with a speaker as an output device 75 in the vehicle. The in-vehicle infotainment system may then initiate output of a warning sound or message at the speaker. In another example, the warning event may be a message sent to an associated application on a mobile phone. The receiving unit 74 may be a computing system of the mobile phone, and the output device 75 may be a display, on which a notification for the violation or a video clip of the violation may be displayed.

[0069] Figure 8 A non-limiting implementation of calculating a safe driving score is shown. The determined driver-related state 81 (which corresponds to the driver-related state according to some embodiments described previously) can also be used to calculate a safe driving score. Therefore, the safe driving parameters 82 can be defined by an administrator (such as a parent, an employer, a car rental company, an insurance company, etc.). The safe driving parameters 82 can be the same as the parameters used to define rules according to other embodiments, or they can be different, or some can be the same and others different. The safe driving parameters 82 can define safe driving rules, which can include the rules described in the previous embodiments as well as additional rules.

[0070] For example, not only critical rules related to triggering warning events as described previously can be defined (e.g., drinking alcohol while operating a vehicle), but also less critical but safety-related rules can be defined, such as whether a seat belt is worn correctly (e.g., by detecting seat belt misuse or incorrect alignment based on cabin images), whether a normal seating position is always maintained (e.g., by determining body key points, bounding boxes, etc. over multiple cabin images), whether both hands are on the steering wheel, whether the eyes are fixed on the road, whether the driver is scanning the surroundings or the rearview mirrors (based on eye gaze intersections), and more.

[0071] Based on the safe driving parameters 82, a safe driving score is calculated in 83. The safe driving score may be calculated for a specified time, for example, for a driving period (i.e., from entering the car to leaving the car), an hour, or another specified time period. If the rules generated by the safe driving parameters 82 are followed, the safe driving score is increased, and if not, if the violation of the rule is critical, the safe driving score is decreased. For example, if phone use, distraction caused by other activities outside the driving task, eating / drinking / smoking, incorrect use of seat belts, etc. are detected in the driver-related state, the safe driving score is reduced.

[0072] The safe driving score is checked against a threshold at 84. For example, if the safe driving score can range between 0 and 10, where 0 means no safe driving and 10 means the driving behavior is safe, the first threshold can be set to 5, which results in a warning 85A, and the second threshold can be set to 7, which results in a reward 85B. For example, if the driver complies with all safety rules most of the time and the safe driving score is calculated to be 9.5 over the past full year, the reward 85B can be a reduction in insurance premiums. For example, if the driver (e.g., a young driver) violates many safety rules and the safe driving score calculated for the most recent driving session is 3.5, a warning 85A can be displayed on the infotainment display after the drive, and a warning message can be sent to the parent's mobile phone, similar to the warning event described above.

[0073] At 86, the safe driving score may additionally or alternatively be recorded to a safe driving score history database 87, which associates the safe driving score with the driver profile. The administrator may monitor the dynamics of the driver's safe driving score via the vehicle's internal user interface or an application (e.g., a mobile app) connected to the safe driving score history database 87. Thus, for example, statistics 88 may be generated based on the administrator's input, and an output 89 of the statistics 88 may be generated for display.

[0074] Fig. 9 The interior of a vehicle with possible camera locations is shown. The vehicle may include only one camera or multiple cameras at different locations. The camera (which may be a color, monochrome or grayscale camera, an infrared camera, a depth camera, a thermal camera, an event camera (i.e., a camera that operates at a very high frame rate and outputs only differences between frames (e.g., motion)), or a combination of these cameras) may be placed in a central location to allow the entire cabin to be captured, for example, in the rearview mirror area (locations 91 and 92) or in the upper center console (location 93).

[0075] Additionally or alternatively, one or more cameras may be located on the instrument cluster or steering column in front of the driver, as shown in position 94, or in the A-pillar, as shown in positions 95 and 96. Additional cameras may be located, for example, in the back of the front seats, in the B-pillars or in the ceiling, to have better visibility of the second or third row of seats. Using two or more cameras, depth images or 3D images can be generated based on multi-view geometry and based on triangulation principles (e.g., stereo cameras). These cameras can then, for example, capture images at regular time intervals or if triggered by an application that requires vehicle safety monitoring as described herein. Applications using the images can be executed on an onboard computing system, or at least partially, for example, remotely in the cloud.

[0076] Fig.10 1 is a diagram of the internal components of a computing system 100 for vehicle safety monitoring that implements the functionality described herein. The computing system 100 may be located in a vehicle and includes at least one processor 101, a user interface 102, a network interface 103, and a main memory 106 that communicate with each other via a bus 105. Optionally, the computing system 100 may also include a static memory 107 and a disk drive unit (not shown), which also communicate with each other via the bus 105. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of the user interface 102.

[0077] In addition, the computing system 100 may also include a designated camera interface 104 for communicating with a vehicle's onboard camera. Alternatively, the computing system 100 may communicate with the camera via a network interface 103. The camera is used to capture images. The computing system 100 may also be connected to a database system (not shown) via a network interface, wherein the database system stores at least a portion of the image to provide the functionality described herein.

[0078] The main memory 106 may be a random access memory (RAM) and / or any other volatile memory. The main memory 106 may store program code for the vehicle monitoring computing software 108 and the communication software 109. Other modules required for other functions described herein may also be stored in the memory 106. The memory 106 may also store additional program data 110 required to provide the functions described herein, such as for pre-processing, etc. A portion of the program data 110, the vehicle monitoring computing software 108, and / or the communication software 109 may also be stored in a separate, such as cloud storage, and at least partially executed remotely.

[0079] According to one aspect, a vehicle is provided. The methods described herein may be stored as program code 108, 109, or 110 and may be at least partially included by the vehicle. Portions of program code 108, 109, or 110 may also be stored and executed on a cloud server to reduce the computational workload on the computing system 100 of the vehicle. The vehicle may also include one or more cameras, for example connected via camera interface 104, for capturing one or more images.

[0080] According to one aspect, a computer program comprising instructions is provided. When a computer executes the program, the instructions cause the computer to perform the methods described herein. The program code embodied in any system described herein can be distributed individually or collectively as a program product in a variety of different forms. Specifically, the program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the embodiments described herein.

[0081] Computer-readable storage media, which are non-transitory in nature, may include volatile and nonvolatile, and removable and non-removable tangible media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media may also include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be read by a computer.

[0082] The computer-readable storage medium itself should not be interpreted as a transient signal (e.g., a radio wave or other propagating electromagnetic wave, an electromagnetic wave propagating through a transmission medium such as a waveguide, or an electrical signal transmitted through a wire). Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded to an external computer or external storage device via a network.

[0083] It should be understood that although specific embodiments and variations are described herein, further modifications and substitutions are obvious to those skilled in the relevant art. In particular, examples are provided by illustrating principles, and a variety of specific methods and arrangements for implementing these principles are provided.

[0084] In some embodiments, the functions and / or actions specified in the flow chart, sequence diagram and / or block diagram may be reordered, processed serially and / or processed concurrently without departing from the scope of the present disclosure. In addition, any flow chart, sequence diagram and / or block diagram may include more or less blocks than those shown in the embodiments of the present disclosure.

[0085] The terms used herein are only used for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present disclosure. It should also be understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of the features, elements, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, elements, steps, operations, elements, components and / or their combinations. In addition, with respect to the use of the terms "include", "have", "has", "with", "composed of" or their variants in the detailed description or claims, these terms are intended to be included in a manner similar to the term "comprise".

[0086] Although the description of various embodiments has illustrated the method, and although these embodiments have been described in considerable detail, it is not the intention of the applicant to restrict or in any way limit the scope of the appended claims to such details. Additional advantages and modifications will be apparent to those skilled in the art. Therefore, the present disclosure in its broader aspects is not limited to the specific details, representative devices and methods, and illustrative examples shown and described. Therefore, the described embodiments should be understood as provided as examples for the purpose of teaching general features and principles, and should not be understood as limiting the scope as defined in the appended claims.

Claims

1. A computerized method for vehicle safety monitoring, the method comprising the following steps: determining a driver-related state based on data from one or more sensors of the vehicle, wherein at least one sensor photographs a cabin of the vehicle; comparing the driver-related state to one or more predetermined rules; and In response to detecting a violation of a rule of the one or more rules, a warning event is triggered.

2. The method according to claim 1, wherein: A sensor of the one or more sensors is an onboard camera, and wherein determining the driver-related state comprises: Obtaining image data from the vehicle-mounted camera; determining, based on the image data, at least one of a seating posture of a driver of the vehicle, an interaction of the driver with people and / or objects in the vehicle, and a type of object placed in the cabin; and The driver-related state is determined based on at least one of the seating posture of the driver of the vehicle, the driver's interaction with people and / or objects in the vehicle, and the type of objects placed in the cabin.

3. The method according to claim 1 or 2, wherein: Determining the driver-related state further includes: determining at least one of the number of persons in the cabin, the age of one or more persons, the activity of one or more persons, the seatbelt status of one or more persons, and alcohol use and / or intoxication of one or more persons; and The driver-related status is determined based on at least one of the number of people in the cabin, the age of the one or more people, the activity of the one or more people, the seat belt status of the one or more people, and alcohol use and / or intoxication of the one or more people.

4. The method according to any one of claims 1 to 3, wherein: Determining the driver related state is further based on additional vehicle data, wherein the additional vehicle data includes input from further vehicle sensors, a state of an advanced driver assistance system and / or a state of an in-vehicle infotainment system.

5. The method according to any one of claims 1 to 4, wherein: The one or more rules relate to at least one of the following: the presence of a co-pilot; the maximum number of persons in said vehicle; Phone use while driving; Maximum speed limit; Areas where driving is permitted; and The hours during which driving is permitted.

6. The method according to any one of claims 1 to 5, wherein: The warning event is at least one of: an email sent to an email address associated with the vehicle; a message sent to an application connected to the vehicle; a warning sound output from an in-vehicle speaker; a warning light at a dashboard of the vehicle; and a warning message displayed on an infotainment system of the vehicle.

7. The method according to any one of claims 1 to 6, further comprising: A warning level associated with a violation of a rule of the one or more rules is determined, wherein the warning level defines the warning event to be triggered.

8. The method according to any one of claims 1 to 7, further comprising: In response to a violation of the one or more rules, one or more vehicle driving control systems are interacted with.

9. A vehicle safety monitoring system, comprising: one or more sensors, wherein at least one sensor is configured to image a cabin of the vehicle; A vehicle monitoring computing unit, wherein the vehicle monitoring computing unit is configured to: determining a driver-related state based on data from the one or more sensors of the vehicle; and comparing the driver-related state to one or more predetermined rules; and A communication unit is configured to trigger a warning event in response to detecting a violation of a rule among the one or more rules.

10. The vehicle safety monitoring system according to claim 9, wherein: The cabin sensor of the one or more cabin sensors is an onboard camera, and wherein, in order to determine the driver-related state, the vehicle monitoring computing unit is further configured to: Obtaining image data from the vehicle-mounted camera; determining, based on the image data, at least one of a seating posture of a driver of the vehicle, an interaction of the driver with people and / or objects in the vehicle, and a type of object placed in the cabin; and The driver-related state is determined based on at least one of the seating posture of the driver of the vehicle, the driver's interaction with people and / or objects in the vehicle, and the type of objects placed in the cabin.

11. The vehicle safety monitoring system according to claim 9 or 10, wherein: In order to determine the driver-related state, the vehicle monitoring computing unit is further configured to: determining at least one of the number of persons in the cabin, the age of one or more persons, the activity of one or more persons, the seatbelt status of one or more persons, and alcohol use and / or intoxication of one or more persons; as well as The driver-related status is further determined based on the number of people in the cabin, the age of the one or more people, the activity of the one or more people, the seat belt status of the one or more people, and at least one of alcohol use and / or intoxication of the one or more people.

12. The vehicle safety monitoring system according to any one of claims 9 to 11, wherein: The vehicle monitoring computing unit is further configured to determine the driver-related state based on additional vehicle data, wherein the additional vehicle data includes inputs from further vehicle sensors, states of advanced driver assistance systems and / or states of an onboard infotainment system.

13. The vehicle safety monitoring system according to any one of claims 9 to 12, further configured to interact with one or more vehicle driving control systems in response to a violation of the one or more rules.

14. A vehicle, comprising the vehicle safety monitoring system according to any one of claims 9 to 13.

15. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 8.

Images