Apparatus and method for assisting vehicle operation based on exponential risk fused situation evaluation (SAFER)

Abstract

The apparatus comprises a first sensor configured to provide a first input related to an environment external to a vehicle, a second sensor configured to provide a second input related to operation of the vehicle, and a processing unit including a first stage processing system and a second stage processing system, the first stage processing system configured to receive the first input and the second input, process the first input to obtain a first time series of information and process the second input to obtain a second time series of information, the second stage processing system comprising a neural network model configured to receive the first time series of information and the second time series of information in parallel, the neural network being configured to process the first time series and the second time series to determine a probability of a predicted event related to operation of the vehicle.

Description

[Technical field]

[0001] Related Application Data

[0001] This application claims priority to and benefit of U.S. Provisional Patent Application No. 63 / 285,073, filed December 1, 2021, U.S. Patent Application No. 17 / 726,236, filed April 21, 2022, and U.S. Patent Application No. 17 / 726,269, filed April 21, 2022.

[0002]

[0002] The present invention relates to devices and methods for assisting vehicle driving and scoring and ensuring driving behavior, and more particularly to devices and methods for identifying driving and situational risks. [Background technology]

[0003]

[0003] Sensors (cameras, radar, Lidar, etc.) are used in vehicles to capture images of road conditions outside the vehicle. For example, a camera can be installed on a target vehicle to monitor the path the target vehicle is traveling or to monitor vehicles around the target vehicle.

[0004]

[0004] It is desirable to utilize camera imagery to provide collision prediction and / or intersection violation prediction. It is also desirable to provide a warning to the driver or automatically operate the vehicle in response to a predicted risk of collision, intersection violation, etc. It is further desirable to provide an output indicative of good or bad driving, which can be useful for driver training and fleet management. Alternatively or additionally, the output can be useful for comparing the actual driver's behavior to the behavior of a good driver in a similar situation.

[0005]

[0005] Described herein are new techniques for determining and tracking risk of a collision and / or determining and tracking risk of an intersection violation. Also described are new techniques for providing control signals that activate warning or feedback generators to alert drivers to the risk of a collision and / or risk of an intersection violation and / or to mitigate such risk. Summary of the Invention

[0006]

[0006] The apparatus comprises a first sensor configured to provide a first input related to an environment external to the vehicle, a second sensor configured to provide a second input related to operation of the vehicle, and a processing unit configured to receive a first input from the first sensor and a second input from the second sensor, the processing unit including a first stage processing system and a second stage processing system, the first stage processing system configured to receive a first input from the first sensor, receive a second input from the second sensor, process the first input to obtain a first time series of information, and process the second input to obtain a second time series of information, the second stage processing system comprising a neural network model configured to receive the first time series of information and the second time series of information in parallel, the neural network being configured to process the first time series and the second time series to determine a probability of a predicted event related to operation of the vehicle.

[0007]

[0007] By way of non-limiting example, the first sensor may be a camera, a lidar, a radar, or any combination thereof configured to sense characteristics of an environment external to the vehicle. The first input from the first sensor may include one or more time series.

[0008]

[0008] Also, by way of non-limiting example, the second sensor may be a camera, a depth sensor, a radar, or any combination thereof configured to capture images and / or detect a driver's state. Alternatively or additionally, the second sensor may include one or more sensing units configured to sense one or more states of the vehicle (e.g., speed, acceleration, deceleration, braking, heading, steering angle, braking operation, wheel traction, engine state, brake pedal position, accelerator pedal position, turn signal state, etc.) (vehicle state). The second input from the second sensor may include one or more time series.

[0009]

[0009] Optionally, the processing unit is configured to combine (e.g., fuse) multiple metadata streams indicative of or associated with different respective situation aspects to determine whether the risks are combined or combined in such a manner (e.g., non-linearly, exponentially, etc.) that results in a very dangerous situation. For example, a driver smoking a cigarette may not be very dangerous. However, if the same driver also uses a mobile phone and is too close to the preceding vehicle, the combined risk (e.g., smoking risk + mobile phone use risk + preceding vehicle too close risk) may indicate a very dangerous situation. The risky situation may be represented by a risk value that increases non-linearly (e.g., exponentially) due to the combination of risk factors. In some embodiments, each metadata stream may be a time series data obtained by processing raw data from one or more sensors.

[0010] In some embodiments, the processing unit is configured to identify peaks (or spikes) in a risk combination, where the peaks (or spikes) represent an escalating risk situation. In some embodiments, the escalating risk situation may be represented by a risk value that increases non-linearly (e.g., exponentially) with a combination of risk factors.

[0011] Also, in some embodiments, the processing unit is configured to identify dangerous situations early enough to intervene (e.g., provide a warning, feedback, recommendation, etc., to reduce one or more risk factors). In some embodiments, the processing unit may identify what other good drivers have done in similar situations to reduce risk and use such knowledge to provide a warning, feedback, or recommendation to reduce one or more risk factors (e.g., increase distance from the vehicle ahead, change speed, change direction, change attention state, etc.).

[0012] Optionally, the first input has low dimensionality or complexity compared to the first time series of information.

[0013] Optionally, the first time series of information is indicative of a first risk factor and the second time series of information is indicative of a second risk factor.

[0014] Optionally, the processing unit is configured to package the first time series and the second time series into a data structure for feeding to a neural network model.

[0015] Optionally, the data structure is comprised of a two-dimensional data matrix. In some embodiments, the data matrix may include multiple sensor streams from multiple sensors of the same or different sensor types.

[0016] Optionally, the first time series represents conditions external to the vehicle at different respective points in time, and the second time series represents conditions of the driver and / or the vehicle at different respective points in time.

[0017]

[0017] Optionally, the probability of the predicted event is a first probability of a first predicted event, and the processing unit is configured to determine a second probability of a second predicted event, the first predicted event and the second predicted event being related to operation of the vehicle, and the processing unit is configured to calculate a risk score based on the first probability of the first predicted event and based on the second probability of the second predicted event.

[0018]

[0018] Optionally, the first predicted event is a collision event and the second predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of a collision event and based on a second probability of a non-hazardous event.

[0019]

[0019] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and adding the first weighted probability and the second weighted probability.

[0020]

[0020] Optionally, the processing unit is configured to determine a third probability of a third predicted event, and the processing unit is configured to calculate the risk score based on the first probability of the first predicted event, based on the second probability of the second predicted event, and based on the third probability of the third predicted event.

[0021]

[0021] Optionally, the first predicted event is a collision event, the second predicted event is a near-collision event, and the third predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of the collision event, based on a second probability of the near-collision event, and based on a third probability of the non-hazardous event.

[0022]

[0022] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and applying a third weighting to the third probability to obtain a third weighted probability, and adding together the first weighted probability, the second weighted probability, and the third weighted probability.

[0023]

[0023] Optionally, the first input and the second input are obtained within the past T seconds, and the processing unit is configured to process the first input and the second input obtained within the past T seconds to determine a probability of the predicted event, where T is at least 3 seconds.

[0024] Optionally, the predicted event is for a future time at least one second after the current time.

[0025] Optionally, the predicted event is for a future time at least 2 seconds after the current time.

[0026]

[0026] Optionally, the processing unit is configured to calculate a first risk score for a first time point based on a probability, and the processing unit is also configured to calculate a second risk score for a second time point and identify a difference between the first risk score and the second risk score, the difference indicating whether a dangerous situation is escalating or subsiding.

[0027]

[0027] Optionally, the processing unit is configured to determine the risk score based on a probability of a predicted event.

[0028] Optionally, the processing unit is configured to generate a control signal based on the risk score.

[0029]

[0029] Optionally, the processing unit is configured to generate a control signal if the risk score meets a criterion.

[0030]

[0030] Optionally, the processing unit is configured to generate a control signal for operating a device if the risk score meets a criterion.

[0031] Optionally, the device includes a speaker for sounding an alarm, a display or light emitting device for providing a visual signal, a tactile feedback device, a collision avoidance system, or a vehicle control device for a vehicle.

[0032]

[0032] Optionally, the first time series and the second time series each include any two or more of: distance to a leading vehicle, distance to an intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions, information about special zones, information about the environment (weather, road type, etc.), information about traffic conditions, time of day, information about visibility conditions, information about identified objects, object position, object direction of movement, object speed, bounding box, vehicle operating parameters, information about the driver's status, information about the driver's history, continuous driving time, proximity to a meal time, information about accident history, and audio information.

[0033]

[0033] Optionally, the first sensor comprises a camera, a Lidar, a radar, or any combination thereof configured to sense the environment outside the vehicle.

[0034] Optionally, the second sensor comprises a camera arranged to view the driver of the vehicle.

[0035]

[0035] Optionally, the second sensor comprises one or more sensing units configured to sense one or more characteristics of the vehicle.

[0036]

[0036] Optionally, the first input includes a first image and the second input includes a second image, and the first stage processing system is configured to receive the first image and the second image, process the first image to obtain a first time series of information, and process the second image to obtain a second time series of information.

[0037]

[0037] The apparatus comprises a first sensor configured to provide a first input, a second sensor configured to provide a second input, and a processing unit configured to receive the first input from the first sensor and the second input from the second sensor, wherein the processing unit is configured to identify a first probability of a first predicted event and a second probability of a second predicted event, the first predicted event and the second predicted event are associated with operation of a vehicle, and the processing unit is configured to calculate a risk score based on the first probability of the first predicted event and based on the second probability of the second predicted event.

[0038]

[0038] Optionally, the first predicted event is a collision event and the second predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of a collision event and based on a second probability of a non-hazardous event.

[0039]

[0039] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and adding the first weighted probability and the second weighted probability.

[0040]

[0040] Optionally, the processing unit is configured to determine a third probability of a third predicted event, and the processing unit is configured to calculate the risk score based on the first probability of the first predicted event, based on the second probability of the second predicted event, and based on the third probability of the third predicted event.

[0041]

[0041] Optionally, the first predicted event is a collision event, the second predicted event is a near-collision event, and the third predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of the collision event, based on a second probability of the near-collision event, and based on a third probability of the non-hazardous event.

[0042]

[0042] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and applying a third weighting to the third probability to obtain a third weighted probability, and adding up the first weighted probability, the second weighted probability, and the third weighted probability.

[0043]

[0043] Optionally, the first input and the second input include data acquired within the past T seconds, and the processing unit is configured to process the data acquired within the past T seconds to determine a first probability of the first predictive event and a second probability of the second predictive event, where T is at least 3 seconds.

[0044] Optionally, the first predicted event is for a future time at least one second after the current time.

[0045] Optionally, the first predicted event is for a future time at least 2 seconds after the current time.

[0046]

[0046] Optionally, the processing unit is configured to calculate a risk score for a first time point, and the processing unit is also configured to calculate a further risk score for a second time point and to identify a difference between the risk score and the further risk score, which difference indicates whether a dangerous situation is escalating or subsiding.

[0047]

[0047] Optionally, the processing unit is configured to generate a control signal based on the risk score.

[0048]

[0048] Optionally, the processing unit is configured to generate a control signal if the risk score meets a criterion.

[0049]

[0049] Optionally, the processing unit is configured to generate a control signal for operating a device if the risk score satisfies a criterion.

[0050] Optionally, the device includes a speaker for sounding an alarm, a display or light emitting device for providing a visual signal, a tactile feedback device, a collision avoidance system, or a vehicle control device for a vehicle.

[0051]

[0051] Optionally, the processing unit is configured to determine a first probability of the first predicted event and a second probability of the second predicted event based on the first input and the second input.

[0052]

[0052] Optionally, the processing unit comprises a first stage processing system and a second stage processing system, the first stage processing system configured to receive the first input and process the first input to provide a first output, and the second stage processing system configured to receive the first output and process the first output to provide a second output, the first output having lower dimensionality or less complexity compared to the first input, and the second output having lower dimensionality or less complexity compared to the first output.

[0053] Optionally, the processing unit includes a neural network model.

[0054] Optionally, the neural network model is configured to receive a first time series of information indicative of a first risk factor and a second time series of information indicative of a second risk factor.

[0055] Optionally, the neural network model is configured to receive the first time series and the second time series in parallel and / or to process the first time series and the second time series in parallel.

[0056] Optionally, the processing unit is configured to package the first time series and the second time series into a data structure for feeding to a neural network model.

[0057] Optionally, the data structure is comprised of a two-dimensional data matrix. In some embodiments, the data matrix may include multiple sensor streams from multiple sensors of the same or different sensor types.

[0058] Optionally, said first time series represents conditions external to the vehicle at different respective points in time, and said second time series represents conditions of the driver and / or the vehicle at different respective points in time.

[0059] Optionally, the first time series is indicative of a first characteristic of the vehicle at each different point in time, and the second time series is indicative of a second characteristic of the vehicle at each different point in time.

[0060]

[0060] Optionally, the first time series and second time series include any one or combination of two or more of: distance to leading vehicle, distance to intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions, information about special zones, information about the environment (weather, road type, etc.), information about traffic conditions, time of day, information about visibility conditions, information about identified objects, object position, object direction of movement, object speed, bounding box, vehicle operating parameters, information about driver status, information about driver history, continuous driving time, proximity to meal time, information about accident history, and audio information.

[0061]

[0061] Optionally, the first sensor comprises a camera, a Lidar, a radar, or any combination thereof configured to sense the environment outside the vehicle.

[0062]

[0062] Optionally, said second sensor comprises a camera arranged to view the driver of the vehicle.

[0063]

[0063] Optionally, the second sensor comprises one or more sensing units configured to sense one or more characteristics of the vehicle.

[0064]

[0064] Optionally, the first input includes a first image and the second input includes a second image, and the first stage processing system is configured to receive the first image and the second image, process the first image to obtain a first time series of information and process the second image to obtain a second time series of information.

[0065]

[0065] The apparatus comprises a first camera viewing an environment external to the vehicle, a second camera viewing a driver of the vehicle, and a processing unit configured to receive a first image from the first camera and a second image from the second camera, the processing unit including a first stage processing system and a second stage processing system, the first stage processing system configured to receive a first image from the first camera and a second image from the second camera, process the first image to obtain a first time series of information and process the second image to obtain a second time series of information, the second stage processing system comprising a neural network model configured to receive the first time series of information and the second time series of information in parallel, the neural network being configured to process the first time series and the second time series to determine a probability of a predicted event associated with operation of the vehicle.

[0066] Optionally, the first image has reduced dimensionality or complexity compared to the first time series of information.

[0067] Optionally, the first time series of information is indicative of a first risk factor and the second time series of information is indicative of a second risk factor, in some embodiments the number of time series may be two or more and each time series may represent two or more risk factors.

[0068] Optionally, the processing unit is configured to package the first time series and the second time series into a data structure for feeding to a neural network model.

[0069]

[0069] Optionally, the data structure is comprised of a two-dimensional data matrix. In some embodiments, the data matrix may include multiple sensor streams from multiple sensors of the same or different sensor types.

[0070] Optionally, said first time series represents conditions external to the vehicle at different respective points in time, and said second time series represents conditions of the driver at different respective points in time.

[0071]

[0071] Optionally, the probability of the predicted event is a first probability of a first predicted event, and the processing unit is configured to determine a second probability of a second predicted event, the first predicted event and the second predicted event are related to operation of a vehicle, and the processing unit is configured to calculate a risk score based on the first probability of the first predicted event and based on the second probability of the second predicted event.

[0072]

[0072] Optionally, the first predicted event is a collision event and the second predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of a collision event and based on a second probability of a non-hazardous event.

[0073]

[0073] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and adding the first weighted probability and the second weighted probability.

[0074]

[0074] Optionally, the processing unit is configured to determine a third probability of a third predictive event, and the processing unit is configured to calculate the risk score based on the first probability of the first predictive event, based on the second probability of the second predictive event, and based on the third probability of the third predictive event.

[0075]

[0075] Optionally, the first predicted event is a collision event, the second predicted event is a near-collision event, and the third predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of the collision event, based on a second probability of the near-collision event, and based on a third probability of the non-hazardous event.

[0076]

[0076] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and applying a third weighting to the third probability to obtain a third weighted probability, and adding up the first weighted probability, the second weighted probability, and the third weighted probability.

[0077]

[0077] Optionally, the first image and the second image are acquired within the past T seconds, and the processing unit is configured to process the first image and the second image acquired within the past T seconds to determine the probability of the predicted event, where T is at least 3 seconds.

[0078]

[0078] Optionally, the predicted event is for a future time at least 1 second, or at least 2 seconds from the current time.

[0079]

[0079] Optionally, the processing unit is configured to calculate a first risk score for a first time point based on a probability, and the processing unit is also configured to calculate a second risk score for a second time point and identify a difference between the first risk score and the second risk score, the difference indicating whether a dangerous situation is escalating or subsiding.

[0080]

[0080] Optionally, the processing unit is configured to determine the risk score based on a probability of a predicted event.

[0081]

[0081] Optionally, the processing unit is configured to generate a control signal based on the risk score.

[0082]

[0082] Optionally, the processing unit is configured to generate a control signal if the risk score meets a criterion.

[0083]

[0083] Optionally, the processing unit is configured to generate a control signal for operating a device if the risk score meets a criterion.

[0084]

[0084] Optionally, the device includes a speaker for sounding an alarm, a display or light emitting device for providing a visual signal, a tactile feedback device, a collision avoidance system, or a vehicle control device for a vehicle.

[0085]

[0085] Optionally, the first time series and the second time series each include any two or more of: distance to leading vehicle, distance to intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information regarding road conditions, information regarding special zones, information regarding the environment (weather, road type, etc.), information regarding traffic conditions, time of day, information regarding visibility conditions, information regarding identified objects, object position, object direction of movement, object speed, bounding box, vehicle operating parameters, information regarding driver status, information regarding driver history, continuous driving time, proximity to meal time, information regarding accident history, and audio information.

[0086]

[0086] The apparatus comprises a first camera configured to view an external environment, a second camera configured to view a vehicle driver, and a processing unit configured to receive a first image from the first camera and a second image from the second camera, wherein the processing unit is configured to identify a first probability of a first predicted event and a second probability of a second predicted event, the first predicted event and the second predicted event are associated with operation of the vehicle, and the processing unit is configured to calculate a risk score based on the first probability of the first predicted event and based on the second probability of the second predicted event.

[0087]

[0087] Optionally, the first predicted event is a collision event and the second predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of a collision event and based on a second probability of a non-hazardous event.

[0088]

[0088] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and adding the first weighted probability and the second weighted probability.

[0089]

[0089] Optionally, the processing unit is configured to determine a third probability of a third predictive event, and the processing unit is configured to calculate the risk score based on the first probability of the first predictive event, based on the second probability of the second predictive event, and based on the third probability of the third predictive event.

[0090]

[0090] Optionally, the first predicted event is a collision event, the second predicted event is a near-collision event, and the third predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of the collision event, based on a second probability of the near-collision event, and based on a third probability of the non-hazardous event.

[0091]

[0091] Optionally, the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and applying a third weighting to the third probability to obtain a third weighted probability, and adding up the first weighted probability, the second weighted probability, and the third weighted probability.

[0092]

[0092] Optionally, the processing unit is configured to process data acquired over the past T seconds to determine a first probability of the first predictive event and a second probability of the second predictive event, where T is at least 3 seconds.

[0093]

[0093] Optionally, the first predicted event is for a future time at least 1 second, or at least 2 seconds from the current time.

[0094]

[0094] Optionally, the processing unit is configured to calculate a risk score for a first time point, and the processing unit is also configured to calculate a further risk score for a second time point and to identify a difference between the risk score and the further risk score, which difference indicates whether a dangerous situation is escalating or subsiding.

[0095]

[0095] Optionally, the processing unit is configured to generate a control signal based on the risk score.

[0096]

[0096] Optionally, the processing unit is configured to generate a control signal if the risk score meets a criterion.

[0097]

[0097] Optionally, the processing unit is configured to generate a control signal for operating a device if the risk score meets a criterion.

[0098]

[0098] Optionally, the device includes a speaker for sounding an alarm, a display or light emitting device for providing a visual signal, a tactile feedback device, a collision avoidance system, or a vehicle control device for a vehicle.

[0099]

[0099] Optionally, the processing unit is configured to acquire a first image from the first camera and a second image from the second camera, and the processing unit is configured to determine a first probability of the first predicted event and a second probability of the second predicted event based on the first image and the second image.

[0100]

[0100] Optionally, the processing unit comprises a first stage processing system and a second stage processing system, the first stage processing system configured to receive raw data and process the raw data to provide a first output, and the second stage processing system configured to receive the first output and process the first output to provide a second output, the first output having lower dimensionality or less complexity compared to the raw data, and the second output having lower dimensionality or less complexity compared to the first output.

[0101]

[0101] Optionally, the processing unit includes a neural network model.

[0102]

[0102] Optionally, the neural network model is configured to receive input including at least a first time series of information indicative of a first risk factor and a second time series of information indicative of a second risk factor.

[0103]

[0103] Optionally, the neural network model is configured to receive the first time series and the second time series in parallel and / or to process the first time series and the second time series in parallel.

[0104]

[0104] Optionally, the processing unit is configured to package the input into a data structure for feeding to a neural network model.

[0105]

[0105] Optionally, the data structure comprises a two-dimensional data matrix.

[0106]

[0106] Optionally, the first time series represents conditions external to the vehicle at different respective points in time, and the second time series represents conditions of the driver at different respective points in time.

[0107]

[0107] Optionally, the inputs include any one or combination of two or more of: distance to leading vehicle, distance to intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions, information about special zones, information about the environment (weather, road type, etc.), information about traffic conditions, time of day, information about visibility conditions, information about identified objects, object location, direction of object movement, object speed, bounding boxes, vehicle operating parameters, information about driver status, information about driver history, continuous driving time, proximity to meal time, information about accident history, and audio information.

[0108]

[0108] The apparatus includes a first camera configured to view an environment outside the vehicle, a second camera configured to view a driver of the vehicle, and a processing unit configured to receive a first image from the first camera and a second image from the second camera, the processing unit including a model configured to receive a plurality of inputs and generate a metric based on at least some of the inputs, the plurality of inputs including at least a first time series of information indicative of a first risk factor and a second time series of information indicative of a second risk factor.

[0109]

[0109] Optionally, the plurality of inputs include distance to leading vehicle (which may indicate distance to collision), distance to intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions (e.g., dry, wet, snow, etc.), information about (e.g., identifying) special zones (e.g., construction zones, school zones, etc.), information about (e.g., identifying) the environment (e.g., city, suburban, urban area, etc.), information about (e.g., identifying) traffic conditions, time of day, information about (e.g., identifying) visibility conditions, identified objects (stop signs, traffic lights, pedestrians, cars, etc.), object position, object direction of movement, object speed, bounding boxes, vehicle motion parameters (e.g., acceleration, speed, centrifugal force, steering angle, kinematic signals such as brake, accelerator position, turn signal, etc.), driver information regarding (e.g., identifying) the state of (e.g., driver's gaze direction (e.g., looking up, looking down, looking left, etc.), using a phone, holding an object, smoking a cigarette, eyes closed, head turned away or away so that the face is not detected, gaze direction relative to the driver, gaze direction mapped to external objects or objects within the vehicle (e.g., head unit, mirrors, etc.), drowsiness, fatigue, road rage, stress, medical emergencies (heart attack, stroke, pulse, sweating, dilated pupils, etc.), information about the driver's history (e.g., motor vehicle record, accident rate, experience, age, years driving, age of vehicle, route experience, etc.), continuous driving time, proximity to meal times, information about accident history (e.g., fatal accidents at a given time, accidents at a given time, geospatial heat maps of risk data by location by time, etc.).

[0110]

[0110] Optionally, the processing unit is configured to package two or more inputs into a data structure for feeding to the model.

[0111]

[0111] Optionally, the data structure comprises a two-dimensional data matrix.

[0112]

[0112] Optionally, the model includes a neural network model.

[0113]

[0113] Optionally, the input includes first information relating to a driver's condition and second information relating to a condition external to the vehicle.

[0114]

[0114] Optionally, the neural network model is configured to receive first information relating to a state of a driver and second information relating to a state external to the vehicle.

[0115]

[0115] Optionally, the neural network model is configured to receive the first time series and the second time series in parallel and / or to process the first time series and the second time series in parallel.

[0116]

[0116] Optionally, the processing unit is configured to package the input into a data structure for feeding to the model.

[0117]

[0117] Optionally, the data structure comprises a two-dimensional data matrix.

[0118]

[0118] Optionally, the two-dimensional data matrix includes a value of a first input of a plurality of inputs for each different time point and a value of a second input of the plurality of inputs for each different time point.

[0119]

[0119] Optionally, the two-dimensional data matrix includes first information indicative of a driver's state at different respective times and second information indicative of a state external to the vehicle.

[0120]

[0120] Optionally, the model of the processing unit is configured to predict a near collision and to determine a score indicative of the probability of a predicted near collision.

[0121]

[0121] Optionally, the model of the processing unit is configured to predict a collision and to determine a score indicative of the probability of a predicted collision.

[0122]

[0122] Optionally, the model of the processing unit is configured to predict a non-event (eg a non-risk event) and to determine a score indicative of the probability of the predicted non-event.

[0123]

[0123] Optionally, the metric is a weighted sum of the scores for each of the different classifications.

[0124]

[0124] Optionally, the classification includes at least "non-events" and "associated events."

[0125]

[0125] Optionally, the classifications include at least "non-event", "predicted near-crash", and "predicted crash".

[0126]

[0126] Optionally, at least one of said inputs is based on an output from said first camera.

[0127]

[0127] Optionally, at least one of said inputs is based on an output from said second camera.

[0128]

[0128] Optionally, the processing unit is configured to provide a control signal for operating a device if the metric satisfies a criterion.

[0129]

[0129] Optionally, the device comprises a speaker for generating an alarm.

[0130]

[0130] Optionally, the device includes a display or light emitting device for providing a visual signal.

[0131]

[0131] Optionally, the device comprises a component of a vehicle.

[0132]

[0132] The apparatus comprises a first camera configured to view an environment outside the vehicle, a second camera configured to view a driver of the vehicle, and a processing unit configured to receive a first image from the first camera and a second image from the second camera, wherein the processing unit is configured to determine first information indicative of a risk of collision with the vehicle based at least in part on the first image, and the processing unit is configured to determine second information indicative of a state of the driver based at least in part on the second image, and the processing unit is configured to decide whether to provide a control signal for operating a device based on (1) the first information indicative of the risk of collision with the vehicle and (2) the second information indicative of the state of the driver.

[0133]

[0133] Optionally, said processing unit is configured to predict a collision at least 3 seconds or more in advance of an expected time of occurrence of the predicted collision.

[0134]

[0134] Optionally, the processing unit is configured to predict a collision with sufficient lead time for the driver's brain to process the inputs and for the driver to take action to mitigate the risk of a collision.

[0135]

[0135] Optionally, the sufficient lead time depends on the driver's status.

[0136]

[0136] Optionally, the first information indicative of the risk of collision includes a predicted collision, the processing unit is configured to determine an estimated time until the predicted collision occurs, and the processing unit is configured to provide the control signal if the estimated time until the predicted collision occurs is less than a threshold.

[0137]

[0137] Optionally, the device comprises a warning generator, and the processing unit is configured to provide a control signal so that the device issues a warning to the driver when an estimated time until a predicted collision occurs is less than a threshold.

[0138]

[0138] Optionally, the device includes a vehicle control device, and the processing unit is configured to provide a control signal to the device to control the vehicle when an estimated time until a predicted collision occurs is less than a threshold.

[0139]

[0139] Optionally, the threshold value is variable based on second information indicative of a driver's state.

[0140]

[0140] Optionally, the processing unit is configured to iteratively evaluate the estimated time against a variable threshold as the predicted collision approaches in time as the estimated time until the predicted collision occurs decreases.

[0141]

[0141] Optionally, the threshold is variable in real time based on the driver's state.

[0142]

[0142] Optionally, the processing unit is configured to modify (e.g., decrease or increase) the threshold value if the driver's state indicates that the driver is distracted or not paying attention to the driving task.

[0143]

[0143] Optionally, the processing unit is configured to at least temporarily withhold providing a control signal if an estimated time until a predicted collision occurs is longer than a threshold.

[0144]

[0144] Optionally, said processing unit is configured to determine a level of risk of collision, said processing unit being configured to adjust said threshold value based on the determined level of risk of collision.

[0145]

[0145] Optionally, the driver's state includes a distracted state, and the processing unit is configured to determine a level of the driver's distracted state, and the processing unit is configured to adjust the threshold based on the determined level of the driver's distracted state.

[0146]

[0146] Optionally, when the driver's state indicates the driver is paying attention to the driving task, the threshold has a first value, and when the driver's state indicates the driver is distracted or not paying attention to the driving task, the threshold has a second value higher than the first value.

[0147]

[0147] Optionally, the threshold is also based on sensor information indicating that the vehicle is being operated to reduce the risk of collision.

[0148]

[0148] Optionally, the processing unit is configured to decide whether to provide a control signal based on (1) first information indicative of a risk of collision with the vehicle, (2) second information indicative of a driver's state, and (3) sensor information indicating that the vehicle is being operated to mitigate the risk of collision.

[0149]

[0149] Optionally, the apparatus further includes a non-primary medium for storing a first model, the processing unit being configured to process the first image based on the first model to determine the risk of collision.

[0150]

[0150] Optionally, the first model includes a neural network model.

[0151]

[0151] Optionally, the non-transitory medium is configured to store a second model, and the processing unit is configured to process a second image based on the second model to determine a state of the driver.

[0152]

[0152] Optionally, the processing unit is configured to determine a metric value for each of a plurality of driver state classifications, and the processing unit is configured to determine whether the driver is engaged in a driving task based on one or more of the metric values.

[0153]

[0153] Optionally, the driver state classification includes two or more of the driver's gaze direction (e.g., looking down, looking up, looking left, looking right, etc.), mobile phone use, smoking, holding an object, hands off the steering wheel, not wearing a seat belt, eyes closed, looking ahead, one hand on the steering wheel, and both hands on the steering wheel.

[0154]

[0154] Optionally, said processing unit is configured to compare said metric value with a respective threshold value for each driver state classification.

[0155]

[0155] Optionally, the processing unit is configured to determine that the driver belongs to one of the driver state classifications if one of the corresponding metric values ​​meets or exceeds one of the corresponding threshold values.

[0156]

[0156] Optionally, the first camera, the second camera and the processing unit are integrated as parts of an aftermarket device for the vehicle.

[0157]

[0157] Optionally, the processing unit is configured to determine the second information by processing the second image to determine whether or not an image of the driver satisfies a driver state classification, and the processing unit is configured to determine whether or not the driver is engaged in a driving task based on whether or not the image of the driver satisfies a driver state classification.

[0158]

[0158] Optionally, the processing unit is configured to process the second image based on a neural network model to determine a driver's state.

[0159]

[0159] Optionally, the processing unit is configured to determine whether an object in the first image, or a bounding box of the object, overlaps with a region of interest.

[0160]

[0160] Optionally, the region of interest has a shape that varies corresponding to the shape of the road or lane on which the vehicle is traveling.

[0161]

[0161] Optionally, the processing unit is configured to determine a centre line of a road or lane on which the vehicle is travelling, and the region of interest has a shape based on said centre line.

[0162]

[0162] Optionally, the processing unit is configured to determine the distance between the vehicle and the physical position based on a y coordinate of the physical position in a camera image provided by the first camera, where the y coordinate is relative to an image coordinate frame.

[0163]

[0163] A method performed by the apparatus includes the steps of acquiring a first image generated by a first camera, the first camera configured to view an environment external to the vehicle; acquiring a second image generated by a second camera, the second camera configured to view a driver of the vehicle; determining first information indicative of a risk of collision with a vehicle based at least in part on the first image; determining second information indicative of a driver's state based at least in part on the second image; and determining whether to provide a control signal for operating a device based on (1) the first information indicative of a risk of collision with a vehicle and (2) the second information indicative of the driver's state.

[0164]

[0164] Optionally, the first information is determined by predicting a collision, the collision being predicted at least 3 seconds or more in advance of the expected occurrence time of the predicted collision.

[0165]

[0165] Optionally, the first information is determined by predicting a collision, the collision being predicted with sufficient lead time for the driver's brain to process the input and for the driver to take action to mitigate the risk of the collision.

[0166]

[0166] Optionally, the sufficient lead time depends on the driver's status.

[0167]

[0167] Optionally, the first information indicative of the risk of collision includes a predicted collision, and the method further includes determining an estimated time until the predicted collision occurs, and the control signal is provided to cause the device to provide a control signal if the estimated time until the predicted collision occurs is less than a threshold.

[0168]

[0168] Optionally, the device comprises a warning generator and the control signal is provided such that the device issues a warning to the driver if an estimated time until a predicted collision occurs is less than a threshold.

[0169]

[0169] Optionally, the device includes a vehicle control device and the control signal is provided to cause the device to control the vehicle when an estimated time until a predicted collision occurs is less than a threshold.

[0170]

[0170] Optionally, the threshold value is variable based on second information indicative of a driver's state.

[0171]

[0171] Optionally, the estimated time is iteratively evaluated with respect to a variable threshold as the predicted collision approaches in time as the estimated time until the predicted collision occurs decreases.

[0172]

[0172] Optionally, the threshold is variable in real time based on the driver's state.

[0173]

[0173] Optionally, the method further comprises modifying the threshold if the driver's state indicates that the driver is distracted or not paying attention to the driving task.

[0174]

[0174] Optionally, the method further comprises at least temporarily withholding generation of the control signal if the estimated time until the predicted collision occurs is greater than a threshold.

[0175]

[0175] Optionally, the method further comprises the steps of determining a level of risk of collision and adjusting the threshold value based on the determined level of risk of collision.

[0176]

[0176] Optionally, the driver's state includes a distracted state, and the method further includes the steps of determining a level of the driver's distracted state and adjusting the threshold based on the determined level of the driver's distracted state.

[0177]

[0177] Optionally, when the driver's state indicates the driver is paying attention to the driving task, the threshold has a first value, and when the driver's state indicates the driver is distracted or not paying attention to the driving task, the threshold has a second value higher than the first value.

[0178]

[0178] Optionally, the threshold is also based on sensor information indicating that the vehicle is being operated to reduce the risk of a collision.

[0179]

[0179] Optionally, the step of deciding whether to provide a control signal to operate the device is also performed based on sensor information indicating that the vehicle is being operated to reduce the risk of a collision.

[0180]

[0180] Optionally, the step of determining first information indicative of the risk of collision comprises processing the first image based on a first model.

[0181]

[0181] Optionally, the first model includes a neural network model.

[0182]

[0182] Optionally, the step of determining second information indicative of the driver's state comprises processing the second image based on a second model.

[0183]

[0183] Optionally, the method further includes determining a metric value for each of the plurality of pose classifications and determining whether the driver is engaged in a driving task based on the one or more metric values.

[0184]

[0184] Optionally, the pose classification includes two or more of the driver's gaze direction (e.g., poses looking down, looking up, looking left, looking right, etc.), poses using a mobile phone, poses smoking, poses holding an object, poses with hands off the steering wheel, poses not wearing a seat belt, poses with eyes closed, poses looking ahead, poses with one hand on the steering wheel, and poses with both hands on the steering wheel.

[0185]

[0185] Optionally, the method further comprises the step of comparing the metric value to a respective threshold value for each pose classification.

[0186]

[0186] Optionally, the method further includes determining that the driver belongs to one of the pose classifications if one of the corresponding metric values ​​meets or exceeds one of the corresponding threshold values.

[0187]

[0187] Optionally, the method is performed by an aftermarket device, and the first camera and second camera are integrated as parts of the aftermarket device.

[0188]

[0188] Optionally, the second information is determined by processing the second image to determine whether the image of the driver satisfies a pose classification, and the method further includes determining whether the driver is engaged in a driving task based on whether the image of the driver satisfies a pose classification.

[0189]

[0189] Optionally, the step of determining second information indicative of the driver's state comprises processing the second image based on a neural network model.

[0190]

[0190] Optionally, the method further comprises determining whether an object in the first image or a bounding box of the object overlaps with a region of interest.

[0191]

[0191] Optionally, the region of interest has a shape that varies corresponding to the shape of the road or lane on which the vehicle is traveling.

[0192]

[0192] Optionally, the method further comprises identifying a centre line of a road or lane on which the vehicle is travelling, and the region of interest has a shape based on said centre line.

[0193]

[0193] Optionally, the method further includes determining a distance between the vehicle and the physical location based on a y coordinate of the physical location in a camera image provided by the first camera, where the y coordinate is relative to an image coordinate frame.

[0194]

[0194] Other and further aspects and features will become apparent from reading the following detailed description. [Brief description of the drawings]

[0195]

[0195] The drawings illustrate the design and utility of the embodiments, in which like elements are referred to by common reference numerals. For a better understanding of how the advantages and objects are obtained, a more particular description of the embodiments will now be described with reference to the accompanying drawings. It will be understood that these drawings depict only exemplary embodiments, and therefore are not intended to limit the scope of the invention as claimed. [Figure 1]

[0196] FIG. 1 is a diagram illustrating an apparatus according to some embodiments. [Diagram 2]

[0197] FIG. 2A is a block diagram of the device of FIG. 1 according to some embodiments.

[0198] FIG. 2B illustrates an example of a processing scheme for the apparatus of FIG. 2A. [Diagram 3]

[0199] FIG. 3 is a diagram showing an example of an image captured by the camera of the device of FIG. [Figure 4]

[0200] FIG. 4 is a diagram illustrating an example of a classifier. [Diagram 5]

[0201] FIG. 5 illustrates a method according to some embodiments. [Figure 6]

[0202] FIG. 6 shows an example of an image captured by the camera of FIG. 1 and the output of various classifiers. [Figure 7]

[0203] FIG. 7 shows another example of an image captured by the camera of FIG. 1 and the output of various classifiers. [Figure 8]

[0204] FIG. 8 shows another example of an image captured by the camera of FIG. 1 and the output of various classifiers. [Figure 9]

[0205] FIG. 9 illustrates an example of a processing architecture having a first model and a second model coupled in series. [Figure 10]

[0206] FIG. 10 is a diagram illustrating an example of feature information received by the second model. [Figure 11]

[0207] FIG. 11 is a diagram illustrating another example of feature information received by the second model. [Figure 12]

[0208] FIG. 12 illustrates a drowsiness detection method performed by the device of FIG. 2 according to some embodiments. [Figure 13]

[0209] FIG. 13 is a diagram illustrating an example of a processing architecture that may be implemented using the apparatus of FIG. 2A. [Figure 14]

[0210] FIG. 14 is a diagram illustrating an example of object detection according to some embodiments. [Figure 15]

[0211] FIG. 15A illustrates another example of object identifiers, specifically showing that each object identifier is a box representing a leading vehicle.

[0212] FIG. 15B illustrates another example of object identifiers, specifically showing that each object identifier is a horizontal line.

[0213] FIG. 15C illustrates an example of preceding vehicle amount detection according to some embodiments.

[0214] FIG. 15D illustrates a technique for determining a region of interest based on centerline detection.

[0215] FIG. 15E illustrates the advantage of using the region of interest of FIG. 15D to detect objects at risk of collision. [Figure 16]

[0216] FIG. 16 illustrates three example scenarios involving a collision with a leading vehicle. [Figure 17]

[0217] FIG. 17 is a diagram showing another example of object detection when the object to be detected is a human being. [Figure 18]

[0218] FIG. 18 is a diagram showing an example of human movement prediction. [Figure 19]

[0219] 19A-19B show another example of object detection where the detected object is associated with an intersection.

[0220] FIG. 19C illustrates the concept of braking distance. [Figure 20]

[0221] FIG. 20 illustrates a method for determining the distance between a target vehicle and a position in front of the vehicle. [Figure 21]

[0222] FIG. 21 illustrates an example technique for generating control signals for controlling a vehicle and / or generating warnings for a driver. [Figure 22]

[0223] FIG. 22A illustrates a method including collision prediction according to some embodiments.

[0224] FIG. 22B illustrates a method including predicting intersection violations according to some embodiments. [Figure 23]

[0225] FIG. 23 is a diagram illustrating a technique for determining a model to use by the apparatus of FIG. 2A, according to some embodiments. [Figure 24]

[0226] FIG. 24 illustrates one technique for dealing with risk factors. [Diagram 25]

[0227] FIG. 25 illustrates another technique for dealing with risk factors. [Figure 26]

[0228] 26A-26C show examples of neural network architectures that may be employed to process multiple inputs. [Figure 27]

[0229] Figure 27 shows the number of deaths by time of day and day of the week. [Figure 28]

[0230] FIG. 28 shows an example of multiple inputs processed by the model to generate a prediction at least 2 seconds before the event occurs. [Figure 29]

[0231] FIG. 29 shows an example of a risk score calculation based on model predictions. [Diagram 30]

[0232] FIG. 30 shows an example of model output based on multiple time series inputs, where the model output indicates a high “crash” condition. [Diagram 31]

[0233] FIG. 31 shows an example of model output based on multiple time series inputs, where the model output indicates a high "near crash" condition. [Diagram 32]

[0234] FIG. 32 shows an example of model output based on multiple time series inputs, where the model output indicates a high "non-event" state. [Diagram 33]

[0235] 33A-33K show a series of external and corresponding internal images, illustrating in particular the problem that can be addressed by the technique of FIG. [Diagram 34]

[0236] 34A-34E show a series of external and corresponding internal images, illustrating in particular the problem that can be addressed by the technique of FIG. [Diagram 35]

[0237] 35A-35D show a series of external and corresponding internal images, illustrating in particular the problem that can be addressed by the technique of FIG. [Diagram 36]

[0238] 36A-36C show a series of external and corresponding internal images, illustrating in particular the problem that can be addressed by the technique of FIG. [Figure 37]

[0239] 37A-37C show a series of external and corresponding internal images, illustrating in particular the problem that can be addressed by the technique of FIG. [Figure 38]

[0240] FIG. 38 illustrates a special purpose processing system for implementing one or more of the electronic devices described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0196]

[0241] Various embodiments are described below with reference to the drawings. Please note that the figures may or may not be drawn to scale, and that elements with similar structures or functions are represented by similar reference numerals throughout the drawings. Please also note that the drawings are intended to facilitate the description of the embodiments. They are not intended to be exhaustive descriptions of the claimed invention, nor are they intended to limit the scope of the claimed invention. Furthermore, the illustrated embodiments need not have all aspects or advantages of the invention shown. Aspects or advantages described with respect to a particular embodiment are not necessarily limited to that embodiment, and may be implemented in other embodiments, even if not so illustrated or not so explicitly described.

[0197]

[0242] FIG. 1 illustrates a device 200 according to some embodiments. The device 200 is configured to be mounted on a vehicle, such as on a windshield or rearview mirror of the vehicle. The device 200 includes a first camera 202 configured to view outside the vehicle and a second camera 204 configured to view inside the vehicle. In the illustrated embodiment, the device 200 is in the form of an aftermarket device that can be mounted on the vehicle (i.e., offline from the manufacturing process of the vehicle). The device 200 may include a connector configured to couple the device 200 to the vehicle. By way of non-limiting example, the connector may be a suction cup, adhesive, a clamp, one or more screws, etc. The connector may be configured to removably secure the device 200 to the vehicle, where the device 200 can be selectively removed from and / or coupled to the vehicle as needed. Alternatively, the connector may be configured to permanently secure the device 200 to the vehicle. In other embodiments, the device 200 may be a part of the vehicle that is installed during the manufacturing process of the vehicle. It should be noted that device 200 is not limited to the illustrated configuration, and device 200 may have other configurations in other embodiments. For example, in other embodiments, device 200 may have a different form factor. In other embodiments, device 200 may be an end user device, such as a mobile phone, tablet, etc., having one or more cameras.

[0198]

[0243] 2A is a block diagram of the device 200 of FIG. 1 in accordance with some embodiments. The device 200 includes a first camera 202 and a second camera 204. As shown, the device 200 also includes a processing unit 210 coupled to the first camera 202 and the second camera 204, a non-transitory medium 230 configured to store data, a communication unit 240 connected to the processing unit 210, and a speaker 250 connected to the processing unit 210.

[0199]

[0244] In the illustrated embodiment, the first camera 202, the second camera 204, the processing unit 210, the non-transitory medium 230, the communication unit 240, and the speaker 250 may be integrated as part of an aftermarket device for the vehicle. In other embodiments, the first camera 202, the second camera 204, the processing unit 210, the non-transitory medium 230, the communication unit 240, and the speaker 250 may be integrated into the vehicle or installed on the vehicle during the manufacturing process of the vehicle.

[0200]

[0245] The processing unit 210 is configured to obtain images from the first camera 202 and images from the second camera 204 and process the images from the first and second cameras 202, 204. In some embodiments, the images from the first camera 202 may be processed by the processing unit 210 to monitor the environment outside the vehicle (e.g., for collision detection, collision prevention, driving environment monitoring, etc.). Also, in some embodiments, the images from the second camera 204 may be processed by the processing unit 210 to monitor the driver's driving behavior (e.g., whether the driver is distracted, drowsy, focused, etc.). In further embodiments, the processing unit 210 may process the images from the first camera 202 and / or the second camera 204 to determine a risk of a collision, to predict a collision, to provide a warning to the driver, etc. In other embodiments, the device 200 may not include the first camera 202. In such cases, the device 200 is configured to monitor only the environment inside the vehicle cabin.

[0201]

[0246] The processing unit 210 of the apparatus 200 may include hardware, software, or a combination of both. By way of non-limiting example, the hardware of the processing unit 210 may include one or more processors and / or one or more integrated circuits. In some embodiments, the processing unit 210 may be implemented as a module and / or may be part of any integrated circuit.

[0202]

[0247] The non-transitory medium 230 is configured to store data related to the operation of the processing unit 210. In the illustrated embodiment, the non-transitory medium 230 is configured to store a model that can be accessed by the processing unit 210 to identify a driver's pose appearing in an image from the camera 204 and / or to determine whether the driver is engaged in a driving task. Alternatively, the model may configure the processing unit 210 to have the functionality to identify a driver's pose and / or determine whether the driver is engaged in a driving task. Optionally, the non-transitory medium 230 may be configured to save images from the first camera 202 and / or images from the second camera 204. Also, in some embodiments, the non-transitory medium 230 may be configured to store data generated by the processing unit 210.

[0203]

[0248] The model stored in the non-transitory medium 230 may be any computational or processing model, including but not limited to a neural network model. In some embodiments, the model includes feature extraction parameters based on which the processing unit 210 can extract features from the images provided by the camera 204 to identify objects such as the driver's head, hat, face, nose, eyes, mobile device, etc. In some embodiments, the model may also include program instructions, commands, scripts, etc. In some implementations, the model may be in the form of an application that can be wirelessly received by the device 200.

[0204]

[0249] The communication unit 240 of the device 200 is configured to receive data wirelessly from a network, such as the cloud, the Internet, a Bluetooth network, or the like. In some embodiments, the communication unit 240 may be configured to transmit data wirelessly. For example, images from the first camera 202, images from the second camera 204, data generated by the processing unit, or any combination thereof, may be transmitted by the communication unit 240 to another device (e.g., a server, an accessory device such as a mobile phone, another device 200 in another vehicle, etc.) via a network, such as the cloud, the Internet, a Bluetooth network, or the like. In some embodiments, the communication unit 240 may include one or more antennas. For example, the communication 240 may include a first antenna configured to provide long-range communication and a second antenna configured to provide short-range communication (e.g., via Bluetooth). In other embodiments, the communication unit 240 may be configured to physically transmit and / or receive data via a cable or electrical contacts. In such cases, the communication unit 240 may include one or more communication connectors configured to couple with a data transmission device. For example, the communications unit 240 may include a connector configured to mate with a cable, a USB slot configured to receive a USB drive, a memory card slot configured to receive a memory card, and so on.

[0205]

[0250] Speaker 250 of apparatus 200 is configured to provide audio warnings and / or messages to a driver of the vehicle. For example, in some embodiments, processing unit 210 may be configured to detect an impending collision between the vehicle and an object external to the vehicle. In such a case, in response to detecting the impending collision, processing unit 210 may generate a control signal to cause speaker 250 to output an audio warning and / or message. As another example, in some embodiments, processing unit 210 may be configured to determine whether the driver is engaged in a driving task. If the driver is not engaged in a driving task or has not been engaged in a driving task for a predetermined period of time (e.g., 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc.), processing unit 210 may generate a control signal to cause speaker 250 to output an audio warning and / or message.

[0206]

[0251] Alternatively or additionally, the processing unit 210 may generate a control signal that activates a haptic feedback device to warn the driver and / or provides input to a collision avoidance system.

[0207]

[0252] Although the device 200 is described as having a first camera 202 and a second camera 204, in other embodiments the device 200 may not have a first camera 202 and may only include a second camera (cabin camera) 204. Also, in other embodiments the device 200 may include multiple cameras configured to view a cabin within a vehicle.

[0208]

[0253] As shown in FIG. 2A, the processing unit 210 also includes a driver monitoring module 211, an object detection unit 216, a collision prediction unit 218, an intersection violation prediction unit 222, and a signal generation controller 224. The driver monitoring module 211 is configured to monitor a driver of the vehicle based on one or more images provided by the second camera 204. In some embodiments, the driver monitoring module 211 is configured to determine one or more poses of the driver. In some embodiments, the driver monitoring module 211 may also be configured to determine a state of the driver, such as whether the driver is awake, drowsy, or paying attention to the driving task. In some cases, the driver's pose itself can also be considered as a state of the driver.

[0209]

[0254] The object detector 216 is configured to detect one or more objects in an environment external to the vehicle based on one or more images provided by the first camera 202. As non-limiting examples, the detected objects may be vehicles (e.g., automobiles, motorcycles, etc.), lane boundaries, humans, bicycles, animals, road signs (e.g., stop signs, road signs, no turn signs, etc.), traffic lights, road markings (e.g., stop lines, lane markings, road markings, etc.), etc. In some embodiments, the detected vehicle may be a leading vehicle, which is a vehicle ahead of the target vehicle traveling in the same lane as the target vehicle.

[0210]

[0255] The collision prediction unit 218 is configured to determine a risk of collision based on the output from the object detection unit 216. For example, in some embodiments, the collision prediction unit 218 may determine that there is a risk of collision with a leading vehicle and output information indicative of such a risk of collision. In some embodiments, the collision prediction unit 218 may optionally obtain sensor information indicative of a state of the vehicle, such as a vehicle speed, a vehicle acceleration, a vehicle turning angle, a vehicle turning direction, a vehicle braking, a vehicle driving direction, an engine state, a wheel traction, a turn signal state, an accelerator pedal position, a brake pedal position, or any combination thereof. In such a case, the collision prediction unit 218 may be configured to determine a risk of collision based on the output from the object detection unit 216 and based on the obtained sensor information. Also, in some embodiments, the collision prediction unit 218 may be configured to determine a relative speed between the target vehicle and an object (e.g., a leading vehicle) and determine that there is a risk of collision based on the determined relative speed. In some embodiments, the collision prediction unit 218 may be configured to determine the speed of the target vehicle, the speed of the moving object, the driving path of the target vehicle, and the moving path of the moving object, and determine that there is a risk of collision based on these parameters. For example, if the object is moving along a path that intersects with the path of the target vehicle, and the time until the object and the target vehicle collide based on their respective speeds is less than a time threshold, the collision prediction unit 218 may determine that there is a risk of collision. Note that objects that may collide with the target vehicle are not limited to moving objects (such as automobiles, motorcycles, bicycles, pedestrians, animals, etc.), and the collision prediction unit 218 may be configured to determine the risk of collision with non-moving objects such as parked automobiles, road signs, utility poles, buildings, trees, and mailboxes.

[0211]

[0256] In some embodiments, the collision prediction unit 218 may be configured to determine the time it will take for a predicted collision to occur and compare the time to a threshold time. If the time is less than the threshold time, the collision prediction unit 218 may determine that there is a risk of collision with the target vehicle. In some embodiments, the threshold time for identifying the risk of collision may be at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 seconds, or more. In some embodiments, when predicting a collision, the collision prediction unit 218 may consider the speed, acceleration, driving direction, braking operation, or any combination thereof, of the target vehicle. Optionally, the collision prediction unit 218 may also consider the speed, acceleration, driving direction, or any combination thereof, of a detected object predicted to collide with the vehicle.

[0212]

[0257] The intersection violation predictor 222 is configured to determine a risk of an intersection violation based on the output from the object detector 216. For example, in some embodiments, the intersection violation predictor 222 may determine that there is a risk that the subject vehicle will fail to stop at a target area associated with a stop sign or a red light, and output information indicative of such a risk of intersection violation. In some embodiments, the intersection violation predictor 222 may also optionally obtain sensor information indicative of a state of the vehicle, such as a vehicle speed, a vehicle acceleration, a vehicle turning angle, a vehicle turning direction, a vehicle braking, or any combination thereof. In such a case, the intersection violation predictor 222 may be configured to determine a risk of an intersection violation based on the output from the object detector 216 and based on the obtained sensor information.

[0213]

[0258] In some embodiments, the intersection violation predictor 222 may be configured to identify a target area (e.g., a stop line) where the subject vehicle is expected to stop, determine a distance between the subject vehicle and the target area, and compare the distance to a threshold distance. If the distance is less than the threshold distance, the intersection violation predictor 222 may determine that there is a risk of an intersection violation.

[0214]

[0259] In other embodiments, the intersection violation predictor 222 may be configured to identify a target area (e.g., a stop line) where the subject vehicle is expected to stop, determine the time it takes for the vehicle to reach the target area, and compare the time to a threshold time. If the time is less than the threshold time, the intersection violation predictor 222 may determine that there is a risk of an intersection violation. In some embodiments, the threshold time for identifying a risk of an intersection violation may be at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 seconds, or more. In some embodiments, when predicting an intersection violation, the intersection violation predictor 222 may consider the speed, acceleration, driving direction, braking operation, or any combination thereof, of the subject vehicle.

[0215]

[0260] It should be noted that intersection violations are not limited to stop sign violations or red light violations, and the intersection violation prediction unit 222 may be configured to determine the risk of other intersection violations, such as a vehicle entering a wrong-way lane or a vehicle turning at an intersection with a "no turn on red" sign.

[0216]

[0261] In some embodiments, the signal generation controller 224 is configured to determine whether to generate a control signal based on the output from the collision predictor 218 and the output from the driver monitoring module. Alternatively or additionally, the signal generation controller 224 is configured to determine whether to generate a control signal based on the output from the intersection violation predictor 222, and optionally also based on the output from the driver monitoring module. In some embodiments, the signal generation controller 224 is configured to determine whether to generate a control signal based also on sensor information provided by one or more sensors of the vehicle.

[0217]

[0262] In some embodiments, the control signal is configured to cause a device (e.g., a warning generator) to provide a warning to the driver when the estimated time until the predicted crash occurs is less than a threshold (action threshold). For example, the warning generator may output an audio signal, a visual signal, a mechanical vibration (steering wheel vibration), or any combination thereof to alert the driver. Alternatively or additionally, the control signal is configured to cause a device (e.g., a vehicle controller) to control the vehicle when the estimated time until the predicted crash occurs is less than or equal to the threshold (action threshold). For example, the vehicle controller may automatically brake the vehicle, automatically release the accelerator pedal, automatically activate the hazard lights, automatically steer the vehicle, or any combination thereof. Thus, in some embodiments, the control signal may be provided to a braking system (e.g., an automatic emergency braking system), a steering system, a lane control system, a level 3 automation system, etc., or any combination thereof. In some embodiments, the signal generation controller 224 may be configured to provide a first control signal to provide a warning to the driver. If the driver does not take any action to mitigate the risk of a collision, the signal generation controller 224 may then provide a second control signal to cause the vehicle control device to control the vehicle, such as automatically applying the vehicle brakes.

[0218]

[0263] In some embodiments, the signal generation controller 224 may be a separate component (e.g., module) from the collision predictor 218 and the intersection violation predictor 222. In other embodiments, the signal generation controller 224 or at least a portion of the signal generation controller 224 may be implemented as part of the collision predictor 218 and / or the intersection violation predictor 222. Also, in some embodiments, the collision predictor 218 and the intersection violation predictor 222 may be integrated.

[0219]

[0264] In operation, the device 200 is coupled to a vehicle such that the first camera 202 views the exterior of the vehicle and the second camera 204 views the driver inside the vehicle. While the driver is operating the vehicle, the first camera 202 takes images of the exterior of the vehicle and the second camera 204 takes images of the interior of the vehicle. FIG. 2B illustrates an example of a processing scheme for the device 200. As shown, during use of the device 200, the second camera 204 provides images as input to a driver monitoring module 211. The driver monitoring module 211 analyzes the images to identify one or more poses of the driver of the subject vehicle. By way of non-limiting example, the one or more poses may include one or more of the following: a pose looking down, a pose looking up, a pose looking left, a pose looking right, a pose using a mobile phone, a pose smoking, a pose holding an object, a pose with hands off the steering wheel, a pose not wearing a seat belt, a pose with eyes closed, a pose looking ahead, a pose with one hand on the steering wheel, and a pose with both hands on the steering wheel. In some embodiments, the driver monitoring module 211 may be configured to determine one or more states of the driver based on the identified driver pose. For example, the driver monitoring module 211 may determine whether the driver is distracted based on the identified driver pose. As another example, the driver monitoring module 211 may determine whether the driver is drowsy based on the identified driver pose. In some embodiments, the driver monitoring module 211 may determine that the driver is distracted if the driver is in a particular pose (e.g., a pose using a cell phone). Also, in some embodiments, the driver monitoring module 211 may analyze a series of pose classifications of the driver over a period of time to determine whether the driver is drowsy.

[0220]

[0265] The first camera 202 provides an image as an input to the object detector 216, which analyzes the image to detect one or more objects in the image. As shown, the object detector 216 includes different detectors configured to detect different types of objects. In particular, the object detector 216 has a vehicle detector 260 configured to detect vehicles external to the target vehicle, a fragile object detector 262 configured to detect fragile objects such as humans, bicycles with riders, animals, and an intersection detector 264 configured to detect one or more items (e.g., stop signs, traffic lights, crosswalk markings, etc.) for identifying an intersection. In some embodiments, the object detector 216 may be configured to determine different types of objects based on different models. For example, there may be a vehicle detection model configured to detect vehicles, a human detection model configured to detect humans, an animal detection model configured to detect animals, a traffic light detection model configured to detect traffic lights, a stop sign detection model configured to detect stop signs, a centerline detection model configured to detect a centerline of a road, etc. In some embodiments, the different models may be different respective neural network models trained to detect different types of objects.

[0221]

[0266] The vehicle detection unit 260 is configured to detect vehicles outside the target vehicle and provide information about the detected vehicles (e.g., vehicle identifier, vehicle location, etc.) to the module 221. The module 221 may be the collision prediction unit 218 and / or the intersection violation prediction unit 222. The module 221 includes an object tracker 266 configured to track one or more of the detected vehicles, a course predictor 268 configured to identify a course of a predicted collision, and a time to collision / crossing (TTC) module 269 configured to estimate a time until a predicted collision occurs. In some embodiments, the object tracker 266 is configured to identify a leading vehicle traveling ahead of the target vehicle. Also, in some embodiments, the course predictor 268 is configured to determine a course of a predicted collision based on the identified leading vehicle and sensor information from the sensor 225. For example, based on the speed of the target vehicle and the traveling direction of the target vehicle, the course predictor 268 can identify a course of a predicted collision. The TTC module 269 is configured to calculate a time to collision based on the information about the predicted collision course and the sensor information from the sensor 225. For example, the TTC module 269 may calculate a time-to-collision (TTC) based on the distance of the collision course and the relative speed of the leading vehicle and the target vehicle. The signal generation controller 224 is configured to determine whether to generate a control signal to activate a warning generator to warn the driver and / or activate a vehicle control device to control the vehicle (e.g., automatically release the accelerator operation, apply the brakes, etc.) based on the output from the module 221 and the output from the TTC module 269. In some embodiments, if the TTC is less than a threshold value (e.g., 3 seconds), the signal generation controller 224 generates a control signal to operate the warning generator and / or the vehicle control device. Also, in some embodiments, the threshold value may be adjustable based on the output from the driver monitoring module 211.For example, if the output from the driver monitoring module 211 indicates that the driver is distracted or not paying attention to the driving task, the signal generation controller 224 may increase the threshold (e.g., to 5 seconds) such that the signal generation controller 224 provides a control signal when the TTC with the preceding vehicle is less than 5 seconds.

[0222]

[0267] It should be noted that the module 221 is not limited to predicting a collision with a leading vehicle, but may also be configured to predict a collision with another vehicle. For example, in some embodiments, the module 221 may be configured to detect vehicles traveling toward the path of the target vehicle, such as vehicles approaching an intersection, vehicles merging toward the lane of the target vehicle, etc. In such a situation, the course prediction unit 268 identifies the path of the target vehicle and the path of the other vehicle, and further identifies an intersection of the two paths. The TTC module 269 is configured to determine the TTC based on the location of the intersection, the speed of the other vehicle, and the speed of the target vehicle.

[0223]

[0268] The fragile object detector 262 is configured to detect fragile objects outside the target vehicle and provide information about the detected objects (e.g., object identifiers, object locations, etc.) to the module 221. For example, the fragile object detector 262 may detect a human outside the target vehicle and provide information about the detected human to the module 221. The module 221 includes an object tracker 266 configured to track one or more detected objects (e.g., humans), a course predictor 268 configured to identify a predicted collision course, and a time to collision / crossing (TTC) module 269 configured to estimate a time until an estimated collision occurs. Because certain objects, such as humans, animals, bicycles, etc., may have unpredictable directions of movement, in some embodiments, the course predictor 268 is configured to determine a box surrounding an image of the detected object to indicate the possible location of the object. In some embodiments, the course predictor 268 is configured to identify a predicted collision course based on a box surrounding an identified object (e.g., human) and sensor information from the sensor 225. For example, based on the speed of the target vehicle, the direction of travel of the target vehicle, and the box enclosing the identified object, the course prediction unit 268 may determine that the current driving path of the target vehicle intersects with the box. In this case, the predicted collision course is the driving path of the target vehicle, and the predicted collision location is the intersection of the driving path of the target vehicle and the box enclosing the object. The TTC module 269 is configured to calculate the predicted time to collision based on the information on the predicted collision course and the sensor information from the sensor 225. For example, the TTC module 269 may calculate the time-to-collision (TTC) based on the distance of the collision course and the relative speed of the preceding vehicle and the person. The signal generation controller 224 is configured to determine whether to generate a control signal to activate a warning generator to warn the driver and / or activate a vehicle control device to control the vehicle (e.g., automatically release the accelerator operation, apply the brakes, etc.) based on the output from the module 221 and the output from the TTC module 269.In some embodiments, if the TTC is less than a threshold (e.g., 3 seconds), the signal generation controller 224 generates a control signal to operate a warning generator and / or a vehicle control device. Also, in some embodiments, the threshold may be adjustable based on the output from the driver monitoring module 211. For example, if the output from the driver monitoring module 211 indicates that the driver is distracted or not paying attention to the driving task, the signal generation controller 224 may increase the threshold (e.g., to 5 seconds). This causes the signal generation controller 224 to provide a control signal when the TTC with the object is less than 5 seconds.

[0224]

[0269] It should be noted that the module 221 is not limited to predicting collisions with humans, but may be configured to predict collisions with other objects. For example, in some embodiments, the module 221 may be configured to detect animals, bicyclists, roller skaters, skateboarders, etc. In such situations, the course predictor 268 may be configured to determine the path of the target vehicle, as well as the path of the detected object (if the object is moving in one direction, such as a bicycle), and further determine the intersection of the two paths. In other cases, when the object's movement is more unpredictable (such as an animal), the course predictor 268 may determine the path of the target vehicle and a box encompassing a range of possible positions of the object, as well as determine the intersection of the path of the target vehicle and the box. The TTC module 269 is configured to determine the TTC based on the location of the intersection and the speed of the target vehicle.

[0225]

[0270] The intersection detector 264 is configured to detect one or more objects indicative of an intersection outside the target vehicle and provide information about the intersection (such as the type of intersection, a stopping position required for the vehicle, etc.) to the module 221. As a non-limiting example, the one or more objects indicative of the intersection may include a traffic light, a stop sign, a road sign, etc., or any combination thereof. Also, the intersection that the intersection detector 264 can detect may be a stop sign intersection, a signalized intersection, an intersection with a railroad, etc. The module 221 includes a course predictor 268 configured to identify a course of a predicted intersection violation, and a TTC (Time to Collision / Cross) module 269 configured to estimate a time until the predicted intersection violation occurs. The TTC module 269 is configured to calculate a time until the intersection violation is estimated to occur based on a stopping position required for the vehicle and sensor information from the sensor 225. For example, the TTC module 269 may calculate the TTC (Time to Cross) based on a distance of the course (e.g., a distance between a current position of the vehicle and a stopping position required for the vehicle) and a speed of the target vehicle. In some embodiments, the required stopping position may be determined by the object detection unit 216 detecting a stop line marking on the road. In other embodiments, there may be no stop line marking on the road. In such a case, the course prediction unit 268 may determine a virtual or graphical line indicating the required stopping position. The signal generation controller 224 is configured to determine whether to generate a control signal to activate a warning generator to warn the driver and / or activate a vehicle control device to control the vehicle (e.g., automatically release the accelerator operation, apply the brakes, etc.) based on the output from the module 221 and the output from the TTC module 269. In some embodiments, if the TTC is less than a threshold value (e.g., 3 seconds), the signal generation controller 224 generates a control signal to operate the warning generator and / or the vehicle control device. Also, in some embodiments, the threshold value may be adjustable based on the output from the driver monitoring module 211.For example, if the output from the driver monitoring module 211 indicates that the driver is distracted or not paying attention to the driving task, the signal generation controller 224 may increase the threshold (e.g., to 5 seconds) such that the signal generation controller 224 provides a control signal when the time to clear the intersection is less than 5 seconds.

[0226]

[0271] In some embodiments, with respect to a predicted collision with another vehicle, a predicted collision with an object, or a predicted intersection violation, the signal generation controller 224 may be configured to apply different values ​​of the threshold for generating a control signal based on the type of driver state indicated by the output of the driver monitoring module 211. For example, if the output of the driver monitoring module 211 indicates that the driver is looking at a cell phone, the signal generation controller 224 may generate a control signal to activate a warning generator and / or a control signal to activate a vehicle control device in response to the TTC meeting or falling below a 5-second threshold. On the other hand, if the output of the driver monitoring module 211 indicates that the driver is drowsy, the signal generation controller 224 may generate a control signal to activate a warning generator and / or a control signal to activate a vehicle control device in response to the TTC falling below an 8-second threshold (e.g., longer than the threshold for when the driver is using a cell phone). In some cases, a driver's state (e.g., the driver is drowsy or asleep) may require a longer threshold time (relative to the TTC value) to alert the driver or take control of the vehicle, as the driver may need more time to react to an impending collision. Thus, the signal generation controller 224 may provide a warning to the driver and / or take earlier action on vehicle controls in response to a predicted collision in such circumstances.

[0227]

[0272] Driver status determination

[0228]

[0273] As described herein, the second camera 204 is configured to view the driver inside the vehicle. While the driver is operating the vehicle, the first camera 202 captures images of the exterior of the vehicle and the second camera 204 captures images of the interior of the vehicle. FIG. 3 illustrates an example of an image 300 captured by the second camera 204 of the device 200 of FIG. 2. As illustrated, the image 300 from the second camera 202 may include an image of a driver 310 driving a target vehicle (the vehicle equipped with the device 200). The processing unit 210 is configured to process the image (e.g., image 300) from the camera 202 and determine whether the driver is engaged in a driving task. As a non-limiting example, the driving task may be paying attention to the road or environment ahead of the target vehicle, keeping hands on the steering wheel, etc.

[0229]

[0274] As shown in FIG. 4, in some embodiments, the processing unit 210 is configured to process the image 300 of the driver from the camera 202 and determine whether the driver belongs to a particular pose classification. As non-limiting examples, the pose classifications include one or more of the following: looking down pose, looking up pose, looking left pose, looking right pose, using a mobile phone pose, smoking pose, holding an object pose, hands off the steering wheel pose, not wearing a seat belt pose, eyes closed pose, looking ahead pose, one hand on the steering wheel pose, and both hands on the steering wheel pose. Also, in some embodiments, the processing unit 210 is configured to determine whether the driver is engaged in a driving task based on the one or more pose classifications. For example, if the driver's head is "looking down" and the driver is holding a mobile phone, the processing unit 210 may determine that the driver is not engaged in a driving task (i.e., the driver is not paying attention to the road or the environment in front of the vehicle). As another example, if the driver's head is "looking" to the right or left and the head turning angle exceeds a certain threshold, the processing unit 210 may determine that the driver is not engaged in a driving task.

[0230]

[0275] In some embodiments, the processing unit 210 is configured to determine whether the driver is engaged in a driving task based on one or more poses of the driver displayed in the image, without the need to determine the driver's eye gaze direction. This feature is advantageous because the driver's eye gaze direction may not be visible in the image or may not be accurately determined. For example, the driver may be wearing a hat, which may prevent the dash camera from seeing the driver's eyes. The driver may also be wearing sunglasses that obstruct the view of the eyes. In some cases, the driver may be wearing clear glasses, in which case the frames of the glasses may obstruct the view of the eyes, or the lenses of the glasses may cause the detection of the eyes to be inaccurate. Thus, determining whether the driver is engaged in a driving task without the need to determine the driver's eye gaze direction is advantageous because the processing unit 210 can determine whether the driver is engaged in a driving task even when the driver's eyes cannot be detected and / or the eye gaze direction cannot be determined.

[0231]

[0276] In some embodiments, the processing unit 210 may use context-based classification to determine whether the driver is engaged in a driving task. For example, if the driver's head is facing down and the driver is holding a cell phone on his / her lap toward which the driver's head is facing, the processing unit 210 may determine that the driver is not engaged in a driving task. The processing unit 210 may make such a determination even if the driver's eyes cannot be detected (e.g., obscured by a hat as shown in FIG. 3). The processing unit 210 may also use the context-based classification to identify one or more poses of the driver. For example, if the driver's head is facing down, the processing unit 210 may determine that the driver is looking down even if the driver's eyes cannot be detected. As another example, if the driver's head is facing up, the processing unit 210 may determine that the driver is looking upward even if the driver's eyes cannot be detected. As a further example, if the driver's head is facing right, the processing unit 210 may determine that the driver is looking right even if the driver's eyes cannot be detected. As a further example, if the driver's head is turned to the left, processing unit 210 may determine that the driver is looking to the left even if the driver's eyes cannot be detected.

[0232]

[0277] In one embodiment, processing unit 210 may be configured to use the model to identify one or more poses of the driver and determine whether the driver is engaged in a driving task. The model may be used by processing unit 210 to process images from camera 204. In some embodiments, the model may be stored in non-transitory medium 230. Also, in some embodiments, the model may be transmitted from a server and received by device 200 via communication unit 240.

[0233]

[0278] In some embodiments, the model may be a neural network model. In such a case, the neural network model may be trained based on images of other drivers. For example, the neural network model may be trained using images of drivers to identify various poses such as looking down, looking up, looking left, looking right, using a mobile phone, smoking, holding an object, hands off the wheel, not wearing a seat belt, eyes closed, looking ahead, one hand on the wheel, both hands on the wheel, etc. In some embodiments, the neural network model may be trained to identify different poses even without detection of the person's eyes in the image. This allows the neural network model to identify various poses and / or determine whether the driver is engaged in a driving task based on the context (e.g., based on information captured in the image about the driver's state other than the driver's eye gaze direction). In other embodiments, the model may be any of other types of models different from a neural network model.

[0234]

[0279] In some embodiments, the neural network model may be trained to classify poses based on context and / or determine whether a driver is engaged in a driving task. For example, if a driver is holding a mobile phone and has a head pose that is pointing down toward the mobile phone, the neural network model may determine that the driver is not engaged in a driving task (e.g., not looking at the road or the environment in front of the vehicle) without having to detect the driver's eyes.

[0235]

[0280] In some embodiments, deep learning or artificial intelligence can be used to develop models that identify the pose of the driver and / or determine whether the driver is engaged in the driving task. Such models can distinguish between drivers who are engaged in the driving task and drivers who are not.

[0236]

[0281] In some embodiments, the model utilized by processing unit 210 to identify the driver's pose may be a convolutional neural network model, while in other embodiments the model may simply be any mathematical model.

[0237]

[0282] 5 illustrates an algorithm 500 for determining whether a driver is engaged in a driving task. For example, the algorithm 500 may be utilized to determine whether a driver is paying attention to the road and environment ahead of the vehicle. The algorithm 500 may be implemented and / or executed using the processing unit 210 in some embodiments.

[0238]

[0283] First, the processing unit 210 processes an image from the camera 204 and attempts to detect the driver's face based on the image (item 502). If the driver's face cannot be detected from the image, the processing unit 210 may determine that it is unclear whether the driver is engaged in a driving task. On the other hand, if the processing unit 210 determines that the driver's face is present in the image, the processing unit 210 may determine that the driver's eyes are closed (item 504). In one embodiment, the processing unit 210 may be configured to determine eye visibility based on a model, such as a neural network model. If the processing unit 210 determines that the driver's eyes are closed, the processing unit 210 may determine that the driver is not engaged in a driving task. On the other hand, if the processing unit 210 determines that the driver's eyes are not closed, the processing unit 210 may attempt to detect the driver's eye gaze based on the image (item 506).

[0239]

[0284] Referring to item 510 of algorithm 500, if processing unit 210 successfully detects the gaze of the driver's eyes, processing unit 210 may then determine the gaze direction (item 510). For example, processing unit 210 may analyze the image to determine the pitch (e.g., up and down) and / or yaw (e.g., left and right) of the gaze direction of the driver's eyes. If the gaze direction pitch is within a predetermined pitch range and the gaze direction yaw is within a predetermined yaw range, processing unit 210 may determine that the user is engaged in a driving task (i.e., the user is looking at the road or environment in front of the vehicle) (item 512). On the other hand, if the gaze direction pitch is not within the predetermined pitch range or the gaze direction yaw is not within the predetermined yaw range, processing unit 210 may determine that the user is not engaged in a driving task (item 514).

[0240]

[0285] Referring to item 520 of algorithm 500, if processing unit 210 is unable to successfully detect the driver's eye gaze, processing unit 210 may then determine whether the driver is engaged in a driving task without needing to determine the driver's eye gaze direction (item 520). In some embodiments, processing unit 210 may be configured to make such a determination using a model based on the context (e.g., based on information captured in the image related to the driver's state other than the driver's eye gaze direction). In some embodiments, the model may be a neural network model configured to perform context-based classification to determine whether the driver is engaged in a driving task. In one embodiment, the model is configured to process the image to determine whether the driver belongs to one or more pose classifications. If it is determined that the driver belongs to one or more pose classifications, processing unit 210 may determine that the driver is not engaged in a driving task (item 522). If it is determined that the driver does not belong to one or more pose classifications, processing unit 210 may determine that the driver is engaged in a driving task or determine that it is unclear whether the driver is engaged in a driving task (item 524).

[0241]

[0286] In some embodiments, the above items 502, 504, 506, 510, 520 may be executed repeatedly by the processing unit 210 to process multiple images of a sequence provided by the camera 204, thereby performing real-time monitoring of the driver while the driver is operating the vehicle.

[0242]

[0287] It should be noted that algorithm 500 is not limited to the described examples, and algorithm 500 implemented using processing unit 210 may have other features and / or variations. For example, in other embodiments, algorithm 500 may not include item 502 (detecting the driver's face). As another example, in other embodiments, algorithm 500 may not include item 504 (detecting closed eyes). Also, in further embodiments, algorithm 500 may not include item 506 (attempting to detect gaze) and / or item 510 (determining gaze direction).

[0243]

[0288] Also, in some embodiments, even if the driver's eye gaze direction can be detected by processing unit 210, processing unit 210 may perform context-based classification to determine whether the driver belongs to one or more poses. In some cases, processing unit 210 may use the pose classification to ascertain the driver's eye gaze direction. Alternatively, the driver's eye gaze direction may be used by processing unit 210 to ascertain one or more pose classifications for the driver.

[0244]

[0289] As described, in some embodiments, the processing unit 210 is configured to determine whether the driver belongs to one or more pose classifications based on the image from the camera 204, and determine whether the driver is engaged in a driving task based on the one or more pose classifications. In some embodiments, the processing unit 210 is configured to determine a metric value for each of a plurality of pose classifications, and determine whether the driver is engaged in a driving task based on the one or more metric values. FIG. 6 illustrates an example of a classification output 602 provided by the processing unit 210 based on the image 604a. In this example, the classification output 602 includes metric values ​​for each of the different pose classifications: "looking down", "looking up", "looking left", "looking right", "using mobile phone", "smoking", "holding object", "eyes closed", "no face", and "seatbelt not fastened". The metric values ​​for these different pose classifications are relatively low (e.g., below 0.2), indicating that the driver in the image 604a does not fit any of these pose classifications. Also, in the illustrated example, the driver's eyes are not closed, allowing the processing unit 210 to determine the driver's gaze direction. The gaze direction is represented by a graphical object superimposed on the driver's nose in the image. The graphical object may include a vector or a line parallel to the gaze direction. Alternatively or additionally, the graphical object may include one or more vectors or one or more lines perpendicular to the gaze direction.

[0245]

[0290] 7 illustrates another example of a classification output 602 provided by processing unit 210 based on image 604b. In the illustrated example, the metric value for the "look down" pose is relatively high (e.g., greater than 0.6), indicating that the driver is in a "look down" pose. The metric values ​​for the other poses are relatively low, indicating that the driver in image 604b does not meet these pose classifications.

[0246]

[0291] 8 illustrates another example of a classification output 602 provided by processing unit 210 based on image 604c. In the illustrated example, the metric value for the "Look Left" pose is relatively high (e.g., greater than 0.6), indicating that the driver is in a "Look Left" pose. The metric values ​​for the other poses are relatively low, indicating that the driver in image 604c does not meet these pose classifications.

[0247]

[0292] In some embodiments, the processing unit 210 is configured to compare the metric values ​​to respective thresholds for each pose classification. In such a case, the processing unit 210 is configured to determine that the driver belongs to one of the pose classifications if the corresponding one of the metric values ​​meets or exceeds the corresponding one of the thresholds. For example, the thresholds for the different pose classifications may be set to 0.6. In such a case, if any of the metric values ​​of any pose classification exceeds 0.6, the processing unit 210 may determine that the driver has a pose that belongs to that pose classification (i.e., a pose with a metric value exceeding 0.6). Also, in some embodiments, if any of the metric values ​​of any pose classification exceeds a pre-set threshold (e.g., 0.6), the processing unit 210 may determine that the driver is not engaged in a driving task. Following the above example, if the metric values ​​of the "looking down" pose, the "looking up" pose, the "looking left" pose, the "looking right" pose, the "using mobile phone" pose, or the "eyes closed" pose are higher than 0.6, the processing unit 210 may determine that the driver is not engaged in a driving task.

[0248]

[0293] In the above example, the same pre-set thresholds are implemented for each of the different pose classifications. In other embodiments, at least two of the thresholds for at least two respective pose classifications may have different values. Also, in the above example, the metric values ​​for the pose classifications range from 0.0 to 1.0, with 1.0 being the highest. In other embodiments, the metric values ​​for the pose classifications may have other ranges. Also, in other embodiments, the metric value rule may be reversed, with a lower metric value indicating that the driver is more compatible with the particular pose classification and a higher metric value indicating that the driver is less compatible with the particular pose classification.

[0249]

[0294] Also, in some embodiments, the thresholds of different pose classes may be adjusted in a tuning procedure, such that different pose classes have respective adjusted thresholds such that processing unit 210 can determine whether an image of a driver belongs to a particular pose class or not.

[0250]

[0295] In some embodiments, a single model may be utilized by the processing unit 210 to provide multiple pose classifications. The processing unit 210 may output multiple pose classifications in parallel or sequentially. In other embodiments, the model may include multiple sub-models, each configured to detect a particular classification of poses. For example, there may be a sub-model for detecting faces, a sub-model for detecting gaze direction, a sub-model for detecting upturned poses, a sub-model for detecting downturned poses, a sub-model for detecting rightward poses, a sub-model for detecting leftward poses, a sub-model for detecting mobile phone use poses, a sub-model for detecting no-hands-on-the-wheel poses, a sub-model for detecting no-seat-belt poses, a sub-model for detecting eyes-closed poses, etc.

[0251]

[0296] In the above embodiment, the thresholds of each pose classification are configured to determine whether the image of the driver falls into the respective pose classification. In other embodiments, the thresholds of each pose classification may be configured to enable the processing unit 210 to determine whether the driver is engaged in the driving task. In such a case, if one or more metric values ​​of one or more respective pose classifications meet or exceed the respective one or more thresholds, the processing unit 210 may determine whether the driver is engaged in the driving task. In some embodiments, the pose classification may belong to a "distracted" class. In such a case, if the criteria of any pose classification are met, the processing unit 210 may determine that the driver is not engaged in the driving task (e.g., the driver is distracted). Examples of pose classifications that belong to "distracted" include a "looking left" pose, a "looking right" pose, a "looking up" pose, a "looking down" pose, a "holding a mobile phone" pose, and the like. In other embodiments, the pose classification may belong to an "attention" class. In such a case, if the criteria of any pose classification are met, the processing unit 210 may determine that the driver is engaged in the driving task (e.g., the driver is paying attention to driving). Examples of poses that fall into the "attention" class include the "looking ahead" pose and the "hands on the steering wheel" pose.

[0252]

[0297] As shown in the above examples, context-based classification is advantageous because it allows the processing unit 210 to identify a driver who is not engaged in a driving task even when the driver's eye gaze direction cannot be detected. In some cases, context-based identification allows the processing unit 210 to identify a driver who is not engaged in a driving task even when the device 200 is mounted at a significantly off-angle relative to the vehicle (resulting in the driver appearing at an odd angle and / or position in the camera image). Aftermarket products may be mounted in different locations, making eye and gaze detection difficult. The features described herein are advantageous in that they allow the determination of whether a driver is engaged in a driving task even when the device 200 is mounted in a way that does not allow the driver's eyes and gaze to be detected.

[0253]

[0298] It should be noted that the processing unit 210 is not limited to using a neural network model for pose classification and / or determining whether the driver is engaged in a driving task, and the processing unit 210 may utilize any processing technique, algorithm, or processing architecture for pose classification and / or determining whether the driver is engaged in a driving task. By way of non-limiting example, the processing unit 210 may use equations, regression, classification, neural networks (e.g., convolutional neural networks, deep neural networks), heuristics, selection (e.g., from libraries, graphs, or charts), instance-based methods (e.g., nearest neighbor methods), correlation methods, regularization methods (e.g., ridge regression), decision trees, Bayesian methods, kernel methods, probability, determinism, or a combination of two or more thereof, to process images from the camera 204 for pose classification and / or determining whether the driver is engaged in a driving task. The pose classification can be a binary classification or score (e.g., looking up or not), a score (e.g., continuous or discontinuous), a classification (e.g., high, medium, low), or any other suitable measure of pose classification.

[0254]

[0299] Also, it should be noted that processing unit 210 is not limited to detecting pauses that indicate the driver is not engaged in the driving task (e.g., pauses that belong to the “distracted” class). In other embodiments, processing unit 210 may be configured to detect pauses that indicate the driver is engaged in the driving task (e.g., pauses that belong to the “attention” class). In further embodiments, processing unit 210 may be configured to detect both (1) pauses that indicate the driver is not engaged in the driving task and (2) pauses that indicate the driver is engaged in the driving task.

[0255]

[0300] In one or more embodiments described herein, the processing unit 210 may be further configured to determine the risk of collision based on whether the driver is engaged in the driving task. In some embodiments, the processing unit 210 may be configured to determine the risk of collision based only on whether the driver is engaged in the driving task. For example, the processing unit 210 may determine the risk of collision to be “high” when the driver is not engaged in the driving task and determine the risk of collision to be “low” when the driver is engaged in the driving task. In other embodiments, the processing unit 210 may be configured to determine the risk of collision based on additional information. For example, the processing unit 210 may be configured to track the time the driver is not engaged in the driving task and determine the level of risk of collision based on the duration of the “not engaged in the driving task” state. As another example, the processing unit 210 may process images from the first camera 202 to determine whether there is an obstacle (e.g., a vehicle, a pedestrian, etc.) in front of the target vehicle and determine the risk of collision based on the detection of such obstacle in combination with the pose classification.

[0256]

[0301] In the above embodiments, camera images from camera 204 (which views the environment inside the vehicle cabin) are utilized to monitor the driver's engagement in the driving task. In other embodiments, camera images from camera 202 (which views the vehicle's external environment) may be utilized as well. For example, in some embodiments, camera images capturing the vehicle's external environment may be processed by processing unit 210 to determine whether the vehicle is turning left, going straight, or turning right. Based on the vehicle's direction of travel, processing unit 210 may adjust one or more thresholds for the driver's pose classification and / or one or more thresholds for determining whether the driver is engaged in the driving task. For example, if processing unit 210 determines (based on processing images from camera 202) that the vehicle is turning left, processing unit 210 may adjust the threshold for the "looking left" pose classification such that a driver looking left is not classified as not engaged in the driving task. In one implementation, the threshold for the "looking left" pose classification may have a value of 0.6 for a straight-moving vehicle and a value of 0.9 for a left-turning vehicle. In such a case, if the processing unit 210 determines (based on processing of images from camera 202) that the vehicle is going straight and determines (based on processing of images from camera 204) that the metric value of the "looking left" pose is 0.7, the processing unit 210 can determine that the driver is not engaged in the driving task (because the value of the metric, 0.7, exceeds the threshold value of 0.6 for a straight-going vehicle). On the other hand, if the processing unit 210 determines (based on processing of images from camera 202) that the vehicle is turning left and determines (based on processing of images from camera 204) that the metric value of the "looking left" pose is 0.7, the processing unit 210 can determine that the driver is engaged in the driving task (because the value of the metric, 0.7, does not exceed the threshold value of 0.9 for a left-turning vehicle). Thus, as shown in the above example, a certain pose classification (e.g., the "looking left" pose) may belong to the "distracted" class in some situations and the "attention" class in other situations.In some embodiments, the processing unit 210 is configured to process images of the external environment from the camera 202 to obtain an output and adjust one or more thresholds based on the output. By way of non-limiting examples, the output may be a classification of a driving state, a classification of the external environment, identified features of the environment, a context of the operation of the vehicle, etc.

[0257]

[0302] Drowsiness Detection

[0303] In some embodiments, processing unit 210 may be configured to process images from camera 204 (e.g., image 300) and determine whether the driver is drowsy based on processing of the images. In some embodiments, processing unit 210 may also process images from camera 204 to determine whether the driver is distracted. In further embodiments, processing unit 210 may process images from camera 202 to determine the risk of a collision.

[0258]

[0304] In some embodiments, the driver monitoring module 211 of the processing unit 210 may include a first model and a second model configured to operate together to detect driver drowsiness. FIG. 9 illustrates an example of a processing architecture with a first model 212 and a second model 214 coupled in series. The first and second models 212, 214 may be considered to be within the processing unit 210 and / or part of the processing unit 210 (e.g., part of the driver monitoring module 211). Although the models 212, 214 are shown diagrammatically within the processing unit 210, in some embodiments, the models 212, 214 may be stored in the non-transitory medium 230. In such cases, the models 212, 214 may still be considered part of the processing unit 210. As shown in this example, a series of images 400a-400e from the camera 204 is received by the processing unit 210. The first model 212 of the processing unit 210 is configured to process the images 400a-400e. In some embodiments, the first model 212 is configured to identify one or more poses for a corresponding one of the images 400a-400e. For example, the first model 212 may analyze the image 400a and determine that the driver is in an "eyes open" pose and a "head upright" pose. The first model 212 may analyze the image 400b and determine that the driver is in an "eyes closed" pose. The first model 212 may analyze the image 400c and determine that the driver is in an "eyes closed" pose and a "head down" pose. The first model 212 may analyze the image 400d and determine that the driver is in an "eyes closed" pose and a "head down" pose. The first model 212 may analyze the image 400e and determine that the driver is in an "eyes closed" pose and a "head upright" pose. Although only five images 400a-400e are shown, in other examples, the sequence of images received by the first model 212 may be five or more. In some embodiments, the camera 202 has a frame rate of at least 10 frames per second (e.g., 15 fps), and the first model 212 can continue to receive images from the camera 202 at that rate during operation of the vehicle by the driver.

[0259]

[0305] In some embodiments, the first model may be a single model utilized by the processing unit 210 to provide multiple pose classifications. The processing unit 210 may output multiple pose classifications in parallel or sequentially. In other embodiments, the first model includes multiple sub-models, each configured to detect a particular classification of pose. For example, there may be a sub-model to detect a face, a sub-model to detect head up poses, a sub-model to detect head down poses, a sub-model to detect eyes closed poses, a sub-model to detect head up poses, a sub-model to detect eyes open poses, etc.

[0260]

[0306] In some embodiments, the first model 212 of the processing unit 210 is configured to determine a metric value for each of a plurality of pose classifications. The first model 212 of the processing unit 210 is also configured to compare the metric value to a respective threshold value for each of the pose classifications. In such a case, the processing unit 210 is configured to determine that the driver belongs to one of the pose classifications if a corresponding one of the metric values ​​meets or exceeds a corresponding one of the threshold values. For example, the threshold values ​​for the different pose classifications may be set to 0.6. In such a case, the processing unit 210 may determine that the driver has a pose that belongs to the pose classification (i.e., a pose with a metric value exceeding 0.6) if any of the metric values ​​of any of the pose classifications exceeds 0.6.

[0261]

[0307] In the above example, the same pre-set thresholds are implemented for each of the different pose classifications. In other embodiments, at least two of the thresholds for at least two respective pose classifications may have different values. Also, in the above example, the metric values ​​for the pose classifications range from 0.0 to 1.0, with 1.0 being the highest. In other embodiments, the metric values ​​for the pose classifications may have other ranges. Also, in other embodiments, the metric value rule may be reversed, with a lower metric value indicating that the driver is more compatible with the particular pose classification and a higher metric value indicating that the driver is less compatible with the particular pose classification.

[0262]

[0308] As described, in some embodiments, the first model 212 is configured to process the image of the driver from the camera 204 and determine whether the driver belongs to a particular pose classification. The pose classification belongs to a "drowsy" class, where each pose classification may indicate a sign of drowsiness. As a non-limiting example, the pose classification of the "drowsy" class may be any one or more of a head-down pose, eyes-closed pose, or any other pose that helps determine whether the driver is feeling drowsy. Alternatively or additionally, the pose classification may belong to an "alert" class, where each pose classification may indicate a sign of alertness. As a non-limiting example, the pose classification may be any one or more of a pose that helps determine whether the driver is feeling drowsy, such as a pose using a mobile phone. In some embodiments, a particular pose may belong to both the "drowsy" class and the "alert" class. For example, a pose with the head upright and a pose with the eyes open may belong to both classes.

[0263]

[0309] As shown, the pose identification (or classification) may be output as feature information by the first model 212. The second model 214 takes the feature information from the first model 212 as input and processes the feature information to determine whether the driver is feeling drowsy. The second model 214 also generates an output indicative of whether the driver is feeling drowsy.

[0264]

[0310] In some embodiments, the feature information output by the first model 212 may be time series data. The time series data may be pose classifications of the driver in different images 400 at different respective times. In particular, as images are generated sequentially by the camera 204, the first model 212 sequentially processes the images one by one to identify a pose for each image. Once the pose classification for each image is identified by the first model 212, the identified pose classification for the image is output by the first model 212 as feature information. Thus, as images are received one by one by the first model 212, feature information for each image is also output by the first model 212 sequentially one by one.

[0265]

[0311] FIG. 10 illustrates an example of feature information received by the second model 214. As illustrated, the feature information includes a pose classification for each successive image, where "O" indicates that the driver is in an "eyes open" pose and "C" indicates that the driver is in an "eyes closed" pose. Once the second model 214 has acquired the sequence of feature information, the second model 214 analyzes the feature information to determine whether the driver is drowsy. In one embodiment, the second model 214 may be configured (e.g., programmed, created, trained, etc.) to analyze a pattern of the feature information and determine whether it is a pattern associated with drowsiness (e.g., a pattern indicative of drowsiness). For example, the second model 214 may be configured to determine either blink frequency, eye closure time, time it takes to close eyelids, PERCLOS (percentage of time eyes closed), or other metrics measuring or indicative of wakefulness or drowsiness based on the time series of feature information.

[0266]

[0312] In some embodiments, if the blink frequency has a value that exceeds a blink frequency threshold associated with drowsiness, processing unit 210 may determine that the driver is feeling drowsy.

[0267]

[0313] Alternatively or additionally, if the eye-closure time has a value that exceeds an eye-closure time threshold associated with drowsiness, the processing unit 210 may determine that the driver is feeling drowsy. A drowsy person may have a longer eye-closure time than a person who is awake.

[0268]

[0314] Alternatively or additionally, the processing unit 210 may determine that the driver is drowsy if the time taken to close the eyelids has a value that exceeds a time threshold associated with drowsiness. The time taken to close the eyelids is the time interval from when the eyes are substantially open (e.g., at least 80% open, at least 90% open, 100% open, etc.) to when the eyelids are substantially closed (e.g., at least 70% closed, at least 80% closed, at least 90% closed, 100% closed, etc.). This is an indicator of the speed of closing the eyelids. Drowsy people tend to close their eyelids slower than awake people.

[0269]

[0315] Alternatively or additionally, the processing unit 210 may determine that the driver is drowsy if PERCLOS (percentage of time eyes closed) has a value above a PERCLOS threshold associated with drowsiness. PERCLOS is a drowsiness index that indicates the percentage of time per minute the eyes are 80% or more closed. PERCLOS indicates the percentage of pupil closure over time and reflects slow eyelid closure rather than blinking.

[0270]

[0316] It should be noted that the feature information provided by the first model 212 to the second model 214 is not limited to the example of pose classification described in FIG. 10, and the feature information utilized by the second model 214 to detect drowsiness may include other pose classifications. FIG. 11 illustrates another example of feature information received by the second model 214. As illustrated, the feature information includes a pose classification for each successive image, where "S" indicates that the driver is in a "head upright" pose and "D" indicates that the driver is in a "head down" pose. Once the second model 214 has obtained the sequence of feature information, the second model 214 analyzes the feature information to determine whether the driver is drowsy. For example, if the classifications of "head upright" and "head down" poses are repeated in a certain pattern associated with drowsiness, the processing unit may determine that the driver is drowsy. In one embodiment, the second model 214 may be configured (e.g., programmed, created, trained, etc.) to analyze a pattern of characteristic information and determine whether it is a pattern associated with drowsiness (e.g., a pattern indicative of drowsiness).

[0271]

[0317] In some embodiments, the feature information provided from the first model 212 to the second model 214 may have a data structure that allows different pose classifications to be associated with different time points, and in some embodiments, such a data structure allows one or more pose classifications to be associated with a particular time point.

[0272]

[0318] Also, in some embodiments, the output of the first model 212 may be a numerical vector (e.g., a low-dimensional numerical vector, such as an embedding) that provides a numerical representation of the pose detected by the first model 212. The numerical vector may not be human interpretable, but may provide information about the detected pose. In other embodiments, the output of the first model 212 may be any information that indicates, represents, or relates to the external scene, IMU signals, audio signals, etc. Also, in some embodiments, the embedding may represent any high-dimensional signal, such as an image signal, an IMU signal, an audio signal, etc.

[0273]

[0319] In some embodiments, the first model 212 may be a neural network model. In such a case, the neural network model may be trained based on images of other drivers. For example, the neural network model may be trained using images of drivers to identify different poses, such as head down pose, head up pose, head upright pose, eyes closed pose, eyes open pose, mobile phone use pose, etc. In other embodiments, the first model 212 may be other types of models different from a neural network model.

[0274]

[0320] Also, in some embodiments, the second model 214 may be a neural network model. In such a case, the neural network model may be trained based on the feature information. For example, the feature information may be any information that indicates the state of the driver, such as a pose classification. In one implementation, the neural network model may be trained using the feature information output by the first model 212. In other embodiments, the second model 214 may be another type of model different from a neural network model.

[0275]

[0321] In some embodiments, the first model 212 utilized by the processing unit 210 to identify the driver's pose may be a convolutional neural network model. In other embodiments, the first model 212 may simply be any mathematical model. Also, in some embodiments, the second model 214 utilized by the processing unit 210 to determine whether the driver is drowsy may be a convolutional neural network model. In other embodiments, the second model 214 may simply be any mathematical model.

[0276]

[0322] In some embodiments, the first model 212 may be a first neural network model trained to classify poses based on context. For example, if the driver's head is down, the neural network model may determine that the driver is not looking ahead even if the driver's eyes cannot be detected (e.g., the eyes are obscured by a hat or cap). Also, in some embodiments, the second model 214 may be a second neural network model trained to determine whether the driver is drowsy based on context. For example, the neural network model may determine that the driver is drowsy if the blink frequency exceeds a certain threshold and / or if the head-down pose and the head-up pose are repeated in a periodic pattern. As another example, the neural network model may determine that the driver is drowsy if the time it takes to close the eyelids exceeds a certain threshold.

[0277]

[0323] In some embodiments, deep learning or artificial intelligence may be used to develop one or more models that identify the driver's pose and / or determine whether the driver is drowsy. Such models may distinguish between drowsy and awake drivers.

[0278]

[0324] It should be noted that the processing unit 210 is not limited to using a neural network model for pose classification and / or determining whether the driver is drowsy, and the processing unit 210 may utilize any processing technique, algorithm, or processing architecture for pose classification and / or determining whether the driver is drowsy. By way of non-limiting example, the processing unit 210 may use equations, regression, classification, neural networks (e.g., convolutional neural networks, deep neural networks), heuristics, selection (e.g., from libraries, graphs, or charts), instance-based methods (e.g., nearest neighbor methods), correlation methods, regularization methods (e.g., ridge regression), decision trees, Bayesian methods, kernel methods, probability, determinism, or a combination of two or more thereof, to process images from the camera 204 to process pose classification and / or time series information to determine whether the driver is drowsy. The pose classification can be a binary classification or score (e.g., head down or not), a score (e.g., continuous or discrete), a classification (e.g., high, medium, low), or any other suitable measure of pose classification. Similarly, the drowsiness classification can be a binary classification or score (e.g., drowsy or not), a score (e.g., continuous or discrete), a classification (e.g., high, medium, low), or any other suitable measure of drowsiness.

[0279]

[0325] In some embodiments, a determination of whether a driver is drowsy may be accomplished by analyzing a pattern of driver pose classification occurring over a period of at least less than 1 second, 1 second, 2 seconds, 5 seconds, 10 seconds, 12 seconds, 15 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes, 40 minutes, etc. The period may be any predefined duration of a moving window or box (e.g., the last less than 1 second, 1 second, 2 seconds, 5 seconds, 10 seconds, 12 seconds, 15 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes, 40 minutes, etc. of data to identify data generated during the last duration).

[0280]

[0326] In some embodiments, the first model 212 and the second model 214 may be configured to work in conjunction to detect "microsleep" events, such as slow eyelid closures occurring over periods of less than 1 second, 1-1.5 seconds, or more than 2 seconds. In other embodiments, the first model 212 and the second model 214 may be configured to work in conjunction to detect early signs of drowsiness based on images taken over longer periods, such as 10 seconds, 12 seconds, 15 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 25 minutes, 30 minutes, 40 minutes, etc.

[0281]

[0327] As shown in the above example, using multiple sequential models to detect drowsiness is advantageous. In particular, the technique of combining (1) a first model that processes camera images one by one (as each camera image is generated) to identify the driver's pose, and (2) a second model that processes feature information obtained from the processing result of the camera images by the first model, eliminates the need for the processing unit 210 to collect and process a series of camera images (footage) in a batch. This can significantly save computational resources and memory capacity. Furthermore, as described in the above example, the second model does not process images from the camera. The second model receives feature information as an output from the first model and processes the feature information to determine whether the driver is feeling drowsy. This is advantageous because the processing of feature information is easier and faster than batch processing of camera images. Also, context-based classification is advantageous because it allows the processing unit 210 to accurately identify various poses of the driver. In some cases, context-based identification allows processing unit 210 to accurately identify the driver's pose even when device 200 is mounted at a significantly off-angle relative to the vehicle (which may result in the driver appearing at an odd angle and / or position in the camera image). Aftermarket products may be mounted in a variety of positions. The features described herein are also advantageous in that they can determine whether the driver is feeling drowsy even when device 200 is mounted at different angles.

[0282]

[0328] It should be noted that processing unit 210 is not limited to detecting poses indicative of a driver being drowsy (e.g., poses belonging to the "drowsy" class). In other embodiments, processing unit 210 may be configured to detect poses indicative of a driver being awake (e.g., poses belonging to the "awake" class). In further embodiments, processing unit 210 may be configured to detect both (1) poses indicative of a driver being drowsy and (2) poses indicative of a driver being awake.

[0283]

[0329] In some embodiments, the processing unit 210 may obtain (e.g., by receiving or determining) additional parameters to determine whether the driver is drowsy. As non-limiting examples, the processing unit 210 may be configured to obtain information regarding vehicle acceleration, vehicle deceleration, vehicle position relative to the driving lane, driver engagement in driving, and the like. In some cases, one or more of the above parameters may be obtained by the second model 214, which may determine whether the driver is drowsy based on such parameters along with the output from the first model 212. It should be noted that the acceleration, deceleration, and driver engagement information are indicative of whether the driver is actively driving. For example, if the driver is changing speed or turning the steering wheel, the driver is less likely to be drowsy. In some embodiments, sensors built into the vehicle may provide the acceleration and deceleration information. In such cases, the processing unit 210 may be hardwired to the vehicle system to receive such information. Alternatively, the processing unit 210 may be configured to receive such information wirelessly. In further embodiments, the device 200 including the processing unit 210 may optionally further include an accelerometer for detecting acceleration and deceleration. In such a case, the second model 214 may be configured to obtain acceleration information and / or deceleration information from the accelerometer. Also, the information regarding the driver's engagement may be any information indicating whether the driver is operating the vehicle. As a non-limiting example, such information may include one or more of turning or missing a steering wheel, engaging or missing a turn signal lever, changing gears or missing a gear, engaging or missing a brake, depressing or missing an accelerator pedal, and the like. In some embodiments, the information regarding the driver's engagement may be information regarding driver's engagement that occurred within a certain past period (e.g., the last 10 seconds or more, the last 20 seconds or more, the last 30 seconds or more, the last minute or more, etc.).

[0284]

[0330] Further, in some embodiments, the position of the vehicle relative to the lane of travel may be determined by the processing unit 210 processing images from the externally facing camera 202. In particular, the processing unit 210 may be configured to determine whether the vehicle is traveling within a certain threshold of the lane centerline. If the vehicle is traveling within a certain threshold of the lane centerline, the driver is actively participating in the driving. On the other hand, if the vehicle is away from the lane centerline beyond the threshold, the driver may not be actively participating in the driving. In some embodiments, the second model 214 may be configured to receive images from the first camera 202 and determine whether the vehicle is traveling within a certain threshold of the lane centerline. In other embodiments, a separate module may be configured to provide this functionality. In such a case, the output of the module is input to the second model 214, allowing the model 214 to determine whether the driver is experiencing drowsiness based on the output of the module.

[0285]

[0331] Also, in one or more embodiments described herein, the processing unit 210 may be further configured to determine the risk of collision based on whether the driver is drowsy. In some embodiments, the processing unit 210 may be configured to determine the risk of collision based only on whether the driver is drowsy. For example, the processing unit 210 may determine the risk of collision to be “high” if the driver is drowsy and “low” if the driver is not drowsy (e.g., awake). In other embodiments, the processing unit 210 may be configured to determine the risk of collision based on additional information. For example, the processing unit 210 may be configured to track how long the driver has felt drowsy and determine the level of risk of collision based on the duration of the drowsiness.

[0286]

[0332] As another example, the processing unit 210 may process images from the first camera 202 to determine an output and determine a risk of collision based on such output in combination with the pose classification and / or drowsiness determination. As non-limiting examples, the output may be a classification of the driving state, a classification of the external environment, identified features of the environment, a context of the operation of the vehicle, etc. For example, in some embodiments, camera images capturing the vehicle's external environment may be processed by the processing unit 210 to determine whether the vehicle is turning left, going straight, turning right, whether there is an obstacle (vehicle, pedestrian, etc.) in front of the subject vehicle, etc. If the vehicle is turning right or left when drowsiness is detected and / or an obstacle is detected in the vehicle's driving path, the processing unit 210 may determine that there is a high risk of collision.

[0287]

[0333] It should be noted that the second model 214 of the processing unit 210 is not limited to receiving only the output from the first model 212. The second model 214 may be configured to receive other information (as input) in addition to the output from the first model 212. For example, in other embodiments, the second model 214 may be configured to receive sensor signals from one or more sensors mounted on the vehicle, which may be configured to sense information regarding the vehicle's movement and / or operation characteristics. As non-limiting examples, the sensor signals acquired by the second model 214 may be acceleration signals, gyroscope signals, velocity signals, position signals (e.g., GPS signals), etc., or any combination thereof. In further embodiments, the processing unit 210 may include a processing module that processes the sensor signals. In such a case, the second model 214 may be configured to receive the processed sensor signals from the processing module. In some embodiments, the second model 214 may be configured to process the sensor signals (provided from the sensors) or the processed sensor signals (provided from the processing module) to determine the risk of a collision. The determination of the risk of a collision may be based on the drowsiness detection and the sensor signal. In other embodiments, the determination of the risk of a collision may be based on the drowsiness detection, the sensor signal, and an image of the surrounding environment outside the vehicle captured by the camera 202.

[0288]

[0334] Also, in some embodiments, processing unit 210 may include a facial landmark detection module configured to detect one or more facial landmarks of the driver captured in the camera 204 images. In such a case, second model 214 may be configured to receive output from the facial landmark detection module. In some cases, the output from the facial landmark detection module may be utilized by second model 214 to determine drowsiness and / or alertness. Alternatively or additionally, the output from the facial landmark detection module may be used to train second model 214.

[0289]

[0335] Also, in some embodiments, the processing unit 210 may include an eye landmark detection module configured to detect one or more eye landmarks of the driver captured in the image of the camera 204. In such a case, the second model 214 may be configured to receive an output from the eye landmark detection module. In some cases, the output from the eye landmark detection module may be utilized by the second model 214 to determine drowsiness and / or alertness. Alternatively or additionally, the output from the eye landmark detection module may be used to train the second model 214. The eye landmark may be a pupil, an eyeball, an eyelid, etc., or any feature associated with the driver's eye.

[0290]

[0336] In some embodiments, if the second model 214 is configured to receive one or more other pieces of information in addition to the output from the first model 212, the second model 214 may be configured to receive the one or more pieces of information in parallel as well as the output from the first model 212. This allows the second model 214 to receive different pieces of information independently and / or simultaneously.

[0291]

[0337] Figure 12 illustrates a method 650 performed by the apparatus 200 of Figure 2A according to some embodiments. The method 650 includes generating an image of a driver of the vehicle by a camera (item 652), processing the image by a first model in a processing unit to obtain feature information (item 654), providing the feature information by the first model (item 656), obtaining the feature information from the first model by a second model (item 658), and processing the feature information by the second model to obtain an output indicative of whether the driver is drowsy (item 660).

[0292]

[0338] It should be noted that the poses that may be identified by the driver monitoring module 211 are not limited to the described examples, and the driver monitoring module 211 may identify other poses or behaviors of the driver. As a non-limiting example, the driver monitoring module 211 may be configured to detect the driver talking, singing, eating, daydreaming, etc., or any combination thereof. Even if the driver is looking at the road, it is advantageous to detect cognitive distraction (e.g., talking) because if the driver is cognitively distracted (compared to paying attention to driving), there may be a higher risk of intersection violations and a higher risk of collisions.

[0293]

[0339] Collision prediction

[0340] FIG. 13 illustrates an example of a processing architecture 670 according to some embodiments. At least a portion of the processing architecture 670 may be implemented using the device of FIG. 2A in some embodiments. The processing architecture 670 includes a calibration module 671 configured to determine an area of ​​interest for detecting objects in an image that are at risk of collision with a target vehicle, a vehicle detector 672 configured to detect a vehicle, and a vehicle state module 674 configured to obtain information regarding one or more states of the target vehicle. The processing architecture 670 also includes a collision predictor 675 having a tracker 676 and a time to collision (TTC) calculation unit 680. The processing architecture 670 further includes a driver monitoring module 678 configured to determine whether a driver of the target vehicle is distracted. The processing architecture 670 also includes an event trigger module 682 configured to generate a control signal 684 in response to detection of a particular event based on the output provided by the collision predictor 675, and a contextual event module 686 configured to provide a contextual alert 688 based on the output provided by the driver monitoring module 678.

[0294]

[0341] In some embodiments, the vehicle detector 672 may be implemented by the object detector 216 and / or may be considered as an example of the object detector 216. The collision predictor 675 may be an example of the collision predictor 218 of the processing unit 210 in some embodiments. The driver monitoring module 678 may be implemented by the driver monitoring module 211 of the processing unit 210 in some embodiments. The event trigger module 682 may be implemented using the signal generation controller 224 of the processing unit 210 and / or may be considered as an example of the signal generation controller 224.

[0295]

[0342] In operation, the calibration module 671 is configured to determine a region of interest for the first camera 202 to detect vehicles at risk of colliding with the target vehicle. The calibration module 671 is further described with reference to Figures 15A-15C. The vehicle detector 672 is configured to identify vehicles in a camera image provided by the first camera 202. In some embodiments, the vehicle detector 672 is configured to detect vehicles in the image based on a model, such as a neural network model, trained to identify vehicles.

[0296]

[0343] The driver monitoring module 678 is configured to determine whether a driver of the target vehicle is distracted. In some embodiments, the driver monitoring module 678 can identify one or more poses of the driver based on an image provided by the second camera 204. The driver monitoring module 678 can determine whether the driver is distracted based on the driver's pose. In some cases, the driver monitoring module 678 may determine the one or more poses of the driver based on a model, such as a neural network model, trained to identify driver poses.

[0297]

[0344] The collision prediction unit 675 is configured to select one or more of the vehicles detected by the vehicle detection unit 672 as possible candidates for collision prediction. In some embodiments, the collision prediction unit 675 is configured to select a vehicle for collision prediction if an image of the vehicle intersects with a region of interest (determined by the calibration module 671) in an image frame. The collision prediction unit 675 is also configured to track a state of the selected vehicle (by the tracker 676). As a non-limiting example, the state of the selected vehicle tracked may be a position of the vehicle, a speed of the vehicle, an acceleration or deceleration of the vehicle, a direction of movement of the vehicle, etc., or any combination thereof. In some embodiments, the tracker 676 may be configured to determine whether the detected vehicle is on a collision course with the target vehicle based on the travel path of the target vehicle and / or the travel path of the detected vehicle. Also, in some embodiments, the tracker 676 may be configured to determine that a vehicle is a leading vehicle if an image of the detected vehicle appearing in an image frame from the first camera 202 intersects with a region of interest in the image frame.

[0298]

[0345] The TTC unit 680 of the collision prediction section 675 is configured to calculate, for a predicted collision, an estimated time it takes for the selected vehicle to collide with the target vehicle based on the tracking state of the selected vehicle and the state of the target vehicle (provided by the vehicle state module 674). For example, if the tracking state of the selected vehicle indicates that the vehicle is in the path of the target vehicle and traveling at a slower speed than the target vehicle, the TTC unit 680 identifies an estimated time until the selected vehicle collides with the target vehicle. As another example, if the tracking state of the selected vehicle indicates that the vehicle is a leading vehicle in front of the target vehicle, the TTC unit 680 determines an estimated time until the selected vehicle collides with the target vehicle. In some embodiments, the TTC unit 680 may determine an estimated time until the predicted collision based on the relative speed between the two vehicles and / or the distance between the two vehicles. The TTC unit 680 is configured to output the estimated time (TTC parameter).

[0299]

[0346] It should be noted that the collision prediction unit 675 is not limited to predicting a collision between a leading vehicle and a target vehicle, but may be configured to predict other types of collisions. For example, in some embodiments, the collision prediction unit 675 may be configured to predict a collision between the target vehicle and another vehicle traveling on two different respective roads (e.g., intersecting roads) and heading toward an intersection. As another example, in some embodiments, the collision prediction unit 675 may be configured to predict a collision between the target vehicle and another vehicle traveling in an adjacent lane merging or entering into the lane of the target vehicle.

[0300]

[0347] The event trigger module 682 is configured to provide a control signal based on the output provided by the crash predictor 675 and the output provided by the driver monitoring module 678. In some embodiments, the event trigger module 682 is configured to continuously or periodically monitor the driver's state based on the output provided by the driver monitoring module 678. The event trigger module 682 also monitors the TTC parameter in parallel. If the TTC parameter indicates that the estimated time it will take for a predicted crash to occur is less than or equal to a certain threshold (e.g., 8 seconds, 7 seconds, 6, seconds, 5 seconds, 4 seconds, 3 seconds, etc.) and the output by the driver monitoring module 678 indicates that the driver is distracted or not paying attention to the driving task, the event trigger module 682 generates the control signal 682.

[0301]

[0348] In some embodiments, the control signal 684 from the event trigger module 682 may be sent to a warning generator configured to provide a warning to the driver. Alternatively or additionally, the control signal 684 from the event trigger module 682 may be sent to a vehicle controller configured to control the vehicle (e.g., automatically release the accelerator pedal, apply the brakes, etc.).

[0302]

[0349] In some embodiments, the threshold is variable based on the output from the driver monitoring module 678. For example, if the output from the driver monitoring module 678 indicates that the driver is not distracted and / or paying attention to the driving task, the event trigger module 682 may generate a control signal 684 to operate an alert generator and / or operate a vehicle control device in response to the TTC meeting or falling below a first threshold (e.g., 3 seconds). On the other hand, if the output from the driver monitoring module 678 indicates that the driver is distracted or not paying attention to the driving task, the event trigger module 682 may generate a control signal 684 to operate an alert generator and / or operate a vehicle control device in response to the TTC meeting or falling below a second threshold (e.g., 5 seconds) that is higher than the first threshold.

[0303]

[0350] Also, in some embodiments, the event trigger module 682 may be configured to apply different values ​​of the threshold for generating the control signal 684 based on the type of driver state indicated by the output of the driver monitoring module 678. For example, if the output of the driver monitoring module 678 indicates that the driver is looking at a cell phone, the event trigger module 682 may generate a control signal 684 that activates an alert generator and / or activates a vehicle control device in response to the TTC meeting or falling below a 5 second threshold. On the other hand, if the output of the driver monitoring module 678 indicates that the driver is drowsy, the event trigger module 682 may generate a control signal 684 that activates an alert generator and / or activates a vehicle control device in response to the TTC meeting or falling below an 8 second threshold (e.g., longer than the threshold when the driver is using a cell phone). In some cases, a driver's state (e.g., the driver is drowsy or asleep) may require a longer threshold time (relative to the TTC value) to alert the driver or take control of the vehicle, as the driver may need more time to react to an impending collision. Thus, the event trigger module 682 may provide a warning to the driver and / or take earlier action on vehicle control in response to a predicted collision in such circumstances.

[0304]

[0351] In some embodiments, the TTC unit 680 is configured to determine a TTC value of a predicted crash and then track the passage of time with respect to the TTC value. For example, if the TTC unit 680 determines that the TTC of a predicted crash is 10 seconds, the TTC unit 680 may perform a time countdown of 10 seconds. While the TTC unit 680 is counting down, the TTC unit 680 periodically outputs the TTC and informs the event trigger module 682 of the current TTC value. Thus, the TTC output by the TTC unit 680 at different times for a predicted crash is different values ​​based on the countdown. In other embodiments, the TTC unit 680 is configured to repeatedly determine the TTC value of a predicted crash based on images from the first camera 202. In such a case, the TTC output by the TTC unit 680 at different times for a predicted crash is different values ​​calculated by the TTC unit 680 based on images from the first camera 202.

[0305]

[0352] Also, in some embodiments, the collision prediction unit 675 may continue to monitor the state of the other vehicle and / or the target vehicle after a collision is predicted. For example, if the other vehicle moves out of the path of the target vehicle and / or the distance between the two vehicles increases (e.g., because the other vehicle accelerates and / or the target vehicle decelerates), the collision prediction unit 675 may provide an output indicating that the risk of collision has passed. In some embodiments, the TTC unit 680 may output a signal to the event trigger module 682 indicating that the control signal 684 does not need to be generated. In other embodiments, the TTC unit 680 may output a predefined arbitrary TTC value that is very high (e.g., 2000 seconds), or a TTC with a negative value, such that the event trigger module 682 does not generate the control signal 684 when processing the TTC value.

[0306]

[0353] An embodiment of the collision predictor 675 (an example of the collision predictor 218) and an embodiment of the event trigger module 682 (an example of the signal generating controller 224) are described further below.

[0307]

[0354] The contextual event module 686 is configured to provide a contextual alert 688 based on the output provided by the driver monitoring module 678. For example, if the output of the driver monitoring module 678 indicates that the driver is distracted for a period of time exceeding a duration threshold or with a frequency exceeding a frequency threshold, the contextual event module 686 may generate an alert to warn the driver. Alternatively or additionally, the contextual event module 686 may generate a message to notify a fleet manager, insurance company, etc. In other embodiments, the contextual event module 686 is optional and the processing architecture 670 may not include the contextual event module 686.

[0308]

[0355] In other embodiments, item 672 may be a human detector, and processing architecture 670 may be configured to predict a collision with a human and generate a control signal based on the predicted collision and the driver state output by driver monitoring module 678, as also described herein.

[0309]

[0356] In a further embodiment, item 672 may be an object detector configured to detect objects associated with the intersection, and processing architecture 670 may be configured to predict an intersection violation and generate a control signal based on the predicted intersection violation and the driver state output by driver monitoring module 678, as also described herein.

[0310]

[0357] In yet another embodiment, item 672 may be an object detector configured to detect multiple classes of objects, such as vehicles, humans, intersection-related objects, etc. In such a case, processing architecture 670 may be configured to perform vehicle collision predictions, human collision predictions, intersection violation predictions, etc., and generate control signals based on any of these predicted events and based on the driver state output by driver monitoring module 678, as also described herein.

[0311]

[0358] FIG. 14 is a diagram illustrating an example of object detection according to some embodiments. As shown, the detected object is a vehicle captured in the image provided by the first camera 202. The detection of the object may be performed by the object detector 216. In the illustrated example, the identified vehicles are given a respective identifier (e.g., in the form of a bounding box indicating the spatial extent of each identified vehicle). It should be noted that the object detector 216 is not limited to providing an identifier that is a rectangular bounding box for the identified vehicles, and the object detector 216 may be configured to provide other forms of identifiers for each identified vehicle. In some embodiments, the object detector 216 may distinguish a vehicle that is a leading vehicle from other vehicles that are not leading vehicles.

[0312]

[0359] Also, in some embodiments, processing unit 210 may track identified leading vehicles and determine a region of interest based on the spatial distribution of such identified leading vehicles. For example, as shown in FIG. 15A, processing unit 210 may use identifiers 750 (in the form of bounding boxes in the example) of leading vehicles identified over a period of time (e.g., the last 5 seconds, the last 10 seconds, the last minute, the last 2 minutes, etc.) and form a region of interest based on the spatial distribution of identifiers 750. In the illustrated embodiment, the region of interest has a fixed dimension and location (which is approximately horizontally centered at the bottom of the image frame).

[0313]

[0360] In some embodiments, instead of using a bounding box, the processing unit 2 may form a region of interest using horizontal lines 752 (FIG. 15B). In the illustrated example, each horizontal line 752 represents a preceding vehicle identified over a period of time. The horizontal lines 752 may be considered as an example of an identifier of the identified preceding vehicle. In some cases, the horizontal lines 752 may be obtained by extracting only the bottom side of a bounding box (e.g., 750 shown in FIG. 15A). As shown in FIG. 15B, the distribution of the horizontal lines 752 forms a region of interest (represented by the area filled by the horizontal lines 752) having a substantially triangular or trapezoidal shape. The processing unit 210 may use such a region of interest as a detection region to detect a future preceding vehicle. For example, as shown in FIG. 15C, the processing unit 210 may use the identifier of the identified preceding vehicle (e.g., line 752) to form a region of interest 754, which in this embodiment has a triangular shape. The region of interest 754 may be used by the object detection unit 216 to identify the preceding vehicle. In the illustrated example, the object detection unit 216 detects a vehicle 756. Because at least a portion of the detected vehicle 756 is located in the region of interest 754, the object detection unit 216 may determine that the identified vehicle is a leading vehicle.

[0314]

[0361] In some embodiments, the region of interest 754 may be determined by a calibration module in the processing unit 210 during a calibration process. Also, in some embodiments, the region of interest 754 may be periodically updated during operation of the device 200. It should be noted that the region of interest 754 for detecting a leading vehicle is not limited to the described example and may have other configurations (e.g., size, shape, location, etc.) in other embodiments. Also, in other embodiments, the region of interest 754 may be determined using other techniques. For example, in other embodiments, the region of interest 754 for detecting a leading vehicle may be pre-determined (e.g., programmed during manufacture) without using a distribution of previously detected leading vehicles.

[0315]

[0362] In the above example, the region of interest 754 has a triangular shape that may be determined during a calibration process. In other embodiments, the region of interest 754 may have other shapes and may be determined based on detection of a lane centerline. For example, in other embodiments, the processing unit 210 may include a centerline detection module configured to identify a centerline of a lane or road on which the target vehicle is traveling. In some embodiments, the centerline detection module may be configured to identify the centerline by processing images from the first camera 202. In one implementation, the centerline detection module analyzes images from the first camera 202 to identify the lane or road centerline based on a model. The model may be a neural network model trained to identify the centerline based on images of various road conditions. Alternatively, it may be other types of models, such as mathematical models, equations, etc. FIG. 15D illustrates an example of the centerline detection module identifying the centerline and an example region of interest based on the detected centerline. As illustrated, the centerline detection module determines a set of points 757a-757e that represent the centerline of the lane or road on which the target vehicle is traveling. Although five points 757a-757e are shown, in other examples, the centerline detection module may identify more than five points 757 representing the centerline or may identify fewer than five points 757. Also as shown, the processing unit 210 may determine a left set of points 758a-758e and a right set of points 759a-759e based on the points 757a-757e. Also as shown, the processing unit 210 may determine a first set of lines connecting the left points 758a-758e and a second set of lines connecting the right points 759a-759e. As shown, the first set of lines forms a left boundary of the region of interest 754 and the second set of lines forms a right boundary of the region of interest 754.

[0316]

[0363] In the illustrated example, processing unit 210 is configured to determine left point 758a as having the same y coordinate as centerline point 757a and an x ​​coordinate that is a distance d1 to the left from the x coordinate of centerline point 757a. Processing unit 210 is also configured to determine right point 759a as having the same y coordinate as centerline point 757a and an x ​​coordinate that is a distance d1 to the right from the x coordinate of centerline point 757a. Thus, left point 758a, centerline point 757a, and right point 759a are horizontally aligned. Similarly, processing unit 210 is configured to determine left points 758b-758e as having respective y coordinates as respective centerline points 757b-757e and respective x coordinates that are a respective distance d2-d5 to the left from the respective x coordinates of centerline points 757b-757e. The processing unit 210 is also configured to determine right points 759b-759e as having respective y coordinates that are the same as respective centreline points 757b-757e and that are respective x coordinates that are respective distances d2-d5 to the right of the respective x coordinates of centreline points 757b-757e.

[0317]

[0364] In the illustrated example, d1>d2>d3>d4>d5, and the region of interest 754 has a tapered shape corresponding to the shape of the road appearing in the camera image. Since the first camera 202 repeatedly provides camera images of the road on which the vehicle is traveling, the processing unit 210 repeatedly determines a center line and left and right boundaries of the region of interest 754 based on the center line. In this manner, the region of interest 754 changes its tapered shape in response to changes in the shape of the road appearing in the camera image (for example, the curvature of the tapered region of interest 754 changes). That is, since the center line is identified based on the shape of the road and the shape of the region of interest 754 is determined based on the identified center line, the shape of the region of interest 754 is variable in response to the shape of the road on which the vehicle is traveling.

[0318]

[0365] FIG. 15E illustrates the advantage of using the region of interest 754 of FIG. 15D to detect objects at risk of collision. In particular, the right side of the figure illustrates a region of interest 754 determined based on the centerline of the road or lane on which the target vehicle is traveling, as described with reference to FIG. 15D. The left side of the figure illustrates another region of interest 754 determined based on camera calibration, as described with reference to FIGS. 15A-15C, and having a shape that is independent of the centerline of the road / lane (e.g., centerline curvature). Because the region of interest 754 on the left side is independent of the curvature of the road / lane, the shape of the region of interest 754 does not necessarily match the shape of the road / lane. Thus, in the illustrated example, the processing unit 210 may erroneously detect that a pedestrian is at risk of collision because it intersects with the region of interest 754. In another similar situation, the region of interest 754 on the left side may erroneously detect a parked vehicle outside the target lane as an object that poses a collision risk. In some embodiments, the processing unit 210 may be configured to perform additional processing to address the problem of false detection (e.g., falsely detecting an object as at risk of collision). On the other hand, the right region of interest 754 is advantageous because it does not suffer from the false positive problem discussed above.

[0319]

[0366] However, in other embodiments, other techniques may be employed to determine region of interest 754 having a shape that varies corresponding to the shape of the road. For example, in other embodiments, processing unit 210 may include a road or lane boundary module configured to identify left and right boundaries of a lane or road on which the target vehicle is traveling. Processing unit 210 may also determine one or more lines that fit a left boundary and one or more lines that fit a right boundary, and determine region of interest 754 based on the determined lines.

[0320]

[0367] In some embodiments, processing unit 210 may be configured to determine both (1) a first region of interest (such as triangular region of interest 754 described with reference to FIGS. 15A-15C) and (2) a second region of interest, such as region of interest 754 described with reference to FIG. 15D. The first region of interest may be used by processing unit 210 to crop the camera image. For example, certain portions of the camera image that are away from the first region of interest or that are a distance away from the first region of interest may be cropped to reduce the amount of image data that needs to be processed. The second region of interest may be used by processing unit 210 to determine whether a detected object poses a collision risk. For example, if a detected object, or a bounding box of a detected object, overlaps with the second region of interest, processing unit 210 may determine that there is a risk of collision with the detected object. In some embodiments, the first region of interest may also be used by processing unit 210 to detect leading vehicles in the camera image. The detected vehicle width and its corresponding position relative to the image coordinate system may be used by processing unit 210 to determine a y-distance mapping, which will be described in more detail below with reference to FIG. 20.

[0321]

[0368] In some embodiments, the collision predictor 218 may be configured to determine whether the region of interest 754 (e.g., a polygon created based on the centerline) intersects with a bounding box of a detected object, such as a leading vehicle, a pedestrian, etc. If so, the collision predictor 218 determines that there is a risk of collision, and the object corresponding to the bounding box is considered for TTC calculation.

[0322]

[0369] In some embodiments, the collision prediction unit 218 may be configured to predict a collision with a leading vehicle in at least three different scenarios. FIG. 16 illustrates three exemplary scenarios involving a collision with a leading vehicle. In the top diagram (first scenario), the target vehicle (left vehicle) is traveling at a non-zero speed Vsv, and the leading vehicle (right vehicle) is completely stopped, resulting in a speed Vpov=0. In the middle diagram (second scenario), the target vehicle (left vehicle) is traveling at a non-zero speed Vsv, and the leading vehicle (right vehicle) is traveling at a non-zero speed Vpov that is smaller than the speed Vsv. In the bottom diagram (third scenario), the target vehicle (left vehicle) is initially traveling at a non-zero speed Vsv, and the leading vehicle (right vehicle) is also initially traveling at a non-zero speed Vpov=Vsv. The leading vehicle then brakes, decreasing the speed Vpov, and the target vehicle's speed Vsv becomes greater than the leading vehicle's speed Vpov. In some embodiments, the collision prediction unit 218 is configured to predict a collision between the target vehicle and the leading vehicle that may occur in any of the three scenarios shown in FIG. 16. In one implementation, the collision prediction unit 218 may analyze an image sequence from the first camera 202 to determine the relative speed between the target vehicle and the leading vehicle. In another implementation, the collision prediction unit 218 may obtain sensor information indicative of the relative speed between the target vehicle and the leading vehicle. For example, the collision prediction unit 218 may obtain a sequence of sensor information indicative of the distance between the target vehicle and the leading vehicle over a period of time. The collision prediction unit 218 may determine the relative speed between the target vehicle and the leading vehicle by analyzing the change in the distance during this period of time. Also, in some embodiments, the collision prediction unit 218 may obtain the speed of the target vehicle from a speed sensor of the target vehicle, a GPS system, or a speed sensor separate from the speed sensor of the target vehicle, etc.

[0323]

[0370] In some embodiments, the collision prediction unit 218 may be configured to predict a collision between the target vehicle and the leading vehicle based on the relative speed between the target vehicle and the leading vehicle, the speed of the target vehicle, the speed of the leading vehicle, or any combination thereof. For example, the collision prediction unit 218 may determine that there is a risk of collision if (1) the object detection unit 216 detects a leading vehicle, (2) the relative speed between the leading vehicle and the target vehicle is not zero, and (3) the distance between the leading vehicle and the target vehicle is decreasing. In some cases, the criteria (2) and (3) may be combined to indicate whether the target vehicle is traveling faster than the leading vehicle. In such a case, the collision prediction unit 218 may determine that there is a risk of collision if (1) the object detection unit 216 detects a leading vehicle, and (2) the target vehicle is traveling faster than the leading vehicle (the target vehicle is moving toward the leading vehicle).

[0324]

[0371] In some embodiments, the collision prediction unit 218 may obtain other information that it uses to determine whether there is a risk of a collision. By way of non-limiting example, the collision prediction unit 218 may obtain information indicating that the leading vehicle is braking (e.g., camera imagery, detected light, etc.), motion parameters of the target vehicle (e.g., information indicating accelerating, decelerating, turning, etc.), motion parameters of the leading vehicle (e.g., information indicating accelerating, decelerating, turning, etc.), or any combination thereof.

[0325]

[0372] In some embodiments, the collision predictor 218 is configured to predict a collision at least 3 seconds or more before the expected occurrence time of the predicted collision. For example, the collision predictor 218 may be configured to predict a collision at least 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, etc. seconds before the expected occurrence time of the predicted collision. Also, in some embodiments, the collision predictor 218 is configured to predict a collision with sufficient lead time for the driver's brain to process the inputs and for the driver to take action to mitigate the risk of a collision. In some embodiments, the sufficient lead time may depend on the driver's state as determined by the driver monitoring module 211.

[0326]

[0373] It should be noted that the object detector 216 may be configured to detect humans in some embodiments. In such a case, the collision predictor 218 may be configured to predict a collision with the human. FIG. 17 illustrates another example of object detection where the detected object is a human. As illustrated, the detected object is a human in the image provided by the first camera 202. The detection of the object may be performed by the object detector 216. In the illustrated example, the identified humans are given a respective identifier (e.g., in the form of a bounding box 760 that indicates the spatial extent of each identified vehicle). It should be noted that the object detector 216 is not limited to providing an identifier that is a rectangular bounding box 760 for the identified humans, and the object detector 216 may be configured to provide other forms of identifiers for each identified human. In some embodiments, the object detector 216 may distinguish humans that are in front of the target vehicle (e.g., in the path of the vehicle) from other humans that are not in the path of the target vehicle. In some embodiments, the same region of interest 754 described above for detecting leading vehicles may be used by the object detector 216 to detect humans in the path of the target vehicle.

[0327]

[0374] In some embodiments, the collision prediction unit 218 may be configured to determine the direction of movement of the detected human by analyzing the sequence of images of the human provided by the first camera 202. The collision prediction unit 218 may also be configured to determine the speed of movement of the detected human (e.g., the walking speed or running speed of the human) by analyzing the sequence of images of the human. The collision prediction unit 218 may also be configured to determine whether there is a risk of collision with a human based on the travel path of the target vehicle and based on the direction of movement of the detected human. Such a function may be desirable to prevent collisions with humans on the sidewalk who are not in the path of the vehicle but are moving towards the path of the target vehicle.

[0328]

[0375] In some embodiments, the collision prediction unit 218 is configured to determine an area next to the detected human, indicative of a possible location of the human at some future time (e.g., the next 0.5 seconds, the next 1 second, the next 2 seconds, the next 3 seconds, etc.), based on the moving speed and moving direction of the detected human. The collision prediction unit 218 may then determine, based on the speed of the target vehicle, whether the target vehicle will pass through the determined area (e.g., within a box) indicative of the predicted location of the human. In one embodiment, the collision prediction unit 218 may determine whether the determined area intersects with the region of interest 754. If so, the collision prediction unit 218 may determine that there is a risk of collision with the human and generate an output indicative of the predicted collision.

[0329]

[0376] FIG. 18 illustrates an example of a predicted human location based on a human walking speed and direction. Because human movement is inherently somewhat unpredictable, in some embodiments, even if the detected human is standing (e.g., a pedestrian standing next to a roadway), the collision prediction unit 218 may determine a region about the human that indicates the human's possible location (e.g., for when the human starts walking or running). For example, the collision prediction unit 218 may determine a bounding box 760 (e.g., a rectangular box) surrounding the detected human and then increase the dimensions of this bounding box 760 to account for the uncertainty of the human's predicted future location, where the enlarged box defines an area that indicates the human's predicted location. The collision prediction unit 218 may then determine whether the target vehicle passes through the determined region that indicates the human's predicted location. In one embodiment, the collision prediction unit 218 may determine whether the determined region of the enlarged box intersects with the region of interest 754. If so, the collision prediction unit 218 may determine that there is a risk of a collision with the human and generate an output that indicates the predicted collision.

[0330]

[0377] In some embodiments, the collision prediction unit 218 may be configured to predict a collision with a human in at least three different scenarios. In a first scenario, the detected human (or a bounding box 760 surrounding the detected human) intersects with the region of interest 754, indicating that the human is already in the path of the target vehicle. In a second scenario, the detected human is not in the path of the target vehicle, but is standing on the side of the roadway. In such a case, the collision prediction unit 218 may use the area of ​​the enlarged bounding box of the detected human to determine whether there is a risk of collision, as described above. If the enlarged bounding box intersects with the region of interest (for detecting collisions) 754, the collision prediction unit 218 may determine that there is a risk of collision with the standing human. In a third scenario, the detected human is moving, and the image of the human (or its bounding box 760) does not intersect with the region of interest (for detecting collisions) 754. In such a case, the collision prediction unit 218 may use the area of ​​the predicted position of the human to determine whether there is a risk of collision, as described above. If the region of the predicted position intersects with the region of interest 754, the collision prediction unit 218 may determine that there is a risk of collision with a human.

[0331]

[0378] In some embodiments, the expanded bounding box has dimensions of the detected object plus an additional length, which is preset to account for the uncertainty of the object's movement. In other embodiments, the expanded bounding box may be determined based on a prediction of the object's location. As shown in FIG. 18, the detected object may have an initial bounding box 760. Based on the object's location in the image from the first camera 202, the processing unit 210 may predict the location of the moving object. As shown, the processing unit 210 may determine a box 762a representing a possible location of the object 0.3 seconds into the future. The processing unit 210 may also determine a box 762b representing a possible location of the object 0.7 seconds into the future and a box 762c representing a possible location of the object 1 second into the future. In some embodiments, the collision prediction unit 218 may continue to predict the future location of the detected object (e.g., a human) at a particular future time and determine whether the path of the target vehicle intersects with any of these locations. If so, the collision prediction unit 218 may determine that there is a risk of collision with the object.

[0332]

[0379] In some embodiments, the region of interest 754 may be enlarged in response to the driver monitoring module 211 detecting that the driver is distracted. For example, in response to the driver monitoring module 211 detecting that the driver is distracted, the region of interest 754 may be widened. This has the advantage that objects outside the road or lane can be considered as a collision risk. For example, if the driver is distracted, the processing unit 210 then enlarges the region of interest 754. This has the effect that the threshold for detecting overlap of detected objects with the region of interest 754 is relaxed. If a bicycle is traveling on the edge of the lane, it may overlap the enlarged region of interest 754, and therefore the processing unit 210 may detect the bicycle as being at risk of collision. On the other hand, if the driver is attentive (e.g., not distracted), the region of interest 754 will be smaller and the bicycle will not intersect with the region of interest 754. Thus, in this scenario, the processing unit 210 does not consider the bicycle to be at risk of collision, which makes sense since an attentive driver is more likely to avoid a collision with a bicycle.

[0333]

[0380] In some embodiments, to reduce computational load, the collision prediction unit 218 may not determine the collision risk for all detected humans in the image. For example, in some embodiments, the collision prediction unit 218 may exclude humans inside a vehicle, humans standing at a bus stop, humans sitting outside, etc. In other embodiments, the collision prediction unit 218 may consider all detected humans for collision prediction.

[0334]

[0381] In some embodiments, object detector 216 may use one or more models to detect various objects, such as automobiles (as shown), motorcycles, pedestrians, animals, lane dividers, road signs, traffic signs, traffic lights, etc. In some embodiments, the models utilized by object detector 216 may be neural network models trained to identify various objects. In other embodiments, the models may be any other type of model, such as a mathematical model configured to identify objects. The models utilized by object detector 216 may be stored in non-transitory medium 230 and / or incorporated as part of object detector 216.

[0335]

[0382] Intersection Violation Prediction

[0336]

[0383] 19A-19B show another example of object detection where the object detected by the object detection unit 216 is related to an intersection. As shown in FIG. 19A, the object detection unit 216 may be configured to detect a traffic light 780. As shown in FIG. 19B, the object detection unit 216 may be configured to detect a stop sign 790. The object detection unit 216 may also be configured to detect other items related to the intersection, such as road signs, curb corners, ramps, etc.

[0337]

[0384] In some embodiments, the intersection violation predictor 222 is configured to detect an intersection based on the detected object 216 detected by the object detector 216. In some cases, the object detector 216 may detect a stop line at the intersection indicating a planned stopping position of the target vehicle. The intersection violation predictor 222 may determine a time-to-crossing (TTC) based on the position of the stop line and the speed of the target vehicle. For example, the intersection violation predictor 222 may determine a distance d between the target vehicle and the position of the stop line and calculate the TTC based on the formula TTC=d / V, where V is the speed of the target vehicle. Also, in some embodiments, as shown in FIG. 19B, the intersection violation predictor 222 may be configured to determine a line 792 corresponding to the detected stop line and perform a calculation to determine the TTC based on the line 792. In some cases, if a stop line is not detected by the object detector 216, the intersection violation predictor 222 may estimate an expected stopping position based on an object detected at the intersection. For example, the intersection violation predictor 222 can estimate an expected stopping location based on known relative positions between the expected stopping location and surrounding objects such as stop signs, traffic lights, etc.

[0338]

[0385] In some embodiments, instead of determining the TTC, the intersection violation predictor 222 may be configured to determine a time-to-brake (TTB) based on the location of the stop line and the speed of the target vehicle. The TTB measures the time the driver has left at the current speed to initiate a braking maneuver to safely stop at or before the required stop position associated with the intersection. For example, the intersection violation predictor 222 may determine a distance d between the target vehicle and the location of the stop line and calculate the TTB based on the current speed of the target vehicle. In some embodiments, the intersection violation predictor 222 may be configured to determine a braking distance BD indicating the distance required for the vehicle to come to a complete stop based on the speed of the vehicle, and determine the TTB based on the braking distance. The faster the traveling speed, the longer the braking distance. In some embodiments, the braking distance may also be based on road conditions. For example, the braking distance may be longer on a wet road surface compared to a dry road surface for the same vehicle speed. FIG. 19C illustrates the difference in braking distance required for various vehicle speeds and various road conditions. For example, as shown, a vehicle traveling at 40 km / h requires a braking distance of 9 m on a dry road and 13 m on a wet road. On the other hand, a vehicle traveling at 110 km / h requires a braking distance of 67 m on a dry road and 97 m on a wet road. Figure 19C also shows how far a vehicle will travel if the driver's reaction time is 1.5 seconds. For example, a vehicle traveling at 40 km / h will travel 17 m in about 1.5 seconds (driver's reaction time) before the driver applies the brakes. Therefore, the total distance required to stop a vehicle traveling at 40 km / h is 26 m on a dry road and 30 m on a wet road (taking into account the driver's reaction time).

[0339]

[0386] In some embodiments, the intersection violation predictor 222 can determine the TTB based on the formula: TTB=(d-BD) / V, where V is the speed of the vehicle. Since d is the distance from the current vehicle position to a stop position (e.g., a stop line) and BD is the braking distance, the term (d-BD) represents the remaining distance the subject vehicle has to travel during which the driver may react to the environment before braking the vehicle. Thus, the term (d-BD) / V represents the time the driver has to react to the environment before braking the vehicle. In some embodiments, if TTB=(d-BD) / V<=threshold reaction time, the intersection violation predictor 222 can generate a control signal to activate a device that alerts the driver and / or activates a device that automatically controls the vehicle, as described herein.

[0340]

[0387] In some embodiments, this threshold reaction time can be 1 second or more, 1.5 seconds or more, 2 seconds or more, 2.5 seconds or more, 3 seconds or more, 4 seconds or more, and the like.

[0341]

[0388] Also, in some embodiments, the threshold reaction time may be variable based on the driver's state identified by the driver monitoring module 211. For example, in some embodiments, if the driver monitoring module 211 determines that the driver is distracted, the processing unit 210 may increase the threshold reaction time (e.g., from 2 seconds for a non-distracted driver to 4 seconds for a distracted driver). Furthermore, in some embodiments, the threshold reaction time may have different values ​​for different driver states. For example, if the driver is distracted, the threshold reaction time may be 4 seconds, and if the driver is drowsy, the threshold reaction time may be 6 seconds.

[0342]

[0389] In some embodiments, the intersection violation predictor 222 may be configured to determine the distance d between the target vehicle and the stopping position by analyzing images from the first camera 202. Alternatively or additionally, the intersection violation predictor 222 may receive information indicating the location of the target vehicle and the location of the intersection from a GPS system. In such a case, the intersection violation predictor 222 may determine the distance d based on the location of the target vehicle and the location of the intersection.

[0343]

[0390] In some embodiments, the intersection violation predictor 222 may determine the braking distance BD by referencing a table that maps different vehicle speeds to respective braking distances. In other embodiments, the intersection violation predictor 222 may determine the braking distance BD by performing a calculation based on a model (e.g., an equation) that receives the vehicle speed as an input and outputs the braking distance. Also, in some embodiments, the processing unit 210 may receive information indicative of road conditions and determine the braking distance BD based on the road conditions. For example, in some embodiments, the processing unit 210 may receive an output from a moisture sensor that indicates that it is raining. In such a case, the processing unit 210 may determine a high value for the braking distance BD.

[0344]

[0391] In some embodiments, instead of or in addition to determining the TTB, the intersection violation predictor 222 may be configured to determine a braking distance BD based on the speed V of the target vehicle (and optionally based on road conditions and / or vehicle dynamics) and may generate a control signal if the braking distance BD is less than a distance d to the intersection (e.g., the distance between the target vehicle and a predicted stopping position associated with the intersection) or if d-BD<=a distance threshold. The control signal may activate a device that generates a warning to the driver and / or activate a device that controls the vehicle, as described herein. In some embodiments, the distance threshold may be adjusted based on the driver's state. For example, if the driver monitoring module 211 determines that the driver is distracted, the processing unit 210 may increase the distance threshold to account for the driver having a longer distance to react.

[0345]

[0392] Distance Estimation

[0393] In one or more embodiments described herein, the processing unit 210 may be configured to determine a distance d between the target vehicle and a location in front of the vehicle, which may be the location of an object (e.g., a leading vehicle, a pedestrian, etc.) captured in an image from the first camera 202, an expected stopping location of the vehicle, etc. Various techniques may be employed in different embodiments to determine the distance d.

[0346]

[0394] In some embodiments, the processing unit 210 is configured to determine the distance d based on a Y:d mapping, where Y represents a y coordinate in the image frame and d represents the distance between the target vehicle and a location in the image frame corresponding to the Y coordinate. This concept is illustrated in the example of FIG. 20, which shows an example of a technique for determining the distance d between the target vehicle and a location in front of the vehicle. In the graph above, various widths of the bounding box of the preceding vehicle detected in the camera image are plotted against their respective y coordinates (i.e., the y component of each location of the bounding box of the detected object in the camera image), so that a best fit line relating the y coordinates to each width of the bounding box can be determined. The y coordinates in the graph above are based on a coordinate system with the origin y=0 at the top of the camera image. In other embodiments, the y coordinates may be based on other coordinate systems (e.g., a coordinate system with the origin y=0 at the bottom of the image or in the center of the image). In the illustrated example, the larger the y coordinate value, the larger the bounding box width. This is because a vehicle detected closer to the camera will appear larger (with a larger corresponding bounding box) and closer to the bottom of the camera image than another vehicle that is farther away from the camera. Also, in the illustrated example, the best fit line for the top graph in Figure 20 has a linear equation with two parameters B=-693.41 and m=1.46, where B is the value when y=0 and m is the slope of the best fit line.

[0347]

[0395] It should be noted that the width (or horizontal dimension) in the camera image coordinate system is related to a real-world distance d based on the principle of homography. Thus, the width parameter in the top graph of FIG. 20 can be converted to a real-world distance d based on perspective projection geometry in some embodiments. In some embodiments, the width:distance mapping may be obtained empirically by performing calculations based on perspective projection geometry. In other embodiments, the width:distance mapping may be obtained by measuring the actual distance d between the camera and an object at a location and identifying the width of the object in the coordinate system of a camera image that captures the object at a distance d from the camera. Also, in further embodiments, instead of determining a width:distance mapping, a y:d mapping may be determined, which can be determined by measuring the actual distance d between the camera and a real-world location L and identifying the y coordinate of location L in the coordinate system of the camera image.

[0348]

[0396] The information in the bottom graph of FIG. 20 may be used by the processing unit 210 to determine the distance d in some embodiments. For example, in some embodiments, information relating y coordinates to distance d may be stored in a non-transitory medium. This information may be an equation of a curve relating distance d to different y coordinates, a table containing different y coordinates and corresponding distances d, etc. In operation, the processing unit 210 may detect an object (e.g., a vehicle) in a camera image from the first camera 202. An image of the detected object displayed in the camera image has coordinates (x, y) relative to the coordinate system of the camera image. For example, if the y coordinate of the detected object has a value of 510, then based on the curve in FIG. 20, the distance of the detected object from the camera / target vehicle is approximately 25 m.

[0349]

[0397] As another example, during operation, the processing unit 210 may determine a location in the camera image that represents a desired stopping position for the target vehicle. The location in the camera image has coordinates (x, y) relative to the coordinate system of the camera image. For example, if the y coordinate of the location (representing the desired position for the target vehicle) has a value of 490, then based on the curve in FIG. 20, the distance d between the desired stopping position (e.g., a stop line at a real intersection or an artificially created stop line) and the camera / target vehicle is approximately 50 meters.

[0350]

[0398] It should be noted that the manner in which the distance d is determined is not limited to the described example, and processing unit 210 may utilize other techniques for determining distance d. For example, in other embodiments, processing unit 210 may receive distance information from a distance sensor, such as a sensor that utilizes time-of-flight techniques for distance determination.

[0351]

[0399] Warning and / or automatic vehicle control device

[0352]

[0400] As described herein, the signal generation controller 224 is configured to generate a control signal for operating a warning generator and / or a control signal for controlling a target vehicle based on the output from the collision prediction unit 218 or the intersection violation prediction unit 222 and based on the output from the driver monitoring module 211 indicating the driver's state. The output from the collision prediction unit 218 or the intersection violation prediction unit 222 may be a TTC value indicating the time to collision (with another vehicle or other object) or the time to clear the detected intersection.

[0353]

[0401] In some embodiments, the signal generation controller 224 is configured to compare the (time-varying) TTC value with a threshold (threshold time) and determine whether to generate a control signal based on the comparison result. In some embodiments, the threshold utilized by the signal generation controller 224 of the processing unit 210 to determine whether to generate a control signal (in response to a predicted crash or a predicted intersection violation) may have a minimum value that is at least 1 second, or 2 seconds, or 3 seconds, or 4 seconds, or 5 seconds, or 6 seconds, or 7 seconds, or 8 seconds, or 9 seconds, or 10 seconds. This threshold is variable based on the driver's state indicated by the information provided by the driver monitoring module 211. For example, if the driver's state indicates that the driver is distracted, the processing unit 210 may adjust the threshold by increasing the threshold time from its minimum value (e.g., if the minimum value is 3 seconds, the threshold may be adjusted to be 5 seconds). On the other hand, if the driver's state indicates drowsiness, the processing unit 210 may adjust the threshold to be, for example, 7 seconds (i.e., 5 seconds or more in this embodiment). This is because a drowsy driver may take longer to recognize the risk of a collision or a requirement to stop and to take action to mitigate the risk of a collision.

[0354]

[0402] FIG. 21 illustrates an example of a technique for generating a control signal to control the vehicle and / or generating a warning to the driver. In this example, the collision prediction unit 218 determines that the TTC is 10 seconds. The x-axis of the graph is the time elapsed since the TTC determination. At time t=0, an initial TTC=10 seconds was determined by the collision prediction unit 218. As time passes (represented by the x-axis), the TTC (represented by the y-axis) decreases based on the relationship TTC=10-t, where 10 is the initially determined time to collision TTC of 10 seconds. As time passes, the TTC also decreases as the predicted collision approaches in time. In the illustrated example, the processing unit 210 utilizes a first threshold TH1 of 3 seconds to provide a control signal (to warn the driver and / or automatically activate a vehicle control device to mitigate the risk of a collision) when the driver state output by the driver monitoring module 211 indicates that the driver is not distracted. Also in the illustrated example, the processing unit 210 utilizes a second threshold TH2 of 5 seconds to provide a control signal (to warn the driver and / or automatically activate a vehicle control device to mitigate the risk of a collision) when the driver status output by the driver monitoring module 211 indicates that the driver is distracted. In the illustrated example, the processing unit 210 utilizes a third threshold TH3 of 8 seconds to provide a control signal (to warn the driver and / or automatically activate a vehicle control device to mitigate the risk of a collision) when the driver status output by the driver monitoring module 211 indicates that the driver is drowsy.

[0355]

[0403] As shown in Fig. 21, four different scenarios are presented. In scenario 1, the output of the driver monitoring module 211 indicates that the driver is not distracted (N) for 7 seconds from t=0 (corresponding to a TTC of 3 seconds). Therefore, the signal generation controller 224 uses a first threshold TH1 (TTC=3 seconds, corresponding to t=7 seconds) as the time to provide a control signal CS (for operating a warning device and / or a vehicle control device). In other words, when the TTC decreases from the initial 10 seconds to reach the threshold of 3 seconds, the signal generation controller 224 provides the control signal CS.

[0356]

[0404] In scenario 2, the output of the driver monitoring module 211 indicates that the driver is not distracted (N) from t=0 to t=1.5 seconds, and is distracted (D) from t=1.5 to 5 seconds onwards (corresponding to a TTC of 5 seconds). Thus, the signal generation controller 224 uses a second threshold TH2 (TTC=5 seconds, corresponding to t=5 seconds) as the time to provide a control signal CS (to operate a warning device and / or a vehicle control device). In other words, when the TTC decreases from the initial 10 seconds and reaches the 5 second threshold, the signal generation controller 224 provides the control signal CS. Thus, in a situation where the driver is distracted, the signal generation controller 224 provides the control signal earlier to warn the driver and / or operate the vehicle.

[0357]

[0405] In scenario 3, the output of the driver monitoring module 211 indicates that the driver is not distracted (N) from t=0 to t=2 seconds, distracted (D) from t=2 seconds to 3.5 seconds, and not distracted again (N) from t=3.5 seconds to 5 seconds onward. When the second threshold TH2 at t=5 seconds is reached, the driver's state is not distracted (N), but the signal generation controller 224 still uses the second threshold TH2 (for distracted state) because the driver's state in this scenario changed from distracted to non-distracted just before the threshold TH2 at t=5 seconds. Thus, in some embodiments, the signal generation controller 224 may be configured to consider the driver's state within a time window before the threshold (e.g., 1.5 seconds, 2 seconds, etc. before TH2) to determine whether to use the threshold to determine the generation of the control signal. In other embodiments, the signal generation controller 224 may be configured to consider the driver's state at the time of the threshold to determine whether to use the threshold.

[0358]

[0406] In scenario 4, the output of the driver monitoring module 211 indicates that the driver is drowsy (R) from t=0 to t=2 seconds and thereafter (corresponding to a TTC of 8 seconds). Thus, the signal generation controller 224 uses a third threshold TH3 (TTC=8 seconds, corresponding to t=2 seconds) as the time to provide a control signal CS (to operate a warning device and / or a vehicle control device). In other words, when the TTC decreases from the initial 10 seconds and reaches the 8 second threshold, the signal generation controller 224 provides the control signal CS. Thus, in a driver drowsy situation, the signal generation controller 224 provides a control signal even earlier (i.e., earlier than when the driver is awake but distracted) to warn the driver and / or allow him to operate the vehicle.

[0359]

[0407] Thus, as shown in the examples above, in some embodiments the thresholds are variable in real time based on the driver's state as determined by the driver monitoring module 211.

[0360]

[0408] In any of the above scenarios, if the signal generating controller 224 receives sensor information (e.g., information provided by sensor 225) indicating that the driver is manipulating the vehicle (e.g., applying the brakes) to reduce the risk of a collision, the signal generating controller 224 may withhold providing the control signal.

[0361]

[0409] Although the example and four scenarios of FIG. 21 are described in relation to crash prediction, they can also be applied to intersection violation prediction. For intersection violation prediction, the value of TTC indicates the time to clear the intersection. In some embodiments, the same thresholds TH1, TH2, TH3 for determining when to provide a control signal (activate a warning generator and / or activate a vehicle control device) for crash prediction can also be used for intersection violation prediction. In other embodiments, the thresholds TH1, TH2, TH3 for determining when to provide a control signal for crash prediction can be different from the thresholds TH1, TH2, TH3 for determining when to provide a control signal for intersection violation prediction.

[0362]

[0410] As shown in the examples above, in some embodiments, the crash prediction unit 218 is configured to determine an estimated time until the predicted crash occurs, and the signal generation controller 224 of the processing unit 210 is configured to provide a control signal to activate a device when the estimated time until the predicted crash occurs is less than a threshold. In some embodiments, the device includes a warning generator, and the signal generation controller 224 of the processing unit 210 is configured to provide a control signal to cause the device to provide a warning to the driver when the estimated time until the predicted crash occurs is less than a threshold. Alternatively or additionally, the device may include a vehicle control device, and the signal generation controller 224 of the processing unit 210 is configured to provide a control signal to cause the device to control the vehicle when the estimated time until the predicted crash occurs is less than a threshold.

[0363]

[0411] Also, as shown in the examples above, in some embodiments, the signal generation controller 224 of the processing unit 210 is configured to iteratively evaluate the estimated time to occur (TTC) with respect to a variable threshold as the predicted crash / intersection violation approaches in time as the estimated time to occur decreases.

[0364]

[0412] Also, in some embodiments, the processing unit 210 (e.g., the signal generation controller 224 of the processing unit 210) is configured to increase the threshold if the driver's state indicates that the driver is distracted or not paying attention to the driving task.

[0365]

[0413] Also, as shown in the examples above, in some embodiments, the signal generation controller 224 of the processing unit 210 is configured to at least temporarily withhold providing a control signal if the estimated time until a predicted collision occurs is greater than a threshold value.

[0366]

[0414] In some embodiments, the threshold has a first value when the driver's state indicates the driver is paying attention to the driving task, and the threshold has a second value higher than the first value when the driver's state indicates the driver is distracted or not paying attention to the driving task.

[0367]

[0415] Also, as shown in the above example, in some embodiments, the threshold value is also based on sensor information indicating that the vehicle is being operated to reduce the risk of a collision. For example, if the sensor 225 provides sensor information indicating that the driver is braking the vehicle, the processing unit 210 may increase the threshold value to a higher value. In some embodiments, the signal generation controller 224 of the processing unit 210 is configured to determine whether to provide a control signal based on (1) the first information indicating the risk of a collision with the vehicle, (2) the second information indicating the state of the driver, and (3) the sensor information indicating that the vehicle is being operated to reduce the risk of a collision.

[0368]

[0416] In some embodiments, the processing unit 210 is configured to determine a level of risk of a collision, and the processing unit 210 (e.g., the signal generation controller 224 of the processing unit 210) is configured to adjust the threshold value based on the determined level of risk of a collision.

[0369]

[0417] In some embodiments, the driver's state includes a distracted state, the processing unit 210 is configured to determine a level of the driver's distracted state, and the processing unit 210 (e.g., the signal generation controller 224 of the processing unit 210) is configured to adjust the threshold based on the determined level of the driver's distracted state.

[0370]

[0418] Also, in some embodiments, different warnings may be provided based on whether the driver is paying attention at different thresholds. For example, in some embodiments, the processing unit 210 may control the device to provide a first warning with a first characteristic if there is a risk of collision (with a vehicle, pedestrian, etc.) and the driver is paying attention, and to provide a second warning with a second characteristic if there is a risk of collision and the driver is distracted. The first characteristic of the first warning may be a first warning volume, and the second characteristic of the second warning may be a second warning volume that is higher than the first warning volume. Also, in some embodiments, if the processing unit 210 determines that there is a higher risk of collision, the processing unit 210 may control the device to provide a stronger warning (e.g., a louder warning and / or a higher frequency beeping warning). Thus, in some embodiments, a softer warning may be provided when the target vehicle is approaching the object, and a stronger warning may be provided when the target vehicle is closer to the object.

[0371]

[0419] Similarly, in some embodiments, the processing unit 210 may control the device to provide a first alert having a first characteristic when there is a risk of an intersection violation and the driver is paying attention, and to provide a second alert having a second characteristic when there is a risk of an intersection violation and the driver is distracted. The first characteristic of the first alert may be a first alert volume, and the second characteristic of the second alert may be a second alert volume that is higher than the first alert volume. Also, in some embodiments, if the processing unit 210 determines that there is a higher risk of an intersection violation, the processing unit 210 may control the device to provide a stronger alert (e.g., a higher volume alert and / or a higher beep frequency alert). Thus, in some embodiments, a gentler alert may be provided when the target vehicle is approaching the intersection, and a stronger alert may be provided when the target vehicle is closer to the intersection.

[0372]

[0420] As shown in the above examples, the apparatus 200 advantageously considers the driver's state when determining whether to generate a control signal to activate a device that provides a warning and / or to activate a device that controls the vehicle. The driver's state can be used to adjust the monitoring thresholds, so that the apparatus 200 may provide a warning to the driver and / or control the vehicle to reduce the risk of a collision and / or the risk of an intersection violation earlier by considering the particular state of the driver (e.g., the driver is distracted, drowsy, etc.). For example, in some embodiments, the apparatus 200 may warn the driver and / or control the vehicle at least 2 seconds before the predicted risk (e.g., risk of collision or risk of an intersection violation), or even earlier, such as at least 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 11 seconds, 12 seconds, 13 seconds, 14 seconds, 15 seconds, etc. In addition, in conventional monitoring systems that do not consider the driver's state, greater accuracy is built into the system to avoid false positives at the expense of increased sensitivity. By incorporating the driver's state, the device 200 can be configured to operate with low sensitivity (e.g., lower than or equal to existing solutions) and increase the sensitivity of the device 200 only when the driver is inattentive. Increased sensitivity based on the driver's state can be achieved by adjusting one or more thresholds based on the driver's state, such as a threshold for determining time to collision, a threshold for determining time to traverse an intersection, a threshold for determining time to brake, a threshold for determining whether an object intersects a region of interest (e.g., a camera calibration ROI, a ROI determined based on centerline detection, etc.), a threshold for the confidence of object detection, etc.

[0373]

[0421] Tailgating

[0374]

[0422] In some embodiments, the processing unit 210 may be configured to consider scenarios in which the target vehicle is following. In some embodiments, following is determined (e.g., measured) by time-to-headway, which is defined as the distance to the leading vehicle divided by the speed of the target vehicle (ego vehicle). In some embodiments, the speed of the target vehicle may be obtained from a vehicle speed detection system. In other embodiments, the speed of the target vehicle may be obtained from a GPS system. In further embodiments, the speed of the target vehicle may be determined by the processing unit 210 processing external images received from the first camera 202 of the apparatus 200. Also, in some embodiments, the distance to the leading vehicle may be determined by the processing unit 210 processing external images received from the first camera 202. In other embodiments, the distance to the leading vehicle may be obtained from a distance sensor, such as a sensor employing time-of-flight technology.

[0375]

[0423] In some embodiments, processing unit 210 may determine that a pursuit condition exists if the time gap is less than a pursuit threshold, which may be, by way of non-limiting example, 2 seconds or less, 1.5 seconds or less, 1 second or less, 0.8 seconds or less, 0.6 seconds or less, 0.5 seconds or less, etc.

[0376]

[0424] In some embodiments, the processing unit 210 may be configured to determine that there is a risk of collision if the target vehicle is following and the driver monitoring module 211 determines that the driver is distracted. The processing unit 210 may then generate control signals to cause a device (e.g., an alert generator) to provide an alert to the driver and / or to cause a device (e.g., a vehicle controller) to control the vehicle, as described herein. For example, the vehicle controller may automatically apply the brakes of the vehicle, automatically release the accelerator pedal, automatically turn on the hazard lights, or any combination thereof.

[0377]

[0425] Rolling Stop

[0378]

[0426] In some embodiments, the processing unit 210 may include a rolling stop module configured to detect a rolling stop maneuver. The rolling stop module may be implemented as part of the intersection violation predictor 222 in some embodiments. In operation, the processing unit 210 may detect an intersection where the vehicle needs to stop (e.g., the processing unit 210 may identify a stop sign, a red light, etc. based on processing of images from the first camera 202). The rolling stop module may monitor one or more parameters indicative of the vehicle's operation to determine whether the vehicle is performing a rolling stop maneuver at the intersection. For example, the rolling stop module may obtain the vehicle's speed, the vehicle's braking, the vehicle's deceleration, etc., or any combination thereof. In some embodiments, the rolling stop module may determine that there is a rolling stop maneuver by analyzing the vehicle's speed profile over a period of time that the vehicle is approaching the intersection. For example, if the vehicle decelerates (indicating that the driver is aware of the intersection) and the vehicle's speed does not further decrease within a period of time, the rolling stop module may determine that the driver is performing a rolling stop maneuver. As another example, if the vehicle slows down (indicating the driver is aware of the intersection) and the vehicle's speed begins to increase as the vehicle approaches the intersection, the rolling stop module may determine that the driver is performing a rolling stop maneuver. In another approach, if the vehicle's speed decreases as it approaches the intersection, but the vehicle's speed does not decrease enough to reach a certain threshold within a certain distance of the required stopping position, the rolling stop module may determine that the driver is performing a rolling stop maneuver.

[0379]

[0427] In some embodiments, if the rolling stop module determines that the vehicle has not come to a complete stop (e.g., the driver slows down in response to a stop sign or red light but does not come to a complete stop), the intersection violation prediction unit 222 may determine that there is a risk of an intersection violation. In response to the determined risk of an intersection violation, the rolling stop module may generate a control signal to activate a device. For example, the control signal may operate a communication device to wirelessly send a message to a server system (e.g., a cloud system). The server system may be utilized by a fleet management company to provide guidance to drivers or by an insurance company to identify risky drivers. Alternatively or additionally, the control signal may activate a warning system to provide a warning to the driver, which may serve as a way to provide guidance to drivers. Alternatively or additionally, the control signal may activate a braking system of the vehicle to control the vehicle to come to a complete stop.

[0380]

[0428] method

[0381]

[0429] 22A illustrates a method 800 performed by the apparatus 200 of FIG. 2A according to some embodiments. The method 800 includes acquiring a first image generated by a first camera, the first camera configured to view an environment external to the vehicle (item 802), acquiring a second image generated by a second camera, the second camera configured to view a driver of the vehicle (item 804), determining first information indicative of a risk of collision with a vehicle based at least in part on the first image (item 806), determining second information indicative of a driver's state based at least in part on the second image (item 808), and determining whether to provide a control signal to operate a device based on (1) the first information indicative of a risk of collision with a vehicle and (2) the second information indicative of the driver's state (item 810).

[0382]

[0430] Optionally, in the method 800, the first information is determined by predicting a collision, the collision being predicted at least 3 seconds or more in advance of an expected occurrence time of the predicted collision.

[0383]

[0431] Optionally, in method 800, the first information is determined by predicting a collision, the collision being predicted with sufficient lead time for the driver's brain to process the input and for the driver to take action to mitigate the risk of the collision.

[0384]

[0432] Optionally, in the method 800, the sufficient lead time depends on the driver's status.

[0385]

[0433] Optionally, in method 800, the first information indicative of a risk of a collision includes a predicted collision, the method further comprising determining an estimated time until the predicted collision occurs, and the control signal is provided to cause a device to provide a control signal if the estimated time until the predicted collision occurs is less than a threshold.

[0386]

[0434] Optionally, in method 800, the device includes a warning generator, and the control signal is provided such that the device issues a warning to the driver if the estimated time until a predicted crash occurs is less than a threshold.

[0387]

[0435] Optionally, in method 800, the device includes a vehicle controller, and the control signal is provided to cause the device to control the vehicle if an estimated time until a predicted collision occurs is less than a threshold.

[0388]

[0436] Optionally, in the method 800, the threshold is variable based on second information indicative of a driver's state.

[0389]

[0437] Optionally, in method 800, the estimated time is iteratively evaluated with respect to a variable threshold as the predicted crash approaches in time as the estimated time until the predicted crash occurs decreases.

[0390]

[0438] Optionally, in the method 800, the threshold is variable in real time based on the driver's state.

[0391]

[0439] Optionally, in method 800, the method further comprises modifying the threshold if a driver's state indicates that the driver is distracted or not paying attention to the driving task.

[0392]

[0440] Optionally, the method 800 further includes at least temporarily withholding generation of the control signal if the estimated time until the predicted collision occurs is greater than a threshold.

[0393]

[0441] Optionally, method 800 further includes determining a level of risk of collision and adjusting the threshold based on the determined level of risk of collision.

[0394]

[0442] Optionally, in method 800, the driver's state includes a distracted state, and the method further includes determining a level of the driver's distracted state and adjusting the threshold based on the determined level of the driver's distracted state.

[0395]

[0443] Optionally, in method 800, the threshold has a first value when the driver's state indicates the driver is paying attention to the driving task, and the threshold has a second value higher than the first value when the driver's state indicates the driver is distracted or not paying attention to the driving task.

[0396]

[0444] Optionally, in method 800, the threshold is also based on sensor information indicating that the vehicle is being operated to reduce the risk of a collision.

[0397]

[0445] Optionally, in method 800, the step of determining whether to provide a control signal to operate the device is also performed based on sensor information indicating that the vehicle is being operated to reduce the risk of a collision.

[0398]

[0446] Optionally, in the method 800, the step of determining first information indicative of the risk of collision comprises processing the first image based on a first model.

[0399]

[0447] Optionally, in the method 800, the first model includes a neural network model.

[0400]

[0448] Optionally, in the method 800, determining second information indicative of the driver's state comprises processing the second image based on a second model.

[0401]

[0449] Optionally, the method 800 further includes determining a metric value for each of the plurality of pose classifications and determining whether the driver is engaged in the driving task based on the one or more metric values.

[0402]

[0450] Optionally, in method 800, the pose classifications include two or more of: looking down poses, looking up poses, looking left poses, looking right poses, poses using a mobile phone, poses smoking poses, poses holding an object, poses with hands off the steering wheel, poses not wearing a seat belt, poses with eyes closed, poses looking ahead, poses with one hand on the steering wheel, and poses with both hands on the steering wheel.

[0403]

[0451] Optionally, method 800 further comprises comparing the metric value to a respective threshold value for each pose classification.

[0404]

[0452] Optionally, the method 800 further includes determining that the driver belongs to one of the pose classifications if one of the corresponding metric values ​​meets or exceeds one of the corresponding thresholds.

[0405]

[0453] Optionally, method 800 is performed by an aftermarket device, and said first camera and second camera are integrated as parts of said aftermarket device.

[0406]

[0454] Optionally, in method 800, the second information is determined by processing the second image to determine whether the image of the driver satisfies a pose classification, the method further comprising determining whether the driver is engaged in a driving task based on whether the image of the driver satisfies a pose classification.

[0407]

[0455] Optionally, in the method 800, determining second information indicative of the driver's state comprises processing the second image based on a neural network model.

[0408]

[0456] 22B illustrates a method 850 performed by the apparatus 200 of FIG. 2A according to some embodiments. The method 850 includes acquiring a first image generated by a first camera, the first camera configured to view an environment external to the vehicle (item 852), acquiring a second image generated by a second camera, the second camera configured to view a driver of the vehicle (item 854), determining first information indicative of an intersection violation risk based at least in part on the first image (item 856), determining second information indicative of a driver's state based at least in part on the second image (item 858), and determining whether to provide a control signal to operate a device based on (1) the first information indicative of an intersection violation risk and (2) the second information indicative of the driver's state (item 860).

[0409]

[0457] Optionally, the first information is determined by predicting an intersection violation, the predicted intersection violation being predicted at least 3 seconds or more in advance of an expected occurrence time of the predicted intersection violation.

[0410]

[0458] Optionally, the first information is determined by predicting an intersection violation, the predicted intersection violation predicted with sufficient lead time for the driver's brain to process the input and for the driver to take action to mitigate the risk of the intersection violation.

[0411]

[0459] Optionally, the sufficient lead time depends on the driver's status.

[0412]

[0460] Optionally, the first information indicative of a risk of an intersection violation includes a predicted intersection violation, and the method further includes determining an estimated time until the predicted intersection violation occurs, and the control signal is provided to cause the device to provide a control signal if the estimated time until the predicted intersection violation occurs is less than a threshold.

[0413]

[0461] Optionally, the device comprises a warning generator, and the control signal is provided to cause the device to issue a warning to the driver when an estimated time until a predicted intersection violation occurs is less than a threshold.

[0414]

[0462] Optionally, the device includes a vehicle controller, and the control signal is provided to cause the device to control the vehicle if an estimated time until a predicted intersection violation occurs is less than a threshold.

[0415]

[0463] Optionally, the threshold is variable based on second information indicative of a driver's condition.

[0416]

[0464] Optionally, the estimated time is iteratively evaluated with respect to a variable threshold as the predicted intersection violation approaches in time as the estimated time until the predicted intersection violation occurs decreases.

[0417]

[0465] Optionally, the threshold is variable in real time based on the driver's state.

[0418]

[0466] Optionally, the method further comprises modifying the threshold if a driver's state indicates that the driver is distracted or not paying attention to the driving task.

[0419]

[0467] Optionally, the method further comprises at least temporarily withholding generation of the control signal if the estimated time until the predicted intersection violation occurs is greater than a threshold.

[0420]

[0145] Optionally, the method further includes determining a level of risk of an intersection violation and adjusting the threshold value based on the determined level of risk of an intersection violation.

[0421]

[0469] Optionally, the driver's state includes a distracted state, and the method further includes determining a level of the driver's distracted state, and adjusting the threshold based on the determined level of the driver's distracted state.

[0422]

[0470] Optionally, the threshold has a first value when the driver's state indicates the driver is paying attention to the driving task, and the threshold has a second value higher than the first value when the driver's state indicates the driver is distracted or not paying attention to the driving task.

[0423]

[0471] Optionally, the threshold is also based on sensor information indicating that the vehicle is being operated in a manner that reduces the risk of an intersection violation.

[0424]

[0472] Optionally, the step of determining whether to provide a control signal to operate the device is also performed based on sensor information indicating that the vehicle is being operated in a manner that reduces the risk of an intersection violation.

[0425]

[0473] Optionally, the step of determining first information indicative of the risk of an intersection violation comprises processing the first image based on a first model.

[0426]

[0474] Optionally, the first model comprises a neural network model.

[0427]

[0475] Optionally, the step of determining second information indicative of the driver's state comprises processing the second image based on a second model.

[0428]

[0476] Optionally, the method further comprises determining a metric value for each of the plurality of pose classifications and determining whether the driver is engaged in a driving task based on the one or more metric values.

[0429]

[0477] Optionally, the pose classifications include two or more of: looking down poses, looking up poses, looking left poses, looking right poses, poses using a mobile phone, poses smoking poses, poses holding an object, poses with hands off the steering wheel, poses not wearing a seat belt, poses with eyes closed, poses looking ahead, poses with one hand on the steering wheel, and poses with both hands on the steering wheel.

[0430]

[0478] Optionally, the method further comprises the step of comparing the metric value to a respective threshold value for each pose classification.

[0431]

[0479] Optionally, the method further comprises determining that the driver belongs to one of the pose classifications if one of the corresponding metric values ​​meets or exceeds one of the corresponding thresholds.

[0432]

[0480] Optionally, the method is performed by an aftermarket device, and the first camera and second camera are integrated as parts of the aftermarket device.

[0433]

[0481] Optionally, the second information is determined by processing the second image to determine whether an image of the driver satisfies a pose classification, and the method further comprises determining whether the driver is engaged in a driving task based on whether the image of the driver satisfies a pose classification.

[0434]

[0482] Optionally, the step of determining second information indicative of the driver's state comprises processing the second image based on a neural network model.

[0435]

[0483] Creating and embedding models

[0436]

[0484] FIG. 23 illustrates a technique for determining a model to be used by the device 200, according to some embodiments. As shown, there may be a number of vehicles 910a-910d with respective devices 200a-200d. Each of the devices 200a-200d may have the configuration and characteristics described with reference to the device 200 of FIG. 2A. In operation, the cameras (both exterior and interior monitoring cameras) of the devices 200b-200d mounted on the vehicles 910b-910d capture images of the environment outside the respective vehicles 910b-910d and images of the respective drivers. The images are transmitted, directly or indirectly, over a network (e.g., cloud, internet) to a server 920. The server 920 includes a processing unit 922 configured to process images from the devices 200b-300d of the vehicles 910b-910d to determine a model 930, and one or more models 932. The model 930 may be configured to detect the pose of the driver, and the model 932 may be configured to detect various types of objects in the camera image. The models 930, 932 may be stored in a non-transitory medium 924 in the server 920. The server 920 may directly or indirectly transmit the models 930, 932 to the device 200a of the vehicle 910a via a network (e.g., cloud, internet, etc.). The device 200a may then use the model 932 to process images received by the camera of the device 200a and detect various poses of the driver of the vehicle 910a. The device 200a may also use the model 932 to process images received by the camera of the device 200a and detect various objects outside the vehicle 910a and / or determine the area of ​​interest of the camera of the device 200a.

[0437]

[0485] 23, three devices 200b-200d for providing images are mounted on each of three vehicles 910b-910d. In other examples, there may be three or more devices 200 on each of three or more vehicles 910 for providing images to the server 920, or there may be fewer than three devices 200 on fewer than three vehicles 910 for providing images to the server 920.

[0438]

[0486] In some embodiments, the model 930 provided by the server 920 may be a neural network model. The model 932 provided by the server 920 may be one or more neural network models. In such a case, the server 920 may be a neural network, or a part of a neural network, and the images from the devices 200b-200d may be utilized by the server 920 to construct the model 930 and / or the model 932. In particular, the processing unit 922 of the server 920 may construct the model 930 and / or the model 932 by training the model 930 by machine learning. In some cases, the images from the different devices 200b-200d form a rich data set from different cameras mounted in different positions relative to the corresponding vehicle, which helps to train the model 930 and / or the model 932. As used herein, the term "neural network" refers to a computing device, system, or module composed of a number of interconnected processing elements that process information by dynamic state response to inputs. In some embodiments, the neural network may have deep learning capabilities and / or artificial intelligence. In some embodiments, a neural network may be any computing element that can be trained using one or more data sets.As non-limiting examples, the neural network may be a perceptron, a feedforward neural network, a radial basis neural network, a deep feedforward neural network, a recurrent neural network, a long / short-term memory neural network, a gated recurrent unit, an autoencoder neural network, a variational autoencoder neural network, a denoising autoencoder neural network, a sparse autoencoder neural network, a Markov chain neural network, a Hopfield neural network, a Boltzmann machine, a restricted Boltzmann machine, a deep belief network, a convolutional network, a deconvolutional network, a deep convolutional inverse graph network, a generative adversarial network, a liquid state machine, an extreme learning machine, an echo state network, a deep resolver network, a Kohonen network, a support vector machine, a neural Turing machine, a modular neural network, a sequence-to-sequence model, or the like, or any combination thereof.

[0439]

[0487] In some embodiments, the processing unit 922 of the server 920 uses the images to configure (e.g., train) the model 930 to identify specific poses of the driver. As non-limiting examples, the model 930 is configured to identify driver poses of looking down, looking up, looking left, looking right, using a cell phone, smoking, holding an object, hands off the wheel, not wearing a seat belt, eyes closed, looking ahead, one hand on the wheel, both hands on the wheel, etc. Also, in some embodiments, the processing unit 922 of the server 920 may use the images to configure a model that determines whether the driver is engaged in a driving task. In some embodiments, the determination of whether the driver is engaged in a driving task may be accomplished by the processing unit processing a pose classification of the driver. In one implementation, the pose classification may be an output provided by a neural network model. In such a case, the neural network model may be passed to the processing unit, and the processing unit may determine whether the driver is engaged in a driving task based on the pose classification from the neural network model. In other embodiments, the processing unit that receives the pose classification may be another (e.g., a second) neural network model. In this case, the first neural network model is configured to output the pose classification, and the second neural network model is configured to determine whether the driver is engaged in the driving task based on the pose classification output by the first neural network model. In such a case, model 930 may be considered to have both the first neural network model and the second neural network model. In further embodiments, model 930 may be a single neural network model configured to receive an image as an input and provide an output indicative of whether the driver is engaged in the driving task.

[0440]

[0488] Also, in some embodiments, the processing unit 922 of the server 920 uses the images to configure (e.g., train) a model 932 to detect various objects. By way of non-limiting example, the model 932 may be configured to detect vehicles, humans, animals, bicycles, traffic lights, road signs, curbs, road centerlines, etc.

[0441]

[0489] In other embodiments, model 930 and / or model 932 may not be a neural network model, but may be other types of models. In such cases, construction of model 930 and / or model 932 by processing unit 922 may not involve machine learning and / or may not require images from devices 200b-200d. Instead, construction of model 930 and / or model 932 by processing unit 922 may be accomplished by processing unit 922 determining (e.g., obtaining, calculating, etc.) processing parameters (e.g., feature extraction parameters) of model 930 and / or model 932. In some embodiments, model 930 and / or model 932 may include program instructions, commands, scripts, parameters (e.g., feature extraction parameters), etc. In one implementation, model 930 and / or model 932 may be in the form of an application wirelessly receivable by device 200.

[0442]

[0490] After the models 930 and 932 are constructed by the server 920, the models 930, 932 can be used by the devices 200 in the various vehicles 910 to identify objects in camera images. As shown, the models 930, 932 may be transmitted from the server 920 to the device 200a in the vehicle 910a. The models 930, 932 may also be transmitted from the server 920 to the devices 200b-200d in each of the vehicles 910b-910d. After the device 200a receives the models 930, 932, a processing unit in the device 200a may process images generated by the camera (internal camera) of the device 200a based on the model 930 to identify the driver's pose and / or determine whether the driver is engaged in a driving task, and process images generated by the camera (external camera) of the device 200a based on the model 932 to detect objects external to the vehicle 910a, as described herein.

[0443]

[0491] In some embodiments, the transmission of the models 930, 932 from the server 920 to the device 200 (e.g., device 200a) may be performed by the server 920 “pushing” the models 930, 932, so that the device 200 does not have to request the models 930, 932. In other embodiments, the transmission of the models 930, 932 from the server 920 may be performed by the server 920 in response to a signal generated and transmitted by the device 200. For example, the device 200 may generate and transmit a signal after the device 200 is powered on or after a vehicle carrying the device 200 is started. The signal is received by the server 920, which transmits the models 930, 932 for receipt by the device 200. As another example, the device 200 may include a user interface, such as a button, that allows a user of the device 200 to transmit a request for the models 930, 932. In such a case, when the button is pressed, the device 200 sends a request for the models 930, 932 to the server 920. In response to this request, the server 920 sends the models 930, 932 to the device 200.

[0444]

[0492] It should be noted that the server 920 in Fig. 23 is not limited to one server device, but may be multiple server devices. Also, the processing unit 922 of the server 920 may include one or more processors, one or more processing modules, etc.

[0445]

[0493] In other embodiments, the images captured by server 920 may not be generated by devices 200b-200d. Instead, the images used by server 920 to determine (e.g., train, construct, etc.) models 930, 932 may be recorded using other devices, such as a cell phone, a camera in another vehicle, etc. Also, in other embodiments, the images used by server 920 to determine (e.g., train, construct, etc.) models 930, 932 may be downloaded to server 920 from a database associated with server 920 or a database owned by a third party.

[0446]

[0494] Multi-signal prediction model

[0447]

[0495] In the above embodiment, the processing unit 210 is described as being configured to individually determine various risk factors (e.g., risk of collision, risk of intersection violation, driver distraction, speeding, etc.) and determine whether to generate a control signal to activate a device based on any of the individual risk factors meeting certain criteria. FIG. 24 illustrates an example of such an approach, where various risk factors are evaluated with respective triggers (e.g., respective criteria). In some cases, if the criteria are met for one of the triggers, the processing unit 210 may generate a control signal to generate an alert. For example, as illustrated, if a distraction parameter determined by the processing unit 210 (e.g., by processing an interior image of the driver) meets the distraction criteria, the processing unit 210 may then determine that the driver is distracted and generate a control signal to generate an alert. As another example, if a time-to-collision (TTC) parameter determined by the processing unit 210 meets the near-collision criteria, the processing unit 210 may generate a control to generate an alert. In the above examples, the criteria include a static or variable threshold, which may be user-configurable (e.g., for high, medium, or low risk tolerance). In either case, if the risk parameter is determined to meet (e.g., exceed) the threshold, the processing unit 210 may generate a control signal to cause an alert to be generated.

[0448]

[0496] In the above technique of FIG. 24, the algorithms utilizing various corresponding thresholds operate independently and do not communicate with each other. For example, an algorithm that processes a distraction risk signal does not communicate with an algorithm that processes a time-to-crash risk signal. Also, the thresholds utilized by each algorithm may not be adapted to the momentary situation or context. Thus, risk combinations, risk associations, and risk contexts are ignored. Furthermore, nonlinearities are also ignored in the above technique of FIG. 24. The rise in risk in the threshold model is linear until a threshold is reached. However, in reality, risk may escalate nonlinearly (e.g., changing into a lane where the leading vehicle starts braking, or an event is occurring further down the road, etc.). Also, the above technique does not consider various risk factors in combination with the driver's intent, the driver's attention, cognitive load, and other factors (e.g., expected reactions to the environment such as other vehicles, overtaking maneuvers, and changes in traffic control signals, etc.).

[0449]

[0497] In other embodiments, various risk factors may be processed together by the processing unit 210 to determine whether they collectively indicate a non-event (e.g., a non-hazardous event). This feature has the advantage that it can reduce false positives of the system because even if one risk factor may indicate a hazardous situation (e.g., a risk of collision), multiple risk factors considered together may collectively indicate a non-hazardous situation. In some cases, if only a single risk factor is utilized to trigger an alert, excessive false positive cases may occur and unnecessary alerts may be generated. This is undesirable as it may lead the driver to ignore the alert or turn off the in-vehicle device to avoid the false positive. Vice versa. In particular, even if a single risk factor indicates a non-hazardous situation (e.g., the risk factor does not exceed a threshold), multiple risk factors considered together may collectively indicate a hazardous situation, thereby reducing false negatives of the system. In particular, the risk factors may not individually exceed a threshold, but when considered in a fusion / holistic approach, they may collectively indicate a very risky situation.

[0450]

[0498] FIG. 25 illustrates an example of such a technique, where the processing unit 210 is configured to process multiple risk factors together and determine whether they collectively indicate a non-event (non-hazardous event) or a risk event. In the illustrated example, various risk factors (e.g., driver distraction, time to intersection violation, time to collision, speed, etc.) are provided as inputs to a model in the processing unit 210, and the processing unit 210 is configured to process two or more risk factors to determine whether they collectively indicate a hazardous situation. In the illustrated embodiment, the processing unit 210 is configured to determine a risk score (e.g., an instantaneous risk score) based on the probability of each different state (predicted event), such as the probability of a collision, the probability of a near-collision, the probability of a non-risk state, or two or more of the foregoing. In the illustrated example, the model of the processing unit 210 determines three probabilities of predicted events based on the multiple risk factors received by the model: the probability of a collision occurring within the next two seconds, the probability of a near-collision occurring within the next two seconds, and the probability of a non-risk state occurring within the next two seconds. In other embodiments, the model may determine more or less than three probabilities of a predicted event based on multiple risk factors, and in other embodiments, the time into the future for which the probabilities are determined may be greater than 2 seconds (e.g., within 3, 4, 5, 6, 7, 8, 9, 10 seconds, etc.) or less than 2 seconds (e.g., within 1.9, 1.8, 1.7, 1.6, 1.5, 1.4, 1.3, 1.2, 1.1, 1.0, 0.9, 0.8, 0.7, 0.6, 0.5 seconds, etc.).

[0451]

[0499] The model of the processing unit 210 may be a neural network model trained using prior risk factors. The neural network model may be any neural network architecture. FIGS. 26A-26C are diagrams illustrating examples of neural network architectures that may be employed to process multiple risk factors as inputs by a neural network model. The neural network architecture may be, for example, any shallow network (e.g., support-vector-machine (SVM), logistic regression, etc.), an ensemble network (e.g., random forest), or any deep network (e.g., recurrent neural network, long-short-term memory network (LSTM), convolutional neural network (CNN)), etc. The neural network model may be any type of neural network described herein.

[0452]

[0500] In some embodiments, a model may include sub-models that are implemented by respective processing units, in which case these processing units may be considered sub-processing units of processing unit 210.

[0453]

[0501] As described, the processing unit 210 may be part of the apparatus 10 that includes a first camera 202 configured to view an environment external to the vehicle and a second camera 204 configured to view a driver of the vehicle. The processing unit 210 is configured to receive a first image from the first camera and a second image from the second camera. The processing unit 210 includes a model configured to receive a plurality of inputs and generate a metric based at least in part on the inputs. The plurality of inputs includes at least a first time series of information indicative of a first risk factor and a second time series of information indicative of a second risk factor.

[0454]

[0502] In some embodiments, the apparatus 200 may include additional sensors for sensing motion characteristics associated with the operation of the vehicle. As a non-limiting example, the sensors may include one or more sensors for sensing one or more of acceleration, speed, centripetal force, steering angle, brakes, accelerator position, turn signals, etc. In this case, the processing unit 210 (e.g., models therein) may be configured to receive inputs from the sensors and determine a risk score based on such inputs. In some embodiments, such inputs may be utilized by the models to determine probabilities as described with reference to FIG. 25. Also, in some embodiments, the processing unit 210 (e.g., models therein) may be configured to receive CAN / OBD signals as inputs. In such a case, the processing unit 210 may be connected to the vehicle and obtain signals related to brakes, pedal position, steering angle, speed, etc. from a vehicle communication system. Alternatively or additionally, the processing unit 210 may be connected to an on-board diagnostic system of the vehicle and obtain signals from such a system. Inputs from the CAN / OBD and / or the vehicle's on-board diagnostic system may be utilized by models in processing unit 210 to determine the probability of various events, such as those described with reference to FIGURE 25. Additionally, in some embodiments, information received by and / or output by processing unit 210 may be transmitted via either an in-vehicle Ethernet or other data transfer architecture (e.g., one capable of providing high bandwidth data transfer, etc.).

[0455]

[0503] In further embodiments, the device 200 may include multiple sensors that collect raw data (external video, internal video, speed, etc.) as inputs and a processing unit that generates high-level, context-rich risk estimates based on the collected raw data. The processing unit 210 may be a single-stage processing unit (e.g., a single-stage processor) or may include multiple-stage processing units. In embodiments where the processing unit 210 is implemented as a single-stage processing unit, the single-stage processing unit may include an end-to-end (E2E) deep fusion risk model that takes the raw data as inputs and provides an estimate of risk based on the raw data. In embodiments where the processing unit 210 is implemented as a multiple-stage processing unit, the processing unit 210 may include at least (1) a first stage system that takes the raw data as inputs and provides a risk signal as an output, and (2) a second stage system that takes the risk signal from the first stage system and provides an estimated risk. In some embodiments, the first stage system is configured to take data (e.g., raw data) having higher dimensionality or complexity and process the data to provide an output (e.g., vector) having lower dimensionality or complexity. The low dimensionality or complexity output (risk signal) is then input to the second stage system, which processes the output to determine a risk score indicating the probability of various events (predicted events) (e.g., crash event, near crash event, non-hazardous event, etc.) and an estimated risk based on the probability of the various events. The output by the second stage system may be of lower dimensionality or complexity compared to the input (e.g., time series of risk signals) received by the second stage system. In some embodiments, the models described herein may be implemented by the second stage system. Because the estimated risk is based on the risk signals, the estimated risk may be considered as a "fused" risk that combines or incorporates risk signals (corresponding to respective risk factors).

[0456]

[0504] In some embodiments, the first stage system may include multiple first stage processing units. For example, in one embodiment, the device 200 may include multiple sensors that collect raw data (exterior video, interior video, speed, etc.), multiple first stage processing units (e.g., first stage processors) that provide individual risk signals based on the raw data, and a second stage processing unit (e.g., second stage processor) that generates a high-level, context-rich estimate of risk based on the individual risk signals. For example, there may be a first stage processing unit that determines a bounding box based on images or footage from the first camera 202. As another example, there may also be a first stage processing unit that identifies a stop line distance based on images or footage from the first camera 202. As a further example, there may also be a first stage processing unit that identifies facial landmarks based on images or footage from the second camera 204. It should be noted that the second stage processing unit is configured to provide an estimated risk based on the output from the first stage processing unit, and the output from the first stage processing unit may be considered a risk signal (e.g., metadata) for each risk factor that is different from the raw data obtained from the sensors.

[0457]

[0505] In some embodiments, the first stage processing unit may be implemented by one or more first neural network models, and the second stage processing unit may be implemented by one or more second neural network models. In some embodiments, two or more inputs (e.g., risk factors) may be determined by a first neural network model implementing the first stage processing unit and fed to a second neural network model implementing the second stage processing unit. The second neural network model considers the inputs together and determines whether the risk factors collectively result in a dangerous or non-hazardous situation.

[0458]

[0506] In some embodiments, the first stage processing unit and / or the second stage processing unit may be considered part of the processing unit 210. Also, in some embodiments, the second stage processing unit may include multiple sub-processing units.

[0459]

[0507] By way of non-limiting example, the inputs received by the model / processing unit / first stage processing unit / second stage processing unit may include distance to collision, distance to intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions (e.g., information about road surface conditions (dry, wet, snow, etc.) (which may include traction control information for slip indication), information about special zones (construction zones, school zones, etc.), information about the environment (urban, suburban, built-up area, etc.), information about traffic conditions, time of day, information about visibility conditions (e.g., fog, snow, precipitation, sun angle, glare, etc.), information about objects (e.g., labels, etc.) (stop signs, traffic lights, pedestrians, cars, animals, poles, trees, lanes, curbs, tail lights, lane mapping, etc.), object location, object movement direction, object speed. , bounding boxes (e.g. 2D bounding boxes, 3D bounding boxes, etc.), vehicle motion parameters (kinematic signals such as acceleration, speed, centrifugal force, steering angle, brake, accelerator position, turn signals, traction control, etc.), information about (identifying) the driver's state (e.g. looking up, looking down, looking left, looking right, using a mobile phone, holding an object, smoking, eyes closed, turning head or away so that face is not detected, gaze direction relative to the driver, gaze direction mapped to external objects or objects within the vehicle (e.g. head unit, mirrors, etc.), gaze changes, objects the driver is looking at, driver's physiological state (e.g. drowsiness, fatigue, road rage, stress, sudden illness (heart attack, stroke, pulse, sweating, dilated pupils, etc.), etc.), information about driving history (e.g. motor vehicle record, accident rate, experience, age, years driving, years vehicle owned, route experience, VERA (Vision Enhanced Risk Assessment)The risk assessment tools may include, but are not limited to, information provided by risk assessment tools that evaluate drivers based on their driving behavior, such as the Driver Assessment Tool (DRA), continuous driving time, proximity to meal times, information about accident history (e.g., fatal accidents at a specific time, accidents at a specific time, geospatial heat maps of risk data by location over time), sensor signals (e.g., LIDAR signals, radar signals, GPS signals, ultrasonic signals, signals communicated between the vehicle and other vehicles (e.g., distance and relative speed signals communicated via 5G), signals communicated between the vehicle and infrastructure (e.g., via 5G), or any combination of these (e.g., fusion), driver behavior, The information may include one or more combinations of information about driving activity or dynamic response (such as kinematic signals from vehicle systems (e.g., signals indicative of acceleration, speed, cornering, etc.), control signals (e.g., signals indicative of steering angle, brake position, accelerator position, turn signal state, etc.), location-specific information (e.g., any information indicative of dangerous roads, dangerous intersections, risk by location, date, and / or time (e.g., the information shown in FIG. 27 indicating the number of fatalities by time of day and day of week)), audio signals (detected speech, detected baby crying, detected loud music in the vehicle cabin, detected horn on the vehicle or outside the vehicle, etc.), etc.

[0460]

[0508] In some embodiments, the vehicle's GPS system provides timing and position signals that can then be used by the processing unit 210 to obtain (e.g., via an API) road condition information, information about (identifying) special areas (e.g., construction zones, school zones, etc.), information about the environment (e.g., urban, suburban, city, etc.), information about (identifying) traffic conditions, time of day, etc.

[0461]

[0509] Also, in some embodiments, the processing unit 210 may include or communicate with an object detection module. In such a case, whenever new classes are introduced to the object detection module (e.g., animals, utility poles, trees), they can be input directly into the model. The model learns the relevance of the new classes during training. In this way, new risk signals can be introduced into the model without manually designing logic.

[0462]

[0510] In some embodiments, the inputs provided to the model may include at least two of the example inputs described above. For example, in some embodiments, the inputs provided to the model may include first information regarding a state of the driver and second information regarding a state external to the vehicle.

[0463]

[0511] In some embodiments, the processing unit 210 may be configured to package two or more of the above example inputs into a data structure for feeding into the model. The data structure may be configured with a two-dimensional data matrix. In some embodiments, the model is a Temporal-CNN, and the data structure of the model's input may be configured by encoding the input (risk signal) and time as columns and rows of a single channel input image, respectively, or vice versa. The model's input can take the form of a time series matrix of all signals considered individually or jointly to predict momentary situational risk. An example of such a matrix is ​​described below with reference to FIG. 28.

[0464]

[0512] In some embodiments, the model in the processing unit 210 of the apparatus 200 is configured to receive at least a first time series input and a second time series input in parallel and / or process the first time series and the second time series in parallel. The first time series may be, for example, 6 seconds (or any other time period) of TTC data and the second time series may be, for example, 6 seconds (or any other time period) of velocity data. The first time series and the second time series may be combined to form a single channel input image, which may be stored in a non-transitory medium. Alternatively, the first time series and the second time series may be stored separately. In such cases, the input image may still be considered to be formed by logically associating the first time series and the second time series and / or formed when the first time series and the second time series are input to the model in parallel.

[0465]

[0513] As mentioned above, in some embodiments, the processing unit 210 of the apparatus 200 may be configured to determine various probabilities of predicted events (e.g., probability of collision, probability of near collision, probability of non-hazardous event, etc.) based on the time series input. FIG. 28 illustrates an example of multiple inputs processed by a model to generate a prediction at least 2 seconds before the event occurs. As shown, inputs from the past 6 seconds are input to the model of the processing unit 210 to make a model prediction for one or more events that may occur in the future 2 seconds before the event occurs. The inputs from the past 6 seconds include time series data within a 6 second window. In other embodiments, instead of using data from the past 6 seconds, the processing unit 210 may be configured to process less than 6 seconds (e.g., 5, 4, 3, 2, 1 second, etc.) of data to determine the probability of the predicted event, or may be configured to determine the probability of the predicted event from data from 6 seconds or more (e.g., 7, 8, 9, 10, 11, 12, ... 30 seconds, etc.).

[0466]

[0514] In particular, as shown in FIG. 28, the set of inputs may include different parameters and respective values ​​for corresponding time points within the immediately preceding time period (e.g., 6 seconds in this example). The parameters (or risk factors) of the input set include distance, looking down, looking up, looking left, looking right, phone, smoking, holding object, eyes closed, no face, time to collision, and speed. The “distance” parameter indicates the distance between the target vehicle and the leading vehicle. The “looking down” parameter indicates whether or not the driver is looking down or the degree to which the driver is looking down (e.g., a higher value indicates a higher probability that the driver is looking down). The “looking up” parameter indicates whether or not the driver is looking up or the degree to which the driver is looking up (e.g., a higher value indicates a higher probability that the driver is looking up). The “looking left” parameter indicates whether or not the driver is looking left or the degree to which the driver is looking left (e.g., a higher value indicates a higher probability that the driver is looking left). The "looking right" parameter indicates whether or not the driver is looking right, or the degree to which the driver is looking right (e.g., a larger value indicates a higher probability that the driver is looking right). The "phone" parameter indicates whether or not the driver is using a phone (e.g., a larger value indicates a higher probability that the driver is using a phone). The "smoking" parameter indicates whether or not the driver is smoking (e.g., a larger value indicates a higher probability that the driver is smoking). The "object holding" parameter indicates whether or not the driver is holding an object (e.g., a larger value indicates a higher probability that the driver is holding an object). The "eyes closed" parameter indicates whether or not the driver's eyes are closed, or the degree to which the driver's eyes are closed (e.g., a larger value indicates a higher probability that the driver's eyes are closed). The "no face" parameter indicates whether or not the driver's face is detected, or the degree to which the driver's face is detected (e.g., a larger value indicates a higher probability that the driver's face is not detected). The "time to collision" (TTC) parameter indicates a predicted time to a predicted collision (e.g., a larger value indicates a shorter TTC). The "speed" parameter indicates the speed of the target vehicle (eg, a larger value indicates a higher vehicle speed).

[0467]

[0515] It should be noted that the set of inputs (input data) is not limited to the example parameters described above, and the set of inputs may have other parameters that may be related to risk. Thus, in other embodiments, the set of inputs may have additional parameters. In further embodiments, the set of inputs may not have all of the parameters described, or may have fewer parameters (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 parameters). Additionally, in other embodiments, the set of inputs may have different parameters than those described.

[0468]

[0516] As shown in FIG. 28, the model is configured to generate a model prediction based on a set of inputs (e.g., risk factors). In some embodiments, the model prediction may include (1) a probability of a crash within T seconds, (2) a probability of a near crash within T seconds, and (3) a probability of a non-hazardous event within T seconds. Because the model is trained to distinguish between these three events, it can determine whether the set of inputs indicates a crash event, a near crash event, or a non-hazardous event. The model can also determine the probability of each predicted event. In some embodiments, the model can determine the probability of each predicted event by applying a softmax function to the output vector / logits such that the final output for each type of predicted event is a probability value between 0 and 1, and the sum of all outputs (probabilities of different predicted events) is 1. In some embodiments, the model is trained with a sufficient number of similar examples of crash events to enable it to predict a high probability of a crash event for a given set of inputs that lead to a crash event. The model may also be trained with a sufficient number of similar examples of near crash events to enable it to predict a high probability of a near crash event for a given set of inputs that lead to a near crash event. The model may also be trained with a sufficient number of similar examples of non-events to enable it to predict non-events with a high probability for a set of inputs that lead to a non-event. In other embodiments, the model predictions may include additional predictions or may include only one or two of the three probabilities. For example, in other embodiments, instead of predicting probabilities of a crash event, a near crash event, and a non-hazardous event, the model may be configured to predict two probabilities: a "risk event" and a "non-hazardous event."

[0469]

[0517] After the probabilities of various predicted events (e.g., crash events, near-crash events, non-hazardous events, etc.) are determined, processing unit 210 can then determine a metric (e.g., a risk score) based on these probabilities. In some embodiments, the metric determined by processing unit 210 may indicate whether the risk factors collectively result in a hazardous or non-hazardous situation. The metric, in some embodiments, may be a risk score that indicates the degree of risk. For example, a higher value of the risk score may indicate a higher risk and a lower value may indicate a lower risk.

[0470]

[0518] In some embodiments, the metric determined by processing unit 210 based on the probabilities of various events may be a weighted sum of the scores (probabilities) of each of the different events, while in other embodiments the metric may be calculated as a sum of the weighted scores (probabilities).

[0471]

[0519] 29 shows an example of a risk score calculation based on model predictions. In the above example, the model of processing unit 210 predicts a 6% probability of a crash within T seconds, a 22.8% probability of a near crash occurring within T seconds, and a 71.2% probability of a non-hazardous event occurring within T seconds. If a weight of 1.0 is applied to the crash predictions, a weight of 0.5 is applied to the near crash predictions, and a weight of 0.0 is applied to the non-hazardous event predictions, processing unit 210 may calculate a risk score as the sum of the weighted predictions, as follows: 1.0*6%+0.5*22.8%+0*71.2%=17.4.

[0472]

[0520] 29, the model of processing unit 210 predicts that the probability of a collision within T seconds is 73.4%, the probability of a near collision at T seconds is 21%, and the probability of a non-hazardous event at T seconds is 5.6%. If a weight of 1.0 is applied to the collision prediction, a weight of 0.5 is applied to the near collision prediction, and a weight of 0.0 is applied to the non-hazardous event prediction, processing unit 210 may calculate a risk score as the sum of the weighted predictions as follows: 1.0*73.4%+0.5*21%+0*5.6%=83.9.

[0473]

[0521] In some embodiments, the processing unit 210 may be configured to compare the risk score to one or more thresholds. If the threshold is met (e.g., exceeded), the processing unit 210 may then generate a control signal to generate an alert and / or operate a vehicle control (e.g., automatically brake, turn on exterior lights, honk the horn, etc.). Following the above two examples of FIG. 29, the risk score is 17.4 in the upper example and 83.9 in the lower example. If the threshold for applying an alert and / or vehicle control is set to 70, the processing unit 210 determines that the threshold is not met in the upper example (because the risk score of 17.4 is less than the threshold of 70). Thus, in the upper example, the processing unit 210 does not generate a control signal to generate an alert or operate a vehicle control. On the other hand, the processing unit 210 determines that the threshold is met in the lower example (because the risk score of 83.9 is greater than the threshold of 70). In such cases, the processing unit 210 generates control signals to generate an alert and / or operate vehicle controls (e.g., automatically brake, slow down, honk the horn, activate exterior lights, provide haptic feedback, etc., or any combination thereof).

[0474]

[0522] Alternatively or additionally, if the risk score is equal to or greater than a threshold, the processing unit 210 may generate a control signal to notify a fleet manager or fleet management system that the driver has engaged in inappropriate driving.

[0475]

[0523] In some embodiments, if the risk score is below the threshold or another threshold, the processing unit 210 may not generate a control signal, and / or may activate a speaker to provide audio praise to the driver, and / or may generate a control signal to notify a fleet manager or fleet management system that the driver performed well.

[0476]

[0524] Also, in some embodiments, the model may learn what is considered a good driving behavior. In such a case, the model may detect a particular event and determine what is considered a good driving behavior for the detected event. The processing unit 210 may also compare the actual driving behavior (e.g., by acquiring vehicle control device signals or analyzing images of the vehicle interior) with the good driving behavior to ascertain the extent to which the driver is following the good driving behavior. The results of the comparison may, in some embodiments, be transmitted to the vehicle management system. In such a case, the vehicle management system may use such information to improve the driver's driving skills and / or to give praise to good drivers. Alternatively or additionally, the processing unit 210 may generate control signals to provide a message of praise to the driver (if the driver is exhibiting a good driving behavior) and a message of guidance to the driver (if the driver is not exhibiting a good driving behavior). As non-limiting examples, good driving behaviors include slowing down in heavy traffic areas, slowing down when pedestrians are present, slowing down when approaching an intersection, etc.

[0477]

[0525] FIG. 30 shows an example of a model output based on multiple time series inputs, where the model output indicates a high “crash” state. In particular, the left column of the figure shows examples of different inputs that are input to the model. Each example of input includes different parameters and their respective values ​​for a corresponding time point within the preceding period (6 seconds in the example), as shown and explained with reference to FIG. 28. The center column of FIG. 30 shows the intermediate representations learned by the model. In particular, a saliency map of the neural network model is displayed, visualizing which parts of the input the model is paying attention to. The model processes the inputs to determine the probability of each of three events: a crash event, a near-crash event, and a non-hazardous event (shown in the right column of the figure). In all the examples shown in FIG. 30, the “crash event” has the highest probability (compared to the probabilities of the other two events).

[0478]

[0526] FIG. 31 shows an example of a model output based on multiple time series inputs, where the model output indicates a high “near-crash” condition. In particular, the left column of the figure shows examples of different inputs that are input to the model. Each example of input includes different parameters and their respective values ​​for a corresponding time point within the preceding period (6 seconds in the example), as shown and explained with reference to FIG. 28. The center column of FIG. 31 shows the intermediate representations learned by the model. In particular, a saliency map of the neural network model is displayed to visualize which parts of the input the model is paying attention to. The model processes the inputs to determine the probability of each of three events: a crash event, a near-crash event, and a non-hazardous event (shown in the right column of the figure). In all the examples shown in FIG. 31, the “near-crash event” has the highest probability (compared to the probabilities of the other two events).

[0479]

[0527] FIG. 32 shows an example of a model output based on multiple time series inputs, where the model output indicates a high "non-event" state. In particular, the left diagram shows an example of the inputs that are input to the model. Each example of the input includes various parameters and their respective values ​​for a corresponding time point within the preceding period (6 seconds in the example), as shown and described with reference to FIG. 28. The center diagram of FIG. 32 shows the intermediate representation learned by the model. In particular, a saliency map of the neural network model is displayed, visualizing which parts of the input the model is paying attention to. The model processes the inputs to determine the probability of each of three events: a crash event, a near-crash event, and a non-hazardous event (shown in the right column of the diagram). In the example of FIG. 32, the "non-hazardous event" has the highest probability (compared to the probabilities of the other two events).

[0480]

[0528] As shown in the examples of Figures 30-32, each set of inputs may form a certain pattern. In some cases, the model may be a neural network that can be trained to make decisions based on such patterns. For example, the neural network model may be trained using a large number of input sets labeled as "crash" events (such as the example shown in Figure 30). The neural network model may also be trained using a large number of input sets labeled as "near-crash" events (such as the example shown in Figure 31). The neural network may also be trained using a large number of input sets labeled as "non-risk" events (such as the example shown in Figure 32). Through training, the model learns to pay different attention to different sets of inputs. For example, referring to the center diagram of Figures 30-32, it can be seen that the model pays different attention to different event types. In some embodiments, the model may be trained using a supervised approach. In such an approach, the input signal over time is encoded as a single channel image and input into a CNN model. The model is trained to predict the probability of future events occurring in a time interval, such as non-risk events, near-crash events, and crash events. To train the model, these prediction targets may be created by human labelers in a "supervised" training approach. In some embodiments, the model may be trained to make predictions of events occurring within T seconds (e.g., 1.5 seconds, 2 seconds, 3 seconds, 4 seconds, etc.). Note that training of the model may take any form and is not limited to supervised learning, where the model learns directly from the labels provided. For example, in other embodiments, the model may be trained using reinforcement learning, where the agent learns from its environment through trial and error.

[0481]

[0529] As mentioned above, the models described herein are advantageous because they can reduce both false positive cases (e.g., alerts generated for non-hazardous conditions) and false negative cases (e.g., no alerts generated when a hazardous condition exists). Figures 33-37 show examples of these scenarios.

[0482]

[0530] 33A-33K show a series of external images and corresponding internal images, particularly illustrating the problem that can be addressed by the technique of FIG. 25. In this example in particular, it is illustrated that issuing an overly sensitive warning without considering the surrounding situation hinders the purpose of assisting the driver. FIG. 33A-33C are snapshots (frames 1, 5, 10 from a 25 fps (frames per second) video) corresponding to the scenario. In the three frames shown, before the "object holding" event begins, the driver (1) keeps his eyes on the road, (2) is not speeding, and (3) keeps a safe distance from the preceding vehicle. During the "object holding" event, the driver keeps a safe distance from the preceding vehicle, is not speeding, and is looking at the road as is evident from the snapshots (see FIG. 33D-33G). In this example, the distraction module detected the "object holding" pose and generated a warning because it was assumed that the "object holding" pose is a distraction that could cause a dangerous situation. However, considering the context, the "object holding" event alone is not particularly dangerous. Thus, the generation of the warning here is a false positive. As seen in the next frames (see Figs. 33H-33K), the driver seems annoyed by the warning and pulls down the visor blocking the camera. A warning in such a non-critical scenario reduces the relevance of the warning and reduces the driver's trust in the warning. As shown in this example, the distraction signal was triggered because the detection score for "holding an object" exceeded a predefined threshold. However, this is by no means a dangerous scenario since the driver was looking at the road, not speeding, and kept a safe distance from the vehicle ahead. The technique of Fig. 25 is advantageous in that by considering the combination of detected inputs, the model recognizes that the "holding an object" event alone is not dangerous when taken in context with other inputs, and the processing unit 210 appropriately withholds generation of a control signal to issue a warning.

[0483]

[0531] 34A-34E show a series of external images and corresponding internal images, specifically illustrating the problem that can be addressed by the technique of FIG. 25. In this scenario, no warning was triggered before the near-crash. This is because the driver was repeatedly distracted, but the individual distraction segments did not exceed the distraction duration threshold. Thus, a false negative problem occurs in this example. FIGS. 34A-34B show several relevant snapshots (frames 158, 166 from the 25 fps video) corresponding to the scenario. In these frames, we can see that the driver is repeatedly distracted (looking left, chatting) for a relatively short period of time. Eventually, the ego vehicle gets too close to the leading vehicle, resulting in a near-crash scenario (frames 214, 212, 225 from the 25 fps video) (see FIGS. 34C-34E). The combination of repeated distractions and close proximity to the leading vehicle is a dangerous scenario, regardless of the length of the driver's distraction. On the other hand, the new model in Figure 25 was able to detect this "near crash" event and provided a high risk score of 65.7 for this predicted near crash event. The new model was able to do this because it had access to both the distraction stream (first time series of inputs) and the TTH stream (second time series of inputs). This is advantageous because even if the risk factors do not individually exceed the threshold, when combined they actually result in a very dangerous situation that can be detected by the model in Figure 25.

[0484]

[0532] Figures 35A-35D show a series of external images and corresponding internal images, specifically illustrating the problem that can be addressed by the technique of Figure 25. In a similar scenario to Example 2, the driver is momentarily distracted (looking left) while approaching a preceding vehicle stopped at an intersection, as seen in the snapshots (frames 90, 110, 125 from 25 fps video) (see Figures 35A-35C). Although the distraction period is relatively short, it is a rather dangerous scenario that ultimately led to a collision. The bottom frame shows the moment of collision (frame 134 from 25 fps video) (see Figure 35D).

[0485]

[0533] 36A-36C show a series of external images and corresponding internal images, specifically illustrating the problem that can be addressed by the technique of FIG. 25. In this example, as seen in the snapshots (frames 29, 65 from the 25 fps video), the driver was occasionally distracted (using a mobile phone and looking down) and the vehicle ahead was backing up (see FIGS. 36A-36B). However, no warning was triggered in the existing system. Therefore, this case is also an example of a false negative. Using the model of FIG. 25, the processing unit 210 predicted this event to be a "crash" event and assigned a high risk score of 83.9 based on processing the raw distraction stream (first time series of inputs) and the TTH / TTC stream (second time series of inputs). The frame shown in FIG. 36C shows the moment of the crash (frame 126 from the 25 fps video).

[0486]

[0534] Figures 37A-37C show a series of external images and corresponding internal images, specifically illustrating the problem that can be addressed by the technique of Figure 25. Although a warning was generated in this scenario, the driver was distracted (looking down), but the warning was irrelevant because, as seen in the snapshots (frames 49, 143, 213 from the 25 fps footage), there was no leading vehicle ahead for a long time and he was driving on an empty road without speeding up (see Figures 37A-37C). On the other hand, the model of Figure 25 predicted this event as a non-hazardous event (calm / unimportant) and calculated a low risk score of 17.4 using cues from the distraction, speeding, and TTH / TTC streams. Thus, the model was able to prevent the problem of false positives.

[0487]

[0535] It should be noted that the model of FIG. 25 may be implemented in device 200 in a variety of ways. For example, in some embodiments, a trigger-based approach may be used. In such cases, the model runs only when one or more conditions are met (e.g., when a distraction threshold is reached). Once triggered, the model runs for the entire duration of the event. As another example, other embodiments may use a continuous approach. In such an approach, the model runs continuously on the most recent X seconds of data based on a sliding window.

[0488]

[0536] In the above examples, the model is described as providing model predictions including a predicted "crash" event, a predicted "near-crash" event, a predicted "non-risk" event, and their respective probabilities (first output). However, the first output is not limited to these examples and may be any driving condition associated with risk. For example, in other embodiments, the first output may also include a metric indicating the severity of the crash (e.g., high speed vs. low-risk), and / or characteristics of the risk event, such as the presence or absence of hard braking, hard steering, "good" driving behavior, etc.

[0489]

[0537] Also, in the above example, the probabilities of the various predicted events (first output) are utilized by the processing unit 210 to calculate a risk score (second output) based on a sum of the weighted probabilities. In other embodiments, other techniques may be used to calculate the risk score. For example, any continuous function that maps the direct model outputs (e.g., probabilities of the various events) to a single score may be used.

[0490]

[0538] In a further embodiment, the processing unit 210 may determine, as a third output, the change in the risk score (second output) over time, which may be advantageous in indicating whether and to what extent a situation is becoming more dangerous or safer at a particular moment in time.

[0491]

[0539] In further embodiments, processing unit 210 may determine a fourth output indicative of, related to, or based on an optimal action to be taken at any given time, particularly in a dangerous scenario. For example, in some embodiments, processing unit 210 may determine a course of action that a "good driver" would take based on a set of inputs indicative of a particular dangerous situation. In such cases, the model not only processes external risk factors, but also evaluates how the current driver's behavior compares to ideal driving behavior to better assess new risks. These good actions may include slowing down, shifting gaze, changing lanes, etc.

[0492]

[0540] Also, in some embodiments, the processing unit 210 (e.g., models therein) may be configured to process the first image data from the first camera 202 and the second image data from the second camera 204 to generate one or more outputs (e.g., probabilities of different event categories), where the first image data is generated during a first time window and the second image data is generated during a second time window that is at least partially coextensive with the first time window. For example, the first image data may be generated from t=2 seconds to t=8 seconds and the second image data may be generated from t=3 seconds to t=10 seconds. In other embodiments, the first and second time windows may be completely coextensive. For example, the first image data may be generated from t=1 second to t=5 seconds and the second image data may be generated from t=1 second to t=5 seconds. In further embodiments, the first and second time windows may be non-coextensive. For example, a series of closely spaced distractions captured in second image data during time window A may give rise to a state of increased external risk captured in first image data during a later time window B.

[0493]

[0541] Further, in some embodiments, the processing unit 210 (e.g., models therein) may be configured to predict future distractions based on the driver's past behavior. For example, the processing unit 210 may predict that a future distraction (e.g., within the next few minutes) is likely to occur even if a momentary distraction has ended, based on past behavior detected by the processing unit 210 (e.g., distraction events that occurred frequently in a past time window). Distraction events may be considered to occur frequently if there are multiple events occurring close together in time (e.g., events occurring within a certain time period, such as within 2 minutes, within 1 minute, within 30 seconds, within 15 seconds, within 10 seconds, within 5 seconds, within 4 seconds, within 3 seconds, within 2 seconds, within 1 second, etc.).

[0494]

[0542] Further, in some embodiments, processing unit 210 (e.g., models therein) may be configured to identify clustering of more frequent distractions at particular geographic locations and / or times as a more persistent indicator of external risk. For example, processing unit 210 may determine that a driver has been consistently distracted at ...

Claims

1. 1. An apparatus comprising: a first sensor configured to provide a first input related to an environment external to a vehicle; a second sensor configured to provide a second input related to operation of the vehicle; and a processing unit configured to receive a first input from the first sensor and a second input from the second sensor, the processing unit including a first stage processing system and a second stage processing system, the first stage processing system configured to receive a first input from the first sensor, receive a second input from the second sensor, process the first input to obtain a first time series of information, and process the second input to obtain a second time series of information, the second stage processing system comprising a neural network model configured to receive the first time series of information and the second time series of information in parallel, the neural network being configured to process the first time series and the second time series to determine a probability of a predicted event related to operation of the vehicle.

2. The apparatus of claim 1 , wherein the first input has low dimensionality or complexity compared to the first time series of information.

3. The apparatus of claim 1 , wherein the first time series of information is indicative of a first risk factor and the second time series of information is indicative of a second risk factor.

4. The apparatus of claim 1 , wherein the processing unit is configured to package the first and second time series into a data structure for feeding to a neural network model.

5. The apparatus of claim 4 , wherein the data structure comprises a two-dimensional data matrix.

6. The apparatus of claim 1 , wherein the first time series is indicative of conditions external to the vehicle at different respective points in time, and the second time series is indicative of driver conditions and / or vehicle conditions at different respective points in time.

7. 2. The apparatus of claim 1, wherein the probability of the predictive event is a first probability of a first predictive event, the processing unit is configured to determine a second probability of a second predictive event, the first predictive event and the second predictive event are related to operation of a vehicle, and the processing unit is configured to calculate a risk score based on the first probability of the first predictive event and based on the second probability of the second predictive event.

8. 8. The apparatus of claim 7, wherein the first predictive event is a collision event and the second predictive event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of a collision event and based on a second probability of a non-hazardous event.

9. 8. The apparatus of claim 7, wherein the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and adding the first weighted probability and the second weighted probability.

10. 8. The apparatus of claim 7, wherein the processing unit is configured to determine a third probability of a third predictive event, and wherein the processing unit is configured to calculate the risk score based on the first probability of the first predictive event, based on a second probability of the second predictive event, and based on a third probability of the third predictive event.

11. 11. The apparatus of claim 10, wherein the first predicted event is a crash event, the second predicted event is a near crash event, and the third predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of the crash event, based on a second probability of the near crash event, and based on a third probability of the non-hazardous event.

12. 11. The apparatus of claim 10, wherein the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to determine a first weighted probability, applying a second weighting to the second probability to determine a second weighted probability, and applying a third weighting to the third probability to determine a third weighted probability, and adding the first weighted probability, the second weighted probability, and the third weighted probability.

13. 2. The apparatus of claim 1 , wherein the first input and the second input are obtained within the past T seconds, and the processing unit is configured to process the first input and the second input obtained within the past T seconds to determine a probability of the predicted event, where T is at least 3 seconds.

14. The apparatus of claim 1 , wherein the predicted event is for a future time at least one second from a current time.

15. 2. The apparatus of claim 1, wherein the processing unit is configured to calculate a first risk score for a first time point based on a probability, the processing unit is also configured to calculate a second risk score for a second time point, and to identify a difference between the first risk score and the second risk score, the difference indicating whether a dangerous situation is escalating or subsiding.

16. The apparatus of claim 1 , wherein the processing unit is configured to determine the risk score based on a probability of a predicted event.

17. The apparatus of claim 16 , wherein the processing unit is configured to generate a control signal based on the risk score.

18. The apparatus of claim 17 , wherein the processing unit is configured to generate a control signal if the risk score meets a criterion.

19. The apparatus of claim 18 , wherein the processing unit is configured to generate a control signal to operate a device if the risk score meets a criterion.

20. 20. The apparatus of claim 19, wherein the device comprises a speaker for issuing a warning, a display or light emitting device for providing a visual signal, a haptic feedback device, a collision avoidance system, or a vehicle control device for a vehicle.

21. 2. The apparatus of claim 1, wherein the first time series and the second time series each include any two or more of: distance to a leading vehicle, distance to an intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions, information about special zones, information about the environment, information about traffic conditions, time of day, information about visibility conditions, information about identified objects, object location, object direction of movement, object speed, bounding boxes, vehicle operating parameters, information about a driver's state, information about a driver's history, continuous driving time, proximity to a meal time, information about an accident history, and audio information.

22. The apparatus of claim 1 , wherein the first sensor comprises a camera, a Lidar, a radar, or any combination thereof configured to sense an environment outside the vehicle.

23. The apparatus of claim 1 , wherein the second sensor includes a camera configured to view a driver of a vehicle.

24. The apparatus of claim 1 , wherein the second sensor includes one or more sensing units configured to sense one or more characteristics of a vehicle.

25. 2. The apparatus of claim 1, wherein the first input includes a first image and the second input includes a second image, and the first stage processing system is configured to receive the first image and the second image, process the first image to obtain a first time series of information, and process the second image to obtain a second time series of information.

26. 1. An apparatus comprising: a first sensor configured to provide a first input; a second sensor configured to provide a second input; and a processing unit configured to receive the first input from the first sensor and the second input from the second sensor, the processing unit configured to identify a first probability of a first predictive event and a second probability of a second predictive event, the first predictive event and the second predictive event associated with operation of a vehicle, and the processing unit configured to calculate a risk score based on the first probability of the first predictive event and based on the second probability of the second predictive event.

27. 27. The apparatus of claim 26, wherein the first predictive event is a collision event and the second predictive event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of a collision event and based on a second probability of a non-hazardous event.

28. 27. The apparatus of claim 26, wherein the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to obtain a first weighted probability, applying a second weighting to the second probability to obtain a second weighted probability, and adding the first weighted probability and the second weighted probability.

29. 27. The apparatus of claim 26, wherein the processing unit is configured to determine a third probability of a third predictive event, and wherein the processing unit is configured to calculate the risk score based on the first probability of the first predictive event, based on the second probability of the second predictive event, and based on the third probability of the third predictive event.

30. 30. The apparatus of claim 29, wherein the first predicted event is a crash event, the second predicted event is a near crash event, and the third predicted event is a non-hazardous event, and the processing unit is configured to calculate the risk score based on a first probability of the crash event, based on a second probability of the near crash event, and based on a third probability of the non-hazardous event.

31. 30. The apparatus of claim 29, wherein the processing unit is configured to calculate the risk score by applying a first weighting to the first probability to determine a first weighted probability, applying a second weighting to the second probability to determine a second weighted probability, and applying a third weighting to the third probability to determine a third weighted probability, and adding the first weighted probability, the second weighted probability, and the third weighted probability.

32. 27. The apparatus of claim 26, wherein the first input and the second input include data obtained in the past T seconds, and the processing unit is configured to process the data obtained in the past T seconds to determine a first probability of the first predictive event and a second probability of the second predictive event, where T is at least 3 seconds.

33. 27. The apparatus of claim 26, wherein the first predictive event is for a future time at least 26 seconds from a current time.

34. 27. The apparatus of claim 26, wherein the processing unit is configured to calculate a risk score for a first time point, the processing unit is also configured to calculate a further risk score for a second time point, and to determine a difference between the risk score and the further risk score, the difference indicating whether a dangerous situation is escalating or subsiding.

35. 27. The apparatus of claim 26, wherein the processing unit is configured to generate a control signal based on the risk score.

36. 36. The apparatus of claim 35, wherein the processing unit is configured to generate a control signal if the risk score meets a criterion.

37. 37. The apparatus of claim 36, wherein the processing unit is configured to generate a control signal to operate a device if the risk score meets a criterion.

38. 40. The apparatus of claim 37, wherein the device comprises a speaker for issuing a warning, a display or light emitting device for providing a visual signal, a haptic feedback device, a collision avoidance system, or a vehicle control device for a vehicle.

39. 27. The apparatus of claim 26, wherein the processing unit is configured to determine a first probability of the first predictive event and a second probability of the second predictive event based on the first input and the second input.

40. 27. The apparatus of claim 26, wherein the processing unit comprises a first stage processing system and a second stage processing system, the first stage processing system configured to receive the first input and process the first input to provide a first output, and the second stage processing system configured to receive the first output and process the first output to provide a second output, the first output having lower dimensionality or less complexity compared to the first input, and the second output having lower dimensionality or less complexity compared to the first output.

41. 27. The apparatus of claim 26, wherein the processing unit includes a neural network model.

42. 42. The apparatus of claim 41, wherein the neural network model is configured to receive a first time series of information indicative of a first risk factor and a second time series of information indicative of a second risk factor.

43. 43. The apparatus of claim 42, wherein the neural network model is configured to receive the first and second time series in parallel and / or process the first and second time series in parallel.

44. 43. The apparatus of claim 42, wherein the processing unit is configured to package the first and second time series into a data structure for feeding to a neural network model.

45. 45. The apparatus of claim 44, wherein the data structure comprises a two-dimensional data matrix.

46. 43. The apparatus of claim 42, wherein the first time series is indicative of conditions external to the vehicle at different respective points in time, and the second time series is indicative of driver conditions and / or vehicle conditions at different respective points in time.

47. 43. The apparatus of claim 42, wherein the first time series is indicative of a first characteristic of the vehicle at different respective points in time, and the second time series is indicative of a second characteristic of the vehicle at different respective points in time.

48. 43. The apparatus of claim 42, wherein the first time series and the second time series include any one or more combinations of: distance to a leading vehicle, distance to an intersection stop line, vehicle speed, time to collision, time to intersection violation, estimated braking distance, information about road conditions, information about special zones, information about the environment, information about traffic conditions, time of day, information about visibility conditions, information about identified objects, object location, object direction of movement, object speed, bounding boxes, vehicle operating parameters, information about a driver's state, information about a driver's history, continuous driving time, proximity to a meal time, information about accident history, and audio information.

49. 27. The apparatus of claim 26, wherein the first sensor comprises a camera, a Lidar, a radar, or any combination thereof configured to sense an environment outside the vehicle.

50. 27. The apparatus of claim 26, wherein the second sensor includes a camera configured to view a driver of the vehicle.

51. 27. The apparatus of claim 26, wherein the second sensor includes one or more sensing units configured to sense one or more characteristics of the vehicle.

52. 27. The apparatus of claim 26, wherein the first input comprises a first image and the second input comprises a second image, and the first stage processing system is configured to receive the first image and the second image, process the first image to obtain a first time series of information and process the second image to obtain a second time series of information.

Images