Detection and Handling of Abnormal Driver Behavior Using Driver Assistance

Abstract

This technology relates to identifying and addressing abnormal driver behavior. Various driving maneuvers can be evaluated over different time scales and driving distances (902, 906). The system can detect driving errors and suboptimal maneuvers, which are evaluated by an onboard driver assistance system and compared to a model of expected driver behavior (908). The results of this comparison can be used to alert the driver or to immediately implement corrective driving actions (912, 914). The results of the comparison can also be used for real-time or offline training or sensor calibration purposes (912, 914). The behavior model can be driver-specific or a baseline driver model based on aggregate information from many drivers. These approaches can be employed for drivers of passenger cars, buses, freight trucks, and other vehicles.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit and priority of the filing dates of U.S. Patent Application No. 17 / 065,722, filed on October 8, 2020, and U.S. Provisional Patent Application No. 62 / 915,134, filed on October 15, 2019, the entire disclosures of which are incorporated herein by reference.

Background Art

[0002] Existing in - vehicle advanced driver assistance systems (ADAS) can automate certain driving tasks. For example, ADAS can provide warnings regarding getting too close to a vehicle ahead via lane departure warnings and forward collision warnings. In other examples, ADAS can also adjust driving via features such as automotive lane - keeping, automatic braking, and advanced cruise control. Other systems can also track the driver's eye movements and whether the hands are on the steering wheel, which can be used to assess attention. These approaches can operate according to the momentary state of the vehicle over a short time scale, e.g., the previous few seconds. However, such approaches may not be able to provide a robust assessment of specific repetitive behaviors or use that assessment in route planning or other driving operations.

Summary of the Invention

[0003] This technology relates to identifying and addressing abnormal driver behavior in vehicles that can be operated, at least in part, in an autonomous driving mode or that include state-of-the-art driver assistance systems. This can include passenger vehicles, buses, commercial vehicles, and the like. According to aspects of this technology, a driver's operation can be evaluated over a significant time scale, e.g., tens of seconds, minutes, hours, or more. The evaluation considers different factors such as driver errors and sub-optimal maneuvers that may not reach a "threshold" level of "error." Driver errors can include one or more lane departures, braking delays in response to an approaching obstacle, failure to follow traffic signals or signs, and the like. Sub-optimal maneuvers can include, for example, weaving or otherwise drifting within a lane without triggering a lane departure warning, or sudden lane changes, running a red light, hard braking, or jerky or rattling braking. Any or all of these are evaluated by an in-vehicle driver assistance system and compared to a model of expected driver behavior. The information generated as a result of this comparison can be used in various ways. For example, the in-vehicle system can immediately execute corrective driving actions and / or warn the driver. It can also be used for purposes of off-line training or sensor calibration. These and other aspects of this technology are considered further below.

[0004] According to one aspect, a method is provided. The method includes executing, by one or more processors, state tracking of driving actions performed by a driver of a vehicle, and receiving, by one or more processors, sensor information from a perception system of the vehicle. The method also includes determining, by one or more processors, that one or more abnormal driving actions are being performed based on state tracking of driving actions during one or both of a predetermined time frame or a driving distance, comparing, by one or more processors, the determined abnormal driving actions with a behavior model, and further creating, based on the comparison, a disagreement signal. The disagreement signal indicates a deviation of the driver's performance from the behavior model. The method includes selecting, in response to the disagreement signal, an action from among one or more classes of actions and then performing the selected action.

[0005] In one example, the behavior model is an aggregation model based on information from multiple drivers other than the driver of the vehicle. In another example, the behavior model is specific to the driver based on the driver's past driving history.

[0006] Determining that one or more abnormal driving actions are being performed may include evaluating at least one of some sensors of the perception system, the accuracy of each sensor of the perception system, the sensor field of view, or the sensor arrangement around the vehicle. Determining that one or more abnormal driving actions are being performed may include evaluating at least one of lane departure, weaving within a lane, speed profile, braking profile, or steering profile. And determining that one or more abnormal driving actions are being performed may include comparing the state tracking of driver behavior with at least one of map information and the received sensor information.

[0007] The one or more classes of actions may include at least one of a warning to the driver, an autonomous operation of the vehicle, or a warning to other vehicles.

[0008] This method may further include performing calibration of one or more sensors of the perception system based on state tracking. And performing the selected action may include at least one of autonomously performing a corrective driving action or providing a warning to the driver regarding a determined abnormal driving behavior.

[0009] According to another aspect, a control system for a vehicle is provided. The control system may include a memory configured to store a behavior model and one or more processors operably coupled to the memory. The one or more processors perform state tracking of driving actions performed by a driver of the vehicle, receive sensor information from a perception system of the vehicle, and determine that one or more abnormal driving actions are being performed based on state tracking of driving actions during one or both of a predetermined time frame or a driving distance. The processor is also configured to compare the determined abnormal driving action with the behavior model and create a mismatch signal based on the comparison. The mismatch signal indicates a deviation of the driver's performance from the behavior model. The processor is further configured to select an action from among one or more classes of actions in response to the mismatch signal and perform the selected action. Performing the selected action may include at least one of autonomously performing a corrective driving action by instructing the vehicle's drive system to take a corrective action or providing a warning to the driver regarding the determined abnormal driving action via the vehicle's user interface.

[0010] In one example, the behavior model is an aggregation model based on information from a plurality of drivers other than the driver of the vehicle. In another example, the behavior model is specific to the driver based on the driver's past driving history of the vehicle.

[0011] Determining that one or more abnormal driving actions are being performed may include evaluating at least one of several sensors of the perception system, the accuracy of each sensor of the perception system, the sensor field of view, or the sensor arrangement around the vehicle. Alternatively or additionally, determining that one or more abnormal driving actions are being performed may include evaluating at least one of a lane departure, a stagger within a lane, a speed profile, a braking profile, or a steering profile. And determining that one or more abnormal driving actions are being performed may include comparing the state tracking of the driver behavior with at least one of the map information and the received sensor information.

[0012] One or more classes of actions may include at least one of a warning to the driver, an autonomous operation of the vehicle, or a warning to another vehicle.

[0013] The control system may be further configured to perform calibration of one or more sensors of the perception system based on the state tracking.

[0014] The control system may further include the perception system.

[0015] And according to another aspect, the vehicle is configured to evaluate the driver's abnormal driving behavior while operating in a partially autonomous driving mode. This vehicle includes a control system, a perception system, and a driving system as described above and detailed below.

Brief Description of the Drawings

[0016]

Fig. 1A - B

Fig. 1C - D

Fig. 2

Fig. 3A - B

Fig. 4

Fig. 5

Fig. 6

Fig. 7A - B

Fig. 8A - B

Fig. 9

DETAILED DESCRIPTION OF THE INVENTION

[0017] The in-vehicle driver assistance system evaluates information regarding the driver's vehicle operations over time. The evaluation may include comparing specific actions (or inactions) with a stored driver behavior model of the driver. The behavior model may be based on the driver's past actions and driving habits, a reference model based on general driving expectations, or other factors. One or both of driving errors and suboptimal driving may be included in the evaluation. As a result, the system may determine that the vehicle's operation is abnormal, in which case a corrective action or a warning to the driver or others may be executed.

[0018] Exemplary Vehicle System Figure 1A shows a perspective view of an exemplary passenger vehicle 100, such as a sedan, minivan, sports utility vehicle (SUV), or other vehicle. Figure 1B shows a top view of the passenger vehicle 100. The passenger vehicle 100 can include various sensors for obtaining information about the vehicle's external environment, and these sensors can be used by an in-vehicle ADAS system or other systems that can operate the vehicle in a semi-autonomous driving mode. For example, the rooftop unit 102 can include lidar sensors, as well as various cameras, radar units, infrared and / or acoustic sensors. The sensor unit 104 positioned at the front end of the vehicle 100, as well as the driver-side and passenger-side units 106a, 106b of the vehicle, can each incorporate lidar, radar, cameras, and / or other sensors. For example, the unit 106a may be positioned in front of the driver's side door along the quarter panel of the vehicle. As shown in Figure 1B, the sensor unit 108 may be positioned along the rear of the vehicle 100, such as on or adjacent to the bumper. In some examples, the passenger vehicle 100 can also include various sensors for obtaining information about the vehicle's interior space (not shown).

[0019] Figures 1C - 1D show an example of a freight vehicle 150, such as a tractor-trailer truck. The truck can include, for example, single, double, or triple trailers, or can be another medium or large truck, such as a commercial weight class 4 - 8. As shown, the truck includes a tractor unit 152 and a single cargo unit or trailer 154. The trailer 154 can be fully enclosed, open like a flatbed, or partially open, depending on the type of cargo being transported. In this example, the tractor unit 152 includes an engine and a steering system (not shown), and a cab 156 for the driver and any passengers.

[0020] Trailer 154 includes a hitch point 158 known as a kingpin. The kingpin 158 is typically formed as a solid steel shaft and is configured to be directionally attachable to the tractor unit 152. In particular, the kingpin 158 is attached to a trailer coupling 160 known as a fifth wheel mounted behind the cab. In the case of a double or triple tractor trailer, the second and / or third trailer may have a simple hitch connection to the leading trailer. Alternatively, each trailer may have its own kingpin. In this case, at least the first and second trailers can include a fifth wheel type structure arranged to connect to the next trailer.

[0021] As shown, the tractor may have one or more sensor units 162, 164 disposed along the tractor. For example, one or more sensor units 162 can be disposed on the roof or top portion of the cab 156, and one or more side sensor units 164 can be disposed on the left and / or right side of the cab 156. The sensor units may be positioned along other areas of the cab 156, such as along the front bumper or bonnet area, behind the cab, adjacent to the fifth wheel, under the chassis, etc. The trailer 154 may also have one or more sensor units 166 disposed, for example, along the trailer, such as along the side panels, front, rear, roof, and / or undercarriage of the trailer 154.

[0022] Similar to the sensor unit of the passenger vehicle in FIGS. 1A - 1B, each sensor unit of the freight vehicle may include one or more sensors such as, for example, lidar, radar, cameras (e.g., optical or infrared), acoustic sensors (e.g., microphones or sonar - type sensors), inertial sensors (e.g., accelerometers, gyroscopes, etc.), or other sensors (e.g., positioning sensors such as GPS sensors). Certain aspects of the present disclosure are particularly useful in relation to specific types of vehicles, but the vehicle may be any type of vehicle including, but not limited to, passenger cars, trucks, motorcycles, buses, recreational vehicles, etc.

[0023] As described above, the present technology is applicable to state - of - the - art driver assistance systems (ADAS) and other systems that can provide at least partial autonomy in an autonomous driving mode. In vehicles operating in a partial autonomous driving mode, different degrees of autonomy can occur. The National Highway Traffic Safety Administration and the Society of Automotive Engineers have specified various levels to indicate how much or how little a vehicle controls the driving. For example, level 0 is not automated, and the driver makes all decisions related to driving. Level 1, the lowest semi - autonomous mode, includes some driving assistance such as cruise control. Level 2 has partial automation of certain driving operations, and level 3 involves conditional automation where the person in the driver's seat can take control if necessary. In contrast, level 4 is a high level of automation where the vehicle can drive without assistance under selected conditions. The architectures, components, systems, and methods described herein can function in any of these modes. These systems are generally referred to herein as driver assistance systems.

[0024] Figure 2 shows a block diagram 200 having various components and systems of an exemplary vehicle, such as a passenger vehicle 100, to enable driver assistance. As shown, the block diagram 200 includes a control system having one or more computing devices 202, such as a computing device, including one or more processors 204, a memory 206, and other components typically present in a general-purpose computing device. The memory 206 stores information accessible by one or more processors 204, the information including instructions 208 and data 210 that can be executed by the processors 204 or otherwise used. The computing system can control one or more operations of the vehicle when providing driver assistance or when operating in a partially autonomous driving mode.

[0025] The memory 206 stores information accessible by the processor 204, including instructions 208 and data 210 that can be executed by the processor 204 or otherwise used. By way of example, the data 210 can include one or more behavior models related to driving behavior. The memory 206 can be of any type capable of storing information accessible by a processor, including a computer-readable medium. The memory is a non-transitory medium such as a hard drive, memory card, optical disk, solid state, etc. The system can include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

[0026] Command 208 may be any set of instructions that are executed directly (such as machine code) or indirectly (such as a script) by a processor. For example, the instructions may be stored as computing device code on a computing device-readable medium. In that regard, the terms "instructions", "modules", and "programs" may be used interchangeably herein. The instructions may be in object code form for direct processing by the processor, or may include a collection of scripts or independent source code modules that are interpreted on demand or pre-compiled and stored in any other computing device language. Data 210 may be retrieved, stored, or modified by one or more processors 204 according to instructions 208. In one example, part or all of memory 206 may be an event data recorder or other secure data storage system configured to store vehicle diagnostic and / or sensed sensor data, and may be mounted on the vehicle or located remotely, depending on the implementation.

[0027] Processor 204 may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. FIG. 2 functionally shows that the processor, memory, and other elements of computing device 202 are within the same block, but such a device may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, memory 206 may be a hard drive or other storage medium located within a housing different from that of processor 204. Thus, it will be understood that references to a processor or computing device include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0028] In one example, computing device 202 may form a driving computing system incorporated into vehicle 100 such that the vehicle can perform various driver assistance operations. The driving computing system may be capable of communicating with various components of the vehicle. For example, computing device 202 may communicate with various systems of the vehicle, including a deceleration system 212 (for controlling braking of the vehicle), an acceleration system 214 (for controlling acceleration of the vehicle), a steering system 216 (for controlling the orientation of the wheels and the direction of the vehicle), an indication system 218 (for controlling turn indicators), a navigation system 220 (for navigating the vehicle to a location or around an object), and a positioning system 222 (for determining the position of the vehicle, including, for example, the attitude of the vehicle).

[0029] Computing device 202 may also be operably coupled to a perception system 224 (for detecting objects in the vehicle's environment), a power system 226 (e.g., a battery and / or a gasoline or diesel powered engine), and a transmission system 230 to control the movement, speed, etc. of the vehicle according to instructions 208 in memory 206 in a partial autonomous driving mode. Wheels / tires 228 are coupled to transmission system 230, and computing device 202 may be capable of receiving information regarding tire air pressure, balance, and other factors that may affect driving.

[0030] Computing device 202 may control the direction and speed of the vehicle by controlling various components. As an example, computing device 202 may use map information and data from navigation system 220 to navigate the vehicle to a destination. Computing device 202 may use positioning system 222 to determine the location of the vehicle and, when it is necessary to safely arrive at that location, use perception system 224 to detect objects and respond to the objects. To do so, computing device 202 may accelerate the vehicle (e.g., by increasing the fuel or other energy provided to the engine by acceleration system 214), decelerate the vehicle (e.g., by reducing the fuel supplied to the engine, switching gears, and / or applying brakes by deceleration system 212), change the direction of the vehicle (e.g., by changing the direction of the front wheels or other wheels of vehicle 100 by steering system 216), and signal such a change (e.g., by lighting the direction indicator of signaling system 218). Thus, acceleration system 214 and deceleration system 212 may be part of a power transmission device or other type of transmission system 230 that includes various components between the vehicle's engine and the vehicle's wheels. Also in this case, by controlling these systems, computing device 202 may also control transmission system 230 of the vehicle to maneuver the vehicle.

[0031] Navigation system 220 can be used by computing device 202 to determine and follow a route to a location. In this regard, navigation system 220 and / or memory 206 can store map information, such as very detailed maps that computing device 202 can use to navigate or control a vehicle. As an example, these maps can identify roadways, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects as well as the shape and elevation of the information. Lane lines can include features such as solid or dashed, double or single lane boundary lines, solid or dashed lane boundary lines, reflectors, etc. A given lane can be associated with left and / or right lane boundary lines or other lane lines that define the boundaries of the lane. For this reason, most lanes can be bounded by the left end of one lane line and the right end of another lane line.

[0032] Perception system 224, which can be part of or operate in conjunction with a latest driver assistance system, includes one or more vehicle-mounted sensors 232. The sensors can be used to detect objects outside the vehicle. The detected objects can be other vehicles, obstacles on the road, traffic signals, signs, trees, etc. Sensors 232 can also detect specific aspects of weather conditions, such as snow, rain, water spray, or puddles of water, ice, or other materials on the road.

[0033] As a mere example, the perception system 224 can include one or more lidar sensors, radar units, cameras (e.g., optical imaging devices with or without a neutral density (ND) filter), positioning sensors (e.g., gyroscopes, accelerometers, and / or other inertial components), infrared sensors, acoustic sensors (e.g., microphones or sonar transducers), and / or any other detection devices that record data that can be processed by the computing device 202. Such sensors of the perception system 224 can detect objects external to the vehicle and their characteristics, such as location, orientation, size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), direction of travel, and speed of movement relative to the vehicle. The perception system 224 can also include other sensors within the vehicle and can detect objects and conditions within the vehicle, such as in the passenger compartment. For example, such sensors can detect the position of the driver, whether the driver's hands are on the steering wheel, the direction of the line of sight, etc. Yet another sensor 232 of the perception system 224 can measure the rotational speed of the wheels 228, the amount or type of braking by the deceleration system 312, and other factors associated with the equipment of the vehicle itself.

[0034] The raw data from the sensors and the aforementioned characteristics can be processed by the perception system 224 and / or transmitted to the computing device 202 periodically or continuously for further processing when data is generated by the perception system 224. The computing device 202 can use the positioning system 222 to determine the location of the vehicle and can use the perception system 224 to detect and respond to objects in accordance with an assessment of the driver's behavior, as further discussed below. Additionally, the computing device 202 may perform calibration between individual sensors, all sensors within a particular sensor assembly, or sensors between different sensor assemblies or other physical housings.

[0035] As shown in FIGS. 1A-1B, certain sensors of the perception system 224 can be incorporated into one or more sensor units. In one example, these may be incorporated into the side view mirrors of a vehicle. In another example, other sensors may be part of the roof top housing 102. The computing device 202 can communicate with sensor units positioned on the vehicle or otherwise distributed along the vehicle. Each unit can have one or more types of sensors such as those described above.

[0036] Returning to FIG. 2, the computing device 202 can include the above-described processor and memory, as well as all of the components typically used in connection with a computing device such as the user interface subsystem 234. The user interface subsystem 234 can include one or more user inputs 236 (e.g., a mouse, keyboard, touch screen, and / or microphone), as well as one or more display devices 238 (e.g., a monitor having a screen or any other electrical device operable to display information). In this regard, an internal electronic display may be positioned inside the vehicle (not shown) and may be used by the computing device 202 to provide information to passengers within the vehicle. Other output devices such as speakers 240 and / or haptic actuators 241 for providing haptic feedback to the driver may also be disposed within the passenger vehicle.

[0037] The passenger vehicle also includes a communication system 242. For example, the communication system 242 can also include one or more wireless configurations to facilitate communication with other computing devices, such as passenger computing devices within the vehicle, computing devices external to the vehicle such as in another nearby vehicle on the road, and / or a remote server system. The wireless network connection can include various configurations and protocols, including Bluetooth (trademark), Bluetooth (trademark) Low Energy (LE), cellular phone connection, as well as short-range communication protocols such as the Internet, World Wide Web, intranet, virtual private network, wide area network, local network, private network using one or more enterprise-specific communication protocols, Ethernet, WiFi and HTTP, and various combinations of the foregoing.

[0038] FIG. 3A shows a block diagram 300 having various components and systems of a vehicle, such as vehicle 150 of FIG. 1C. As an example, the vehicle can be, for example, a truck, agricultural implement, or construction machine configured to operate in one or more partial autonomous modes in combination with, for example, ADAS. As shown in block diagram 300, the vehicle includes a control system of one or more computing devices, such as a computing device 302 including one or more processors 304, a memory 306, and other components similar or equivalent to components 202, 204, and 206 discussed above with respect to FIG. 2. In this example, the control system can constitute an electronic control unit (ECU) of a tractor unit of a freight vehicle. Similar to instruction 208, instruction 308 can be any set of instructions that are executed directly (such as machine code) or indirectly (such as a script) by a processor. Similarly, data 310 can be retrieved, stored, or modified by one or more processors 304 in accordance with instruction 308.

[0039] In one example, computing device 302 may form a driving computing system incorporated into vehicle 150. Similar to the arrangement discussed above with respect to FIG. 2, the driving computing system of block diagram 300 can communicate with various components of the vehicle to perform driving operations. For example, computing device 302 can communicate with various systems of the vehicle, such as a driving system including a deceleration system 312, an acceleration system 314, a steering system 316, a signaling system 318, a navigation system 320, and a positioning system 322, each of which can function as discussed above with respect to FIG. 2.

[0040] Computing device 302 is also operably coupled to a perception system 324, a power system 326, and a transmission system 330. Some or all of the wheels / tires 328 are coupled to the transmission system 330, and computing device 302 may be able to receive information regarding tire pressure, balance, rotation amount, and other factors that may affect driving. Similar to computing device 202, computing device 302 may control the direction and speed of the vehicle by controlling various components. By way of example, computing device 302 may assist in navigating the vehicle to a destination location using map information and data from the navigation system 320.

[0041] The perception system 324 can be part of or operate in conjunction with a state-of-the-art driver assistance system. Similar to the perception system 224, the perception system 324 also includes one or more sensors or other components as described above for detecting the operation of objects outside the vehicle, objects or states inside the vehicle, and / or specific vehicle equipment such as wheels and the deceleration system 312. For example, as shown in Figure 3A, the perception system 324 includes one or more sensor assemblies 332. Each sensor assembly 232 includes one or more sensors. In one example, the sensor assembly 332 may be arranged as a sensor tower incorporated into a side view mirror of a truck, agricultural implement, construction machine, etc. The sensor assembly 332 can also be positioned at different locations on the tractor unit 152 or the trailer 154 as described above with respect to Figures 1C - 1D. The computing device 302 can communicate with sensor assemblies positioned on both the tractor unit 152 and the trailer 154. Each assembly can have one or more types of sensors such as those described above.

[0042] Also shown in Figure 3A is a coupling system 334 for the connection between the tractor unit and the trailer. The coupling system 334 can include one or more power and / or pneumatic connections (not shown), as well as the fifth wheel 336 of the tractor unit for connecting to the kingpin of the trailer. A communication system 338 corresponding to the communication system 242 is also shown as part of the vehicle system 300. Similarly, a user interface 339 equivalent to the user interface 234 can also be included for interaction with the driver and any passengers of the vehicle.

[0043] FIG. 3B shows an exemplary block diagram 340 of a trailer system such as trailer 154 of FIGS. 1C-1D. As shown, the system includes an ECU 342 of one or more computing devices, such as a computing device, including one or more processors 344, a memory 346, and other components typically present in a general-purpose computing device. The memory 346 stores information accessible by one or more processors 344, the information including instructions 348 and data 350 that can be executed by the processor 344 or otherwise used. The descriptions of the processors, memory, instructions, and data of FIGS. 2 and 3A apply to these elements of FIG. 3B.

[0044] The ECU 342 is configured to receive information and control signals from the trailer unit. The in-vehicle processor 344 of the ECU 342 can communicate with various systems of the trailer, including a deceleration system 352, a signaling system 254, and a positioning system 356. The ECU 342 can also be operably coupled to a perception system 358 including one or more sensors for detecting objects within the environment of the trailer, and a power system 260 (e.g., a battery power source) for powering local components. Some or all of the trailer wheels / tire 362 are coupled to the deceleration system 352, and the processor 344 can receive information regarding tire pressure, balance, wheel speed, and other factors that may affect operation in autonomous mode, and relay that information to the processing system of the tractor unit. The deceleration system 352, signaling system 354, positioning system 356, perception system 358, power system 360, and wheels / tire 362 can operate in a manner as described above with respect to FIGS. 2 and 3A.

[0045] The trailer also includes a set of landing gear 366, and a coupling system 368. The landing gear provides support structure for the trailer when detached from the tractor unit. The coupling system 368, which is part of the coupling system 334, provides connection between the trailer and the tractor unit. Thus, the coupling system 368 can include a connection section 370 (e.g., for power and / or pneumatic links). The coupling system also includes a kingpin 372 configured to connect to the fifth wheel of the tractor unit.

[0046] Exemplary implementations According to aspects of the present technology, information regarding driver errors and sub - optimal maneuvers is tracked over time, for example, by an in - vehicle driver assistance system or other systems of the vehicle. This information can be stored by the vehicle and / or shared with a remote system, for example, to perform corrective driving actions, warn the driver, provide training, and calibrate sensors such as the vehicle's perception system. Anonymizing the acquired information, the vehicle, and specific details regarding the trip may be included. The information can be encrypted as part of the storage or transmission process.

[0047] In an example 400 shown in FIG. 4, the perception system can track the relative position of the vehicle in the lanes of a roadway, either alone or in combination with other systems such as a positioning system and / or a navigation system. As shown, the vehicle may drift within a lane but cannot cross the lane markings without triggering a lane departure warning. In an instantaneous or short - term (e.g., 2 - 3 seconds) evaluation, a problem may not be identified. However, in a longer evaluation over multiple blocks or at least 1 / 4 mile, such as 30 seconds or more, repeated or periodic drifting may be identified. This may be due to driver inattention, poor wheel alignment, or other problems.

[0048] In another example 500 shown in FIG. 5, the vehicle's forward collision warning system can track not only whether the driver failed to apply the brakes, but also whether the driver's reaction time to braking was significantly longer than that of a normal (benchmark) driver. This example shows three braking patterns: delayed hard braking, repeated braking, and ideal (benchmark) braking. Slow, hard braking can be caused by lack of attention, obstruction of the driver's vision, unexpected lane changes of a vehicle, passing a stop sign, etc. Here, the system may evaluate various factors. For example, did the braking exceed a reference threshold value of the time to start braking? How much braking force (deceleration) was applied? Was the braking smooth (continuous) or jerky (rattling type of braking)? Additional details from the perception system can provide information about whether there was interference or unexpected behavior by other vehicles. Similar to the above situation, based on very small data sets, it may be determined by instantaneous or short-term evaluation that the driver's behavior was abnormal (or not). By long-term or long-distance evaluation, the driver's behavior pattern can be identified.

[0049] In a further example 600 shown in FIG. 6, the in-vehicle system tracks the driving speed. Tracking the vehicle's speed over time (speed profile) can be evaluated in relation to the type of road (e.g., highway vs. surface road), traffic volume, times that can cause glare or insufficient lighting from the sun, detection of traffic lights and signs, and other factors.

[0050] In these and other driving situations, the in-vehicle system uses state tracking and behavior analysis to determine whether the driver is operating within a certain range (or other metric) of acceptable driving behavior. This may indicate that the driver's behavior is abnormal due to short-term states (e.g., drunk, sleepy, inattentive) or other factors (e.g., new drivers with limited experience, first use of a particular vehicle such as a rental car, weather, etc.).

[0051] State tracking and behavior analysis approaches may create (or receive) a model of expected behavior. The number and / or type of abnormal behaviors can be tracked over one or both of a given time frame or distance traveled. For example, a driver may have two lane departures and three instances of weaving within a lane over a selected distance (e.g., a four-mile straightaway) or time (e.g., a five-minute period). Or, alternatively, a driver may be evaluated over an entire trip. The trip can be a daily drive to work, or a longer trip that takes hours or days, such as driving a bus route or delivering long-haul freight. Further, the evaluation may be performed periodically, such as weekly or monthly.

[0052] Determining that these events have occurred is one factor in the analysis. Individual actions and overall driving maneuvers can be ranked and / or compared against a baseline driver behavior model. This model may be a general model based on aggregated anonymized driving information from many drivers. Alternatively, the model may be specific to a particular driver, based on, for example, that particular driver's past driving history.

[0053] In one example, time is a factor in the model, as, for example, nighttime not only increases the likelihood of certain hazards but also can reduce a driver's visibility. Weather conditions are another factor that can affect visibility. And a driver's age and experience are additional factors. Further, time of year (season), geography (e.g., hilly vs. flat), road type (e.g., windy vs. straight, narrow lanes vs. wide lanes), and other information can also be evaluated. Any combination of such factors can be incorporated into a baseline behavior model or a driver-specific behavior model. As an example, a following distance, braking reaction time, speed profile, or amount of drift within a lane can be assigned to a hard limit, a score, a probability distribution, etc., and this can be used to evaluate whether a driver's behavior over a particular time or distance is abnormal.

[0054] In behavior analysis, additional information such as the types of sensors and other devices used in the vehicle may also be considered. For example, the accuracy of one type of sensor (e.g., radar) is on the order of several meters, while the accuracy of another type of sensor (e.g., lidar or optical camera) can be on the order of several centimeters or millimeters. The field of view (FOV) of a particular sensor may be limited. Here, if the sensor has meter-level accuracy, the in-vehicle perception system may not be able to determine, within an acceptable level of certainty, that the driver is wobbling within the lane. Furthermore, there may be blind spots in the sensor, preventing the perception system from determining, within an acceptable level of certainty, the reason for a deviation from multiple lanes or a hard brake. As an example, the more severe (abnormal) the driver's operation or reaction, the more the system may want to know the cause of that behavior. Therefore, the number, accuracy, FOV, and / or placement of sensors on the vehicle may also be considered in the evaluation. Information regarding the vehicle itself, such as the vehicle's size, braking distance, visibility (e.g., pickup truck vs. compact car), may also be applicable. Such information may be particularly useful for commercial vehicles such as trucks and buses.

[0055] Also, for example, it may be appropriate to evaluate driving activities in view of map data and perceptual information to evaluate actions according to the situation. Thus, the system may determine that a part of the stagger in the lane is due to construction that has caused the narrowing of the lane or the loss of the adjacent shoulder. This may include a driver assistance system that recognizes construction based on either the received map or traffic information. Also included may be a driver assistance system that obtains real-time information about the driving environment from the analysis of data acquired by the perception system, for example, from the detection of one or more construction vehicles, construction machinery or signs, debris, or other construction zones, or the presence of a tailgate behind the vehicle. See, for example, the consideration of construction zones in U.S. Patent No. 9,937,924 and the consideration of tailgating in U.S. Patent No. 10,168,706, the entire disclosures of which are incorporated herein by reference. In these types of situations, the system may not consider the stagger to be abnormal. In the case of a lane departure, the driver's action may be due to another vehicle making an unexpected lane change or getting too close to the side or rear of the driver's vehicle. Considering this, the evaluation of the driver's behavior needs to be robust for a single incident. That is, a normal driving profile with isolated cases of stagger or sudden (or gradual) braking is likely to be a response to road conditions, and another driving profile in which such actions occur once every few minutes suggests a potential impairment.

[0056] Nevertheless, for fleet driving, such as that of bus drivers, delivery drivers, or tractor-trailer truck drivers, a longer evaluation period may be desirable. In such situations, the system may aggregate driving data of individual drivers over periods of hours, days, weeks, etc. Further, for example, during a particular period (e.g., morning or evening rush), data of many fleet drivers along the same route may be aggregated. In such cases, a new driver can be evaluated against the criteria of other drivers to determine whether the vehicle is being operated in a manner that exceeds the threshold operating criteria (e.g., speed, repeated braking, lane changes, etc.). Based on this information, training and / or intervention can be provided.

[0057] Furthermore, the advancement of ADAS or autonomous driving systems can affect the amount of context required to evaluate a driver's behavior. As an example, in a level 2 type ADAS system that relies only on a deceleration sensor, there may not be sufficient context for why a human driver suddenly brakes when there is a potential interruption or a pedestrian enters the road from the sidewalk. Here, in the ADAS system, it may be necessary to track deviations over a longer period, such as several minutes (or miles), to improve the signal-to-noise ratio so that it becomes possible to determine whether the behavior is abnormal. The longer the time scale, the fewer local context problems there are. For example, multiple braking incidents that are temporally separated (e.g., minutes or hours apart) may indicate abnormal operation even if the in-vehicle system does not evaluate the reason for a particular incident.

[0058] In contrast, a more robust partial or fully autonomous Level 3, 4, or 5 system can better understand the reasons behind a braking event. For example, in addition to detecting a sudden deceleration, the perception system (e.g., lidar, radar, and / or camera sensors) can detect that a vehicle in an adjacent lane has started to move into the driver's lane, or that a pedestrian is getting off the curb and preparing to enter the crosswalk. As an example, another vehicle that repeatedly changes lanes or wobbles in and out of traffic can potentially change the driver's reaction. Therefore, when the driver is behind or next to another vehicle at a signal, the driver can give the other vehicle a lot of time to pull away when the signal changes. Unless the in-vehicle system understands specific deductive information about the other vehicle, this delay may appear as a slow reaction. In such cases, an in-vehicle behavior prediction model can provide the likelihood that another vehicle or pedestrian will enter the driver's travel lane or take other actions. Here, a more advanced in-vehicle system can distinguish between cases where braking or steering is correlated with other actors or events in the external environment and cases where there appears to be no clear cause. In such cases, the system can determine whether the driver's response is within the prediction range or threshold range of the actions of a single incident.

[0059] One result of state tracking and behavior analysis can be an error signal or other type of discrepancy signal indicating how well (or poorly) the driver executed various actions or responded to specific conditions. The system uses sensor information (e.g., camera and / or lidar information) to determine the position within the lane. Departure of the vehicle from the center of the lane is classified into two maneuvers: lane change and wobbling. In one scenario, a lane change is classified by a complete transition from the center of one lane to another lane, while wobbling is a movement back to the center of the lane in which the movement occurred. Wobbling can be classified according to the severity of the movement based on, for example, the amount of lateral distance from the center of the lane and / or lateral acceleration information regarding the amount of "jerks" during wobbling. In the classification, it may also be considered whether the wobbling exceeds a threshold. Alternatively or additionally, more severe wobbling can be weighted more heavily than less severe wobbling. In other scenarios, other driving actions such as rapid speed changes (e.g., multiple increases in speed of at least 5 - 10 mph) or hard braking (e.g., a decrease in speed of 10 - 15 mph) can be classified. In still other scenarios, non - actions such as not being able to use the direction indicator during a lane change or not being able to initiate braking when sensor - detected road conditions indicate that such braking is required can be classified.

[0060] In these scenarios, the system can maintain a driving log that can be indexed according to time and driving distance, in which wobbling and other driving actions and / or non - actions are labeled as events in the driving log. The driving log can be queried by incident. For example, the number of wobbles within a given time or a given driving distance can be queried. This information can be stored in an in - vehicle or remote database. As an example, the information can be stored in combination with a driver - specific motion model. The information can be queried by in - vehicle systems such as a driving planner or a behavior prediction module.

[0061] When analyzing the driver's behavior over a longer period, for example, over several weeks or months, it may be possible to determine whether there is a decline in certain behaviors by the driver, which may indicate other problems to be addressed. For example, the system can be evaluated for whether the reaction time is slower in certain behaviors (e.g., braking, lane changing, proceeding after a complete stop, etc.). Over the long term, the system can also look for periodic patterns. For example, in the context of commercial driving, the system can measure not only the performance over time but also the driver's performance profile within a shift or, for example, between day and night.

[0062] If the system determines that the driver's behavior is abnormal and outside the threshold or other acceptable range of behavior, one or more different actions may be taken by the vehicle. One class of actions includes warning the driver, for example, by drawing the driver's attention or causing the vehicle to stop. Here, auditory, visual, and / or tactile feedback can be provided to the driver regarding individual actions (e.g., weaving within a lane or hard braking) or repeated actions (e.g., deviating from multiple lanes or not decelerating at a yellow light). Another class of actions includes autonomous operation of the vehicle. Here, the speed of the vehicle may be limited, or the speed of the vehicle may decrease over time and come to a complete stop. Alternatively, depending on the vehicle's autonomous capabilities, it may take over partial or full autonomous control of the driving.

[0063] A further class of actions includes warning other drivers. For example, as shown in the examples of FIGS. 7A - 7B, this can be done by turning on the hazard lights (700 in FIG. 7A) or changing the headlight orientation / pattern (710 in FIG. 7B). Other vehicles can receive the warning via a V2V communication system. As shown in FIGS. 8A - 8B discussed below, remote systems and services, such as driver assistance, fleet management, law enforcement agencies, etc., may also receive the warning. A log of behavior-related information can be stored in the vehicle or a remote system for future analysis, improving the behavior model, and other purposes.

[0064] In addition to the actions of the above class, state tracking and behavior analysis can be used for real-time or offline driver training. For example, the acquired information can be used to train novice drivers or other drivers who may need retraining. This can be done via in-vehicle feedback (e.g., auditory, visual, and / or tactile feedback) or reports provided to the driver offline. Information such as local driving regulations can be used in such training. However, in-vehicle training may depend on the sophistication of the in-vehicle driver assistance system. For example, a Level 2 type of partial automation system may not be able to support such training in some cases, while a Level 3 or Level 4 type of system can train the driver during specific manual driving operations.

[0065] Finally, the acquired driving information can be used to assist in calibrating or monitoring the vehicle's system itself. For example, if the system repeatedly detects wobbling on the left side of the lane but not on the right side, this may indicate that the positioning sensors (e.g., accelerometers and gyroscopes) are not properly calibrated, the wheels are not aligned, or the incidental parameters of the camera are no longer adequately calibrated. Or, if one brake sensor detects hard braking but the other brake sensors and / or accelerometers do not, it may indicate that the brake sensor needs to be replaced. The driver's behavior may also be used as another signal or tiebreaker when determining whether the problem is due to driving behavior, sensor failure, or other issues. For example, if a radar-based braking system always tries to apply the brakes or decelerate the vehicle, but a camera-based system does not (or vice versa), the driver's behavior can be used to determine which sensor may be malfunctioning.

[0066] As described above, the present technology is applicable to various types of vehicles, such as passenger cars, buses, RVs, trucks, or other vehicles that carry cargo. In addition to using wheel slip information for vehicle operation, this information can also be shared with other vehicles, such as those that are part of a fleet.

[0067] An example of this is shown in FIGS. 8A and 8B. In particular, FIGS. 8A and 8B are illustrations and functional diagrams of an exemplary system 800 that includes a plurality of computing devices 802, 804, 806, 808, and a storage system 810 connected via a network 816, respectively. System 800 also includes vehicles 812 and 814, which can be configured the same as or similar to vehicles 100 and 150 of FIGS. 1A - 1B and FIGS. 1C - 1D, respectively. Vehicles 812 and / or vehicle 814 can be part of a fleet of vehicles. For simplicity, only some vehicles and computing devices are shown, but a typical system can include many more.

[0068] As shown in FIG. 8B, each of the computing devices 802, 804, 806, and 808 can include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions can be configured similar to those described above with respect to FIG. 2.

[0069] Various computing devices and vehicles can communicate via one or more networks such as network 816. Network 816, and the intervening nodes, can include various configurations and protocols including short-range communication protocols such as Bluetooth (trademark), Bluetooth LE (trademark), the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using one or more company-specific communication protocols, Ethernet, WiFi and HTTP, as well as various combinations of the foregoing. Such communication can be readily effected by any device capable of transmitting data between other computing devices, such as modems and wireless interfaces.

[0070] In one example, computing device 802 can include a plurality of computing devices, such as one or more server computing devices having a load-balanced server farm, which exchange information with different nodes of a network for the purpose of receiving, processing, and transmitting data with other computing devices. For example, computing device 802 can include one or more server computing devices that can communicate with the computing devices of vehicle 812 and / or vehicle 814, and computing devices 804, 806, and 808 via network 816. For example, vehicle 812 and / or vehicle 814 can be part of a fleet of vehicles that can be dispatched to various locations by a server computing device. In this regard, server computing device 802 can function as a dispatch server computing system that can be used to dispatch vehicles to different locations to pick up and drop off passengers or to pick up and deliver cargo. Further, server computing device 802 can use network 816 to send and present information to a user of one of the other computing devices or a passenger of a vehicle. In this regard, computing devices 804, 806, and 808 can be considered client computing devices.

[0071] As shown in FIG. 8A, each of the client computing devices 804, 806, and 808 can be a personal computing device intended for use by a respective user 818, and can have all of the components that are typically connected to and used with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory for storing data and instructions (e.g., RAM and an internal hard drive), a display (e.g., a monitor with a screen, a touch screen, a projector, a television, or other device such as a smartwatch display operable to display information), and user input devices (e.g., a mouse, keyboard, touch screen, or microphone). The client computing devices can also include a camera for recording video streams, speakers, a network interface device, and all of the components used to connect these elements to each other.

[0072] Each client computing device can include a full-size personal computing device, but alternatively can include a mobile computing device that can wirelessly exchange data with a server via a network such as the Internet. By way of example only, client computing devices 806 and 808 can be a cellular phone, or a wireless-enabled PDA, tablet PC, wearable computing device (e.g., a smartwatch), or a device such as a netbook that can obtain information via the Internet or other network.

[0073] In some examples, the client computing device 804 can be a remote assistance workstation used by an administrator or operator to communicate with the driver of a dispatched vehicle. Although FIGS. 8A-8B show only a single remote assistance workstation 804, a given system can include any number of such workstations. Further, although the operator workstation is illustrated as a desktop computer, the operator workstation can include various types of personal computing devices such as laptop, netbook, and tablet computers.

[0074] The storage system 810 can have any type of computerized storage such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, flash drive, and / or tape drive that can store information accessible by the server computing device 802. Further, the storage system 810 can include a distributed storage system in which data is stored on multiple different storage devices that can be physically located in the same or different geographical locations. The storage system 810 can be connected to the computing device via the network 816 as shown in FIGS. 16A-16B, and / or can be directly connected to or incorporated into any of the computing devices.

[0075] Storage system 810 can store various types of information. For example, storage system 810 can be used by a vehicle such as vehicle 812 or 814 and can also store autonomous vehicle control software that operates such a vehicle in autonomous driving mode. Storage system 810 can also store driver-specific or reference driving models. The model information can be shared with a specific vehicle or fleet as needed. When additional driving information is acquired, it can be updated in real-time, periodically, or offline. Storage system 810 can also include map information, route information, braking and / or acceleration profiles, weather information, etc. of vehicles 812 and 814. This information can be shared with vehicles 812 and 814, for example, by an in-vehicle computer system to assist in behavior analysis during real-time driving by a specific driver.

[0076] Remote assistance workstation 804 can access the stored information and use it to assist in the operation of a single vehicle or a fleet of vehicles. Remote assistance can be used to communicate directly with the driver via the in-vehicle user interface or indirectly via the driver's client computing device. Here, for example, information regarding the current driving operation, training, etc. can be provided to the driver.

[0077] Figure 9 shows an exemplary method of operation 900 according to the above considerations. As shown in block 902, the method performs state tracking of driving actions performed by a driver of a vehicle. At block 904, sensor information is received from the vehicle's perception system. At block 906, the method determines one or more abnormal driving actions based on state tracking of driving actions during one or both of a predetermined time frame or a travel distance. At block 908, the determined abnormal driving actions are compared with a behavior model. At block 910, a disagreement signal is created based on the comparison. The disagreement signal indicates a deviation of the driver's performance from the operating model. At block 912, an action is selected from among one or more classes of actions in response to the disagreement signal. And at block 914, the selected action is executed. The selected action may include taking an autonomously corrected driving action or providing a warning to the driver regarding the abnormal driving action.

[0078] Unless otherwise stated, the foregoing alternatives are not mutually exclusive and may be implemented in various combinations to achieve their respective advantages. These and other variations and combinations of the functions discussed above can be utilized without departing from the subject matter defined by the claims, so the foregoing description of the embodiments should be regarded as illustrative rather than limiting the subject matter defined by the claims. Additionally, the examples described herein, as well as the presentation of clauses expressed as "such as", "including", etc., should not be construed as limiting the subject matter of the claims to specific examples, but rather, the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements. The processes or other operations described above, including the method illustrated in FIG. 9, may be performed in a different order or simultaneously, unless specifically stated otherwise herein.

Claims

1. A method comprising: receiving sensor information from a vehicle's perception system by one or more processors; performing, by the one or more processors, state tracking of driving actions performed by the driver of the vehicle based on the sensor information; determining, by the one or more processors, that one or more abnormal driving actions are being performed based on the state tracking of the driving actions during one or both of a predetermined time frame or a travel distance; selecting, by the one or more processors, an action from among one or more classes of actions if it is determined that one or more abnormal driving actions are being performed; performing the selected action.

2. The method according to claim 1, wherein determining that one or more abnormal driving actions are being performed includes determining that one or more abnormal driving actions are being performed based on the number of predetermined behaviors in the driving actions during one or both of the predetermined time frame or the travel distance.

3. The method according to claim 1, wherein determining that one or more abnormal driving actions are being performed includes comparing the driving actions with a behavior model, and the behavior model is an aggregation model based on information from a plurality of drivers other than the driver of the vehicle.

4. The method according to claim 1, wherein determining that one or more abnormal driving actions are being performed includes comparing the driving actions with a behavior model, and the behavior model is specific to the driver based on the driver's past driving history of the vehicle.

5. The method according to claim 1, wherein determining that one or more abnormal driving actions are being performed includes evaluating at least one of the accuracy of each sensor of the perception system, the sensor field of view, or the sensor arrangement around the vehicle.

6. The method according to claim 1, wherein determining that one or more abnormal driving actions are being performed includes evaluating at least one of lane departure, weaving within a lane, speed profile, braking profile, or direction change profile.

7. The method according to claim 1, wherein determining that one or more abnormal driving actions are being performed includes comparing the state tracking of the driver behavior with map information.

8. The method according to claim 1, wherein the one or more classes of actions include at least one of a warning to the driver, an autonomous operation of the vehicle, or a warning to another vehicle.

9. The method according to claim 1, further comprising performing calibration of the first sensor or the second sensor based on the state tracking based on the sensor information of the first sensor of the perception system and the state tracking based on the sensor information of the second sensor of the perception system.

10. The method according to claim 1, wherein performing the selected action includes at least one of autonomously performing a modified driving action or providing a warning to the driver regarding the determined abnormal driving action.

11. A control system for a vehicle, the control system comprising: a memory configured to store a behavior model; one or more processors operably coupled to the memory, the one or more processors being configured to: receive sensor information from a perception system of the vehicle; perform state tracking of a driving action performed by a driver of the vehicle based on the sensor information; determine that one or more abnormal driving actions are being performed based on the state tracking of the driving action during one or both of a predetermined time frame or a travel distance; select an action from one or more classes of actions when it is determined that the one or more abnormal driving actions are being performed; and perform the selected action.

12. The control system according to claim 11, wherein determining that the one or more abnormal driving actions are being performed includes determining that the one or more abnormal driving actions are being performed based on a number of predetermined behaviors in the driving action during one or both of the predetermined time frame or the travel distance.

13. The control system according to claim 11, wherein determining that the one or more abnormal driving actions are being performed includes comparing the driving action with a behavior model, the behavior model being an aggregated model based on information from a plurality of drivers other than the driver of the vehicle.

14. Determining that the one or more abnormal driving actions are being performed includes comparing the driving actions with a behavior model, wherein the behavior model is specific to the driver and is based on the driver's past driving history of the vehicle, the control system according to claim 11.

15. Determining that the one or more abnormal driving actions are being performed includes evaluating at least one of the accuracy of each sensor of the perception system, the sensor field of view, or the sensor arrangement around the vehicle, the control system according to claim 11.

16. Determining that the one or more abnormal driving actions are being performed includes evaluating at least one of lane departure, weaving within a lane, speed profile, braking profile, or steering profile, the control system according to claim 11.

17. Determining that the one or more abnormal driving actions are being performed includes comparing the state tracking of the driver behavior with map information, the control system according to claim 11.

18. The control system according to claim 11, wherein the one or more classes of actions include at least one of a warning to the driver, an autonomous operation of the vehicle, or a warning to another vehicle.

19. The control system according to claim 11, further configured to perform calibration of the first sensor or the second sensor based on the state tracking based on the sensor information of the first sensor of the perception system and the state tracking based on the sensor information of the second sensor of the perception system.

20. The control system according to claim 11, wherein the execution of the selected action includes at least one of autonomously performing a corrected driving action or providing a warning to the driver regarding the determined abnormal driving action.

21. The control system according to claim 11, further comprising the perception system.

22. A vehicle configured to evaluate a driver's abnormal driving behavior while operating in a partially autonomous driving mode, the vehicle including the control system according to claim 11, the perception system, and a driving system operably coupled to the control system.

Images