Use driver assistance to detect and resolve abnormal driver behavior

Abstract

The present technology relates to identifying and resolving abnormal driver behavior. Various driving operations can be evaluated over different time frames and driving distances (902, 906). The system can detect driving errors and suboptimal maneuvers, which are evaluated by the on-board driver assistance system and compared with the model of expected driver behavior (908). The results of this comparison can be used to alert the driver or take corrective driving actions immediately (912, 914). It can also be used for real-time or offline training or sensor calibration purposes (912, 914). The behavior model can be driver-specific, or it can be a nominal driver model based on aggregated information from many drivers. Drivers of passenger vehicles, buses, freight trucks, and other vehicles can adopt these methods.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of the filing dates of U.S. Patent Application No. 17 / 065,722, filed on October 8, 2020, and U.S. Provisional Application No. 62 / 915,134, filed on October 15, 2019, the entire disclosures of which are incorporated herein by reference. Background Art

[0003] Existing in-vehicle advanced driver assistance systems (ADAS) can automate certain driving tasks. For example, ADAS can provide warnings about getting too close to the vehicle in front and provide lane departure warnings via forward collision warnings. In other examples, ADAS can also adjust driving via automatic lane following, automatic braking, and advanced cruise control features. Other systems can also track the driver's eye movements and whether his hands are on the steering wheel, which can be used to assess attention. These methods can operate based on the instantaneous vehicle state within a short time range (e.g., the first few seconds). However, such methods may not be able to provide robustness assessments for certain repetitive behaviors or use this assessment in route planning or other driving operations. Summary of the invention

[0004] The present technology relates to identifying and resolving abnormal driver behavior in a vehicle that can be performed in a partially autonomous driving mode, or otherwise includes an advanced driver assistance system. This can include passenger vehicles, buses, freight vehicles, etc. According to various aspects of the present technology, driver operations can be evaluated within a meaningful time frame (e.g., tens of seconds, minutes, hours, or longer). Different factors can be considered in the evaluation, including driver errors and suboptimal maneuvers that may not rise to a threshold level of "error". Driver errors may include one or more lane departures, delayed braking for upcoming obstacles, failure to comply with traffic signals or signs, etc. Suboptimal maneuvers may include swinging or otherwise drifting within a lane without departing from the lane, which will not trigger a lane departure warning. Other suboptimal maneuvers may include sudden lane changes, failure to signal, hard braking, jittery / intermittent braking, etc. Any or all of these suboptimal maneuvers can be evaluated by an on-board driver assistance system and compared with a model of expected driver behavior. The information generated as a result of this comparison can be used in various ways. For example, the on-board system can immediately take corrective driving actions and / or alert the driver. It can also be used for offline training or sensor calibration purposes. These and other aspects of the present technology are discussed further below.

[0005] According to one aspect, a method is provided. The method includes performing state tracking of driving actions performed by a vehicle driver by one or more processors, and receiving sensor information from a perception system of the vehicle by one or more processors. The method also includes determining that one or more abnormal driving actions have been performed based on state tracking of driving actions during one or both of a given time frame or a distance traveled, and comparing the determined abnormal driving actions with a behavior model. The method also includes creating a variance signal based on the comparison. The variance signal indicates a deviation of the driver's performance relative to the behavior model. The method includes selecting an action from one or more classes of actions in response to the variance signal, and then executing the selected action.

[0006] In one example, the behavior model is an aggregate model based on information from multiple drivers other than the driver of the vehicle. In another example, the behavior model is specific to the driver of the vehicle based on his or her past driving history.

[0007] Determining that one or more abnormal driving maneuvers have been performed may include evaluating at least one of a number of sensors of a perception system, an accuracy of each sensor of the perception system, a sensor field of view, or an arrangement of sensors around the vehicle. Determining that one or more abnormal driving maneuvers have been performed may additionally or alternatively include evaluating at least one of a lane departure, swaying within a lane, a speed profile, a braking profile, or a turning profile. And determining that one or more abnormal driving maneuvers have been performed may additionally or alternatively include comparing a state tracking of the driver's actions with at least one of the map information and the received sensor information.

[0008] The one or more types of actions may include at least one of warning the driver, autonomous operation of the vehicle, or alerting other vehicles.

[0009] The method may also include performing calibration of one or more sensors of the perception system based on the state tracking. And performing the selected action may include at least one of autonomously performing a corrective driving action or providing a warning to the driver regarding the determined abnormal driving action.

[0010] According to another aspect, a control system for a vehicle is provided. The control system includes a memory configured to store a behavior model, and one or more processors operably coupled to the memory. The one or more processors are configured to perform state tracking of driving actions performed by a vehicle driver, receive sensor information from a perception system of the vehicle, and determine that one or more abnormal driving actions have been performed based on the state tracking of driving actions during one or both of a given time frame or a distance traveled. The processor is also configured to compare the determined abnormal driving action with the behavior model, and create a variability signal based on the comparison. The variability signal indicates a deviation of the driver's performance relative to the behavior model. The processor is also configured to select an action from one or more categories of actions in response to the variability signal, and execute the selected action. Executing the selected action may include autonomously executing a corrective driving action by instructing the vehicle's driving system to take a corrective action or providing the driver with at least one of a warning about the determined abnormal driving action via the vehicle's user interface system.

[0011] In one example, the behavior model is an aggregate model based on information from multiple drivers other than the driver of the vehicle. In another example, the behavior model is specific to the driver of the vehicle based on his or her past driving history.

[0012] Determining that one or more abnormal driving maneuvers have been performed may include evaluating at least one of a number of sensors of a perception system, an accuracy of each sensor of the perception system, a sensor field of view, or an arrangement of sensors around the vehicle. Alternatively or additionally, determining that one or more abnormal driving maneuvers have been performed includes evaluating at least one of a lane departure, swaying within a lane, a speed profile, a braking profile, or a turning profile. And determining that one or more abnormal driving maneuvers have been performed may additionally or alternatively include comparing a state tracking of the driver's actions with at least one of the map information and the received sensor information.

[0013] The one or more types of actions may include at least one of warning the driver, autonomous operation of the vehicle, or alerting other vehicles.

[0014] The control system may also be configured to perform calibration of one or more sensors of the perception system based on the state tracking.

[0015] The control system may also include a sensing system.

[0016] And according to another aspect, a vehicle configured to evaluate a driver's abnormal driving maneuvers when operating in a partially autonomous driving mode. The vehicle includes a control system, a perception system, and a driving system as described above and in further detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A-1B An example passenger-type vehicle configured for use with aspects of the present technology is shown.

[0018] Figure 1C-Figure 1D An example cargo-type vehicle configured for use with aspects of the present technology is shown.

[0019] Figure 2 is a block diagram of a system of an example passenger vehicle in accordance with aspects of the present technique.

[0020] Figure 3A-3B is a block diagram of a system of an example freight-type vehicle in accordance with aspects of the present technique.

[0021] Figure 4 An example drift scenario in accordance with aspects of the present technique is shown.

[0022] Figure 5 Example braking scenarios in accordance with aspects of the present technique are shown.

[0023] Figure 6 An example speed profile scenario in accordance with aspects of the present technique is shown.

[0024] Figure 7A-7B An example of alerting nearby vehicles according to aspects of the present disclosure is shown.

[0025] Figure 8A-8B Example systems in accordance with aspects of the present technology are shown.

[0026] Fig. 9 Example methods according to aspects of the present technology are shown. DETAILED DESCRIPTION

[0027] The on-board driver assistance system evaluates information about the driver's operation of the vehicle over time. The evaluation may include comparing a specific action (or inaction) with a stored driver behavior model. The behavior model may be based on the driver's past actions and driving habits, a nominal model based on general driving expectations, or other factors. One or both of driving errors and suboptimal driving may be included in the evaluation. As a result, the system may determine that the operation of the vehicle is abnormal, in which case corrective actions may be performed or the driver or others may be alerted.

[0028] Example Vehicle Systems

[0029] Figure 1A A perspective view of an example passenger vehicle 100 , such as a sedan, minivan, sport utility vehicle (SUV), or other vehicle, is shown. Figure 1BA top view of a passenger vehicle 100 is shown. The passenger vehicle 100 may include various sensors for obtaining information about the vehicle's external environment, which may be used by an onboard ADAS system or other system capable of operating the vehicle in a semi-autonomous driving mode. For example, a roof-top unit 102 may include a Lidar sensor as well as various cameras, radar units, infrared and / or acoustic sensors. The sensor unit 104 located at the front end of the vehicle 100 and units 106a and 106b located on the driver's side and passenger side of the vehicle may each incorporate a Lidar, radar, camera and / or other sensor. For example, unit 106a may be located along the vehicle's quarter panel in front of the driver's side door. As Figure 1B As shown, the sensor unit 108 may also be placed along the rear of the vehicle 100, such as on or adjacent to the bumper. In some examples, the passenger vehicle 100 also includes various sensors (not shown) for obtaining information about the interior space of the vehicle.

[0030] Figure 1C-Figure 1D An example freight vehicle 150 is shown, such as a tractor-trailer truck. The truck may include, for example, a single trailer, a double trailer, or a triple trailer, or may be another medium or heavy truck (such as a commercial weight class of 4 to 8). As shown, the truck includes a tractor unit 152 and a single freight unit or trailer 154. Depending on the type of goods being transported, the trailer 154 may be fully enclosed, open (such as a flatbed), or partially open. In this example, the tractor unit 152 includes an engine and a steering system (not shown) and a cab 156 for the driver and any passengers.

[0031] The trailer 154 includes a hitching point 158, known as a kingpin. The kingpin 158 is typically formed as a solid steel axle that is configured to be pivotally attached to the tractor unit 152. Specifically, the kingpin 158 is attached to a trailer coupling 160, known as a fifth-wheel, mounted to the rear of the cab. For a double tractor-trailer or triple tractor-trailer, the second trailer and / or the third trailer can have a simple bolted connection to the lead trailer. Or, alternatively, each trailer can have its own kingpin. In this case, at least the first trailer and the second trailer can include a fifth-wheel type structure arranged to couple to the next trailer.

[0032] As shown, the tractor can have one or more sensor units 162, 164 positioned along it. For example, one or more sensor units 162 can be positioned on a roof portion or top portion of the cab 156, and one or more side sensor units 164 can be positioned on the left and / or right side of the cab 156. The location of the sensor units can also be along other areas of the cab 156, such as along the front bumper or hood area, at the rear of the cab, adjacent to the fifth wheel, under the chassis, etc. The trailer 154 can also have one or more sensor units 166 positioned along the trailer 154, for example, along the side panels, front, rear, roof, and / or undercarriage of the trailer 154.

[0033] and Figure 1A-1B Like the sensor units of a passenger vehicle, each sensor unit of a freight vehicle may include one or more sensors such as lidar, radar, camera (e.g., optical or infrared), acoustic (e.g., microphone or sonar type sensor), inertial (e.g., accelerometer, gyroscope, etc.), or other sensors (e.g., positioning sensor such as GPS sensor). Although certain aspects of the present disclosure may be particularly useful in conjunction with specific types of vehicles, the vehicle may be any type of vehicle, including but not limited to cars, trucks, motorcycles, buses, recreational vehicles, etc.

[0034] As mentioned above, the present technology is applicable to advanced driver assistance systems (ADAS) and other systems that can provide at least partial autonomy in autonomous driving mode. For vehicles operating in partially autonomous driving mode, there can be different degrees of autonomy. The National Highway Traffic Safety Administration and the Society of Automotive Engineers have identified different levels to indicate the extent to which the vehicle can control driving. For example, level 0 has no automation, and the driver makes all driving-related decisions. The lowest semi-autonomous mode (level 1) includes some driving assistance, such as cruise control. Level 2 has partial automation of certain driving operations, and level 3 involves conditional automation that can enable a person in the driver's seat to take control when necessary. In contrast, level 4 is a highly automated level, in which the vehicle can drive without assistance under selected conditions. The architecture, components, systems and methods described herein can work in any of these modes. These systems are generally referred to as driver assistance systems herein.

[0035] Figure 2A block diagram 200 is shown with various components and systems of an exemplary vehicle, such as the passenger vehicle 100, to enable driver assistance. As shown, the block diagram 200 includes a control system having one or more computing devices 202, such as a computing device including one or more processors 204, memory 206, and other components typically found in a general-purpose computing device. The memory 206 stores information accessible by the one or more processors 204, including instructions 208 and data 210 that can be executed or otherwise used by the processors 204. When providing driver assistance or otherwise operating in a partially autonomous driving mode, the computing system can control one or more operations of the vehicle.

[0036] The memory 206 stores information accessible by the processor 204, including instructions 208 and data 210 that can be executed or otherwise used by the processor 204. As an example, the data 210 may include one or more behavior models associated with driving behavior. The memory 206 may be any type of memory capable of storing information accessible by the processor, including computing device readable media. The memory is a non-transitory medium, such as a hard drive, memory card, optical disk, solid state, etc. The system may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

[0037] Instructions 208 may be any set of instructions executed by a processor directly (such as machine code) or indirectly (such as a script). For example, instructions may be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions," "modules," and "programs" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language including a collection of independent source code modules or scripts that are interpreted on demand or pre-compiled. One or more processors 204 may retrieve, store, or modify data 210 based on instructions 208. In one example, some or all of memory 206 may be an event data recorder or other secure data storage system configured to store sensor data for vehicle diagnostics and / or detection, which may be onboard the vehicle or remote, depending on the implementation.

[0038] Processor 204 may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. Figure 2The processor, memory, and other elements of computing device 202 are functionally shown as being within the same box, but such a device may actually include multiple processors, computing devices, or memories that may or may not be housed within the same physical housing. Similarly, memory 206 may be a hard drive or other storage medium located in a housing different from that of processor 204. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0039] In one example, computing device 202 may form a driving computing system incorporated into vehicle 100 so that the vehicle can perform various driver assistance operations. The driving computing system is capable of communicating with various components of the vehicle. For example, computing device 202 may communicate with various systems of the vehicle, including a driving system, including a deceleration system 212 (for controlling the braking of the vehicle), an acceleration system 214 (for controlling the acceleration of the vehicle), a steering system 216 (for controlling the orientation of the wheels and the direction of the vehicle), a signal system 218 (for controlling turn signals), a navigation system 220 (for navigating the vehicle to a location or around an object), and a positioning system 222 (for determining the location of the vehicle, for example, including the posture of the vehicle).

[0040] As shown in this example, computing device 202 is also operably coupled to perception system 224 (for detecting objects in the vehicle's environment), power system 226 (e.g., a battery and / or a gasoline or diesel powered engine), and drive system 230 to control the vehicle's motion, speed, etc. in a partially autonomous driving mode according to instructions 208 of memory 206. Some or all of wheels / tires 228 are coupled to drive system 230, and computing device 202 is able to receive information about tire pressure, balance, and other factors that can affect driving.

[0041] The computing device 202 can control the direction and speed of the vehicle by controlling various components. As an example, the computing device 202 can use data from the map information and navigation system 220 to navigate the vehicle to the destination location. The computing device 202 can use the positioning system 222 to determine the location of the vehicle, and use the perception system 224 to detect objects and respond to objects when necessary to safely reach the location. To this end, the computing device 202 can accelerate the vehicle (e.g., by increasing the fuel or other energy provided to the engine by the acceleration system 214), decelerate (e.g., by reducing the fuel supplied to the engine, shifting gears and / or applying brakes through the deceleration system 212), change direction (e.g., by turning the front wheels or other wheels of the vehicle 100 through the steering system 216), and signal such changes (e.g., by lighting the turn signal of the signal system 218). Therefore, the acceleration system 214 and the deceleration system 212 can be part of a drivetrain or other type of transmission system 230 including various components between the vehicle engine and the vehicle wheels. Similarly, by controlling these systems, the computing device 202 can also control the vehicle's transmission system 230 in order to manipulate the vehicle.

[0042] The computing device 202 may use the navigation system 220 to determine and follow a route to a location. In this regard, the navigation system 220 and / or the memory 206 may store map information, for example, highly detailed maps that the computing device 202 may use to navigate or control a vehicle. For example, these maps may identify the shape and height of roads, lane markings, intersections, crosswalks, speed limits, traffic lights, buildings, signs, real-time traffic information, vegetation, or other such objects and information. Lane markings may include features such as solid or dashed double lane lines or single lane lines, solid or dashed lane lines, reflectors, and the like. A given lane may be associated with a left lane line and / or a right lane line or other lane markings that define lane boundaries. Thus, most lanes may be bounded by the left edge of one lane line and the right edge of another lane line.

[0043] The perception system 224, which may be part of or work in conjunction with an advanced driver assistance system, includes one or more onboard sensors 232. The sensors may be used to detect objects outside the vehicle. The detected objects may be other vehicles, obstacles in the road, traffic signals, signs, trees, etc. The sensors 232 may also detect certain aspects of weather conditions, such as snow, rain, or mist, or puddles, ice, or other materials on the road.

[0044] By way of example only, the perception system 224 may include one or more lidar sensors, radar units, cameras (e.g., optical imaging devices with or without neutral density (ND) filters), positioning sensors (e.g., gyroscopes, accelerometers, and / or other inertial components), infrared sensors, acoustic sensors (e.g., microphones or sonar transducers), and / or any other detection device that records data that can be processed by the computing device 202. Such sensors of the perception system 224 may detect objects outside the vehicle and their characteristics, such as location, orientation, size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), direction, speed of movement relative to the vehicle, etc. The perception system 224 may also include other sensors within the vehicle (such as within the passenger compartment) to detect objects and conditions within the vehicle. For example, such sensors may detect the orientation of the driver, whether his or her hands are on the steering wheel, eye gaze direction, etc. The sensors 232 of the perception system 224 may further measure the rotational speed of the wheels 228, the amount or type of braking of the deceleration system 312, and other factors associated with the equipment of the vehicle itself.

[0045] The raw data from the sensors and the aforementioned characteristics may be processed by the perception system 224 and / or sent to the computing device 202 for further processing periodically or continuously as the perception system 224 generates data. The computing device 202 may use the positioning system 222 to determine the location of the vehicle, and use the perception system 224 to detect objects and respond to the objects based on an assessment of the driver's behavior as discussed further below. In addition, the computing device 202 may perform calibration of individual sensors, all sensors in a particular sensor assembly, or between sensors in different sensor assemblies or other physical housings.

[0046] like Figure 1A-1B As shown, some sensors of the perception system 224 can be incorporated into one or more sensor units. In some examples, these sensor units can be integrated into the side mirrors on the vehicle. In another example, other sensors can be part of the roof shell 102 or other sensor units. The computing device 202 can communicate with sensor units located on the vehicle or otherwise distributed along the vehicle. Each unit can have one or more types of sensors, such as those described above.

[0047] Back to Figure 2, computing device 202 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, as well as a user interface subsystem 234. User interface subsystem 234 may include one or more user inputs 236 (e.g., a mouse, keyboard, touch screen, and / or microphone) and one or more display devices 238 (e.g., a monitor with a screen or any other electronic device operable to display information). In this regard, an internal electronic display may be located in a cabin of the vehicle (not shown) and may be used by computing device 202 to provide information to passengers within the vehicle. Other output devices (such as speakers 240 and / or haptic actuators 241 that provide tactile feedback to the driver) may also be located within the passenger vehicle.

[0048] The passenger vehicle also includes a communication system 242. For example, the communication system 242 may also include one or more wireless configurations to facilitate communication with other computing devices, such as a passenger computing device within the vehicle, a computing device outside the vehicle (such as within another nearby vehicle on the road), and / or a remote server system. The network connection may include a short-range communication protocol (such as Bluetooth TM ,Bluetooth TM Low Energy (LE), cellular connections) and various configurations and protocols including the Internet, the World Wide Web, an intranet, a VPN, a WAN, a LAN, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi and HTTP, and various combinations of the foregoing.

[0049] Figure 3A A vehicle (e.g., Figure 1C As shown in block diagram 300, the vehicle includes a control system of one or more computing devices, such as a processor 304, a memory 306, and a control system similar to those described above. Figure 2 A computing device 302 of other components similar or equivalent to the components 202, 204, and 206 discussed. In this example, the control system may constitute an electronic control unit (ECU) of a tractor unit of a freight vehicle. As with instructions 208, instructions 308 may be any set of instructions directly executed by a processor (such as machine code) or indirectly executed (such as a script). Similarly, one or more processors 304 may retrieve, store, or modify data 310 according to instructions 308.

[0050] In one example, computing device 302 may form a driving computing system incorporated into vehicle 150. Similar to the above reference Figure 2As discussed above, the driving computing system of block diagram 300 can communicate with various components of the vehicle in order to perform driving operations. For example, computing device 302 can communicate with various systems of the vehicle, such as a driving system, which includes a deceleration system 312, an acceleration system 314, a steering system 316, a signal system 318, a navigation system 320, and a positioning system 322, each of which can be described above with reference to Figure 2 Works as discussed.

[0051] Computing device 302 is also operably coupled to perception system 324, power system 326, and drive system 330. Some or all of wheels / tires 328 are coupled to drive system 330, and computing device 302 is capable of receiving information about tire pressure, balance, rotation speed, and other factors that may affect driving. As with computing device 202, computing device 302 can control the direction and speed of the vehicle by controlling various components. As an example, computing device 302 can use data from map information and navigation system 320 to assist in navigating the vehicle to a destination location.

[0052] Perception system 324 may be part of, or work in conjunction with, an advanced driver assistance system. Similar to perception system 224, perception system 324 may also include one or more sensors or other components, such as those described above, for detecting objects external to the vehicle, objects or conditions internal to the vehicle, and / or the operation of certain vehicle equipment, such as wheels and deceleration system 312. For example, Figure 3A As indicated, the perception system 324 includes one or more sensor assemblies 332. Each sensor assembly 232 includes one or more sensors. In one example, the sensor assembly 332 may be arranged as a sensor tower integrated into a side-view mirror of a truck, farm equipment, construction equipment, etc. As described above with reference to Figure 1C-Figure 1D As described, the sensor assembly 332 may also be placed at different locations on the tractor unit 152 or the trailer 154. The computing device 302 may communicate with the sensor assemblies located on both the tractor unit 152 and the trailer 154. Each assembly may have one or more types of sensors, such as those described above.

[0053] Figure 3A Also shown is a coupling system 334 for connection between the tractor unit and the trailer. The coupling system 334 may include one or more power and / or pneumatic connections (not shown), and a fifth wheel 336 at the tractor unit for connection to a pivot hitch at the trailer. A communication system 338 (equivalent to the communication system 242) is also shown as part of the vehicle system 300. Similarly, a user interface 339 equivalent to the user interface 234 may also be included for interaction with the driver or any passengers of the vehicle.

[0054] Figure 3B A trailer such as Figure 1C-Figure 1D An example block diagram 340 of a system for a vehicle that is equipped with a trailer 154. As shown, the system includes an ECU 342 of one or more computing devices, such as a computing device that includes one or more processors 344, a memory 346, and other components typically found in a general-purpose computing device. The memory 346 stores information that can be accessed by the one or more processors 344, including instructions 348 and data 350 that can be executed or otherwise used by the processor 344. Figure 2 and Figure 3A The description of processors, memory, instructions and data applies to Figure 3B These elements in.

[0055] The ECU 342 is configured to receive information and control signals from the trailer unit. The onboard processor 344 of the ECU 342 can communicate with various systems of the trailer, including the deceleration system 352, the signaling system 254, and the positioning system 356. The ECU 342 can also be operably coupled to a perception system 358 having one or more sensors for detecting objects in the trailer environment and a power system 260 (e.g., a battery power source) that provides power to local components. Some or all of the wheels / tires 362 of the trailer can be coupled to the deceleration system 352, and the processor 344 can receive information about tire pressure, balance, wheel speed, and other factors that may affect driving in autonomous mode, and relay this information to the processing system of the tractor unit. The deceleration system 352, the signaling system 354, the positioning system 356, the perception system 358, the power system 360, and the wheels / tires 362 can be arranged in a manner such as that described above. Figure 2 and Figure 3A Operate in this way.

[0056] The trailer also includes a collection of landing gear 366 and a coupling system 368. The landing gear provides a support structure for the trailer when the trailer is decoupled from the tractor unit. The coupling system 368, which may be part of the coupling system 334, provides a connection between the trailer and the tractor unit. Thus, the coupling system 368 may include a connection portion 370 (e.g., for power and / or pneumatic connection). The coupling system also includes a central hitch 372 configured for connection to a fifth wheel of the tractor unit.

[0057] Example Implementations

[0058] According to various aspects of the technology, information about a driver's driving errors and suboptimal maneuvers is tracked over time, for example, by an onboard driver assistance system or other system of the vehicle. The information may be stored by the vehicle and / or shared with a remote system, for example, to perform corrective driving actions, alert the driver, provide training, calibrate sensors of the vehicle's perception system, etc. Storing or transmitting the information may include anonymizing certain details about the information obtained, the vehicle, and the trip. Information as part of the storage or transmission process may be encrypted.

[0059] exist Figure 4 In one example 400 shown, a perception system (alone or in combination with other systems such as a positioning system and / or a navigation system) can track the relative position of a vehicle in a lane of a road. As shown, the vehicle may drift within the lane without crossing the lane marking, which will not trigger a lane departure warning. An instantaneous or short-term (e.g., 2-3 seconds) evaluation may not identify any problems. However, a longer evaluation, such as 30 seconds or more, over multiple blocks or at least 1 / 4 mile, can identify repeated or regular drifting. This may be caused by driver inattention, poor wheel alignment, or other problems.

[0060] exist Figure 5 In another example 500 shown, the vehicle's forward collision warning system can track not only whether the driver has been unable to brake, but also whether the driver's braking reaction time is significantly higher than that of a normal (nominal) driver. This example shows three braking modes: late hard braking; repeated braking; and ideal (nominal) braking. Late hard braking may be caused by inattention, obstacles in the driver's field of view, unexpected swerving of the vehicle into the lane or running through a stop sign, etc. Here, the system can evaluate various factors. For example, does the braking exceed the nominal time threshold for starting braking? How much braking force is applied (deceleration)? Is the braking smooth (continuous) or abrupt (intermittent type of braking)? Additional details from the perception system may provide information about whether there is an obstacle or unexpected action of another vehicle. Similar to the above situation, an instantaneous or short-term evaluation can make a determination that the driver's behavior is (or is not) abnormal based on a very small data set. Evaluations over a long period of time or at a longer distance can identify patterns of driver behavior.

[0061] exist Figure 6 In yet another example 600, an onboard system tracks driving speed. Tracking of the vehicle's speed over time (speed profile) may be evaluated relative to road type (e.g., freeway vs. surface street), traffic volume, time of day that may cause sun glare or poor lighting, detection of traffic signals and signs, and other factors.

[0062] For these and other driving situations, the in-vehicle system uses state tracking and behavior analysis to determine whether the driver is operating within a certain range (or other measure) of permissible driving behavior. This may indicate that the driver's actions are abnormal due to short-term conditions (e.g., intoxication, drowsiness, inattention) or other factors (e.g., new driver with limited experience, first time using a particular vehicle (such as a rental car), weather, etc.).

[0063] State tracking and behavior analysis methods can create (or receive) an expected behavior model. The number and / or type of abnormal behavior can be tracked over one or both of a given time frame or a distance traveled. For example, within a selected distance (e.g., a 4-mile stretch of road) or time (e.g., during a 5-minute time span), the driver may have 2 lane departures and 3 instances of swinging within the lane. Or, alternatively, the driver can be evaluated over the entire trip. The trip can be a daily drive to work, or a longer trip that takes hours or days, such as driving a bus route or long-distance freight delivery. In addition, the evaluation can be regular, such as weekly or monthly.

[0064] Determining that these events occur is a factor in the analysis. Individual actions and overall driving maneuvers can be ranked and / or compared to a nominal driver behavior model. The model can be a general model based on aggregated, anonymized driving information from many drivers. Or, alternatively, the model can be specific to a particular driver, such as based on their past driving history.

[0065] In one example, the time of day is a factor in the model, for example, because nighttime may increase the possibility of certain damages and may also reduce the visibility of the driver. Weather conditions are another factor that affects visibility. Driver age or experience is another factor. In addition, the time of year (season), geography (e.g., mountainous vs. flat), road type (e.g., winding vs. straight, narrow vs. wide lanes) and other information can also be evaluated. Any combination of such factors can be incorporated into a nominal behavior model or a driver-specific behavior model. As an example, hard limits, scores, probability distributions, etc. can be assigned to driving distances, braking reaction times, speed profiles, or drift amounts in lanes, which can be used to evaluate whether the driver's behavior within a certain time or distance is considered abnormal.

[0066] Behavior analysis can also take into account additional information, such as the type of sensors and other equipment used by the vehicle. For example, one type of sensor (e.g., radar) may have an accuracy of about a few meters, while another type of sensor (e.g., lidar or optical camera) may have an accuracy of about a few centimeters or millimeters. A given sensor may have a limited field of view (FOV). Here, if the sensor has an accuracy of meters, the on-board perception system may not be able to determine within an acceptable degree of certainty that the driver is swinging in the lane. Alternatively, the sensor may have a blind spot that prevents the perception system from determining within an acceptable degree of certainty why there are multiple lane departures or hard braking. As an example, the more severe (abnormal) the driver's operation or reaction, the more the system may want to determine the cause of the behavior. Therefore, the number, accuracy, FOV and / or arrangement of sensors on the vehicle may also be factored into the evaluation. Information about the vehicle itself, such as vehicle size (e.g., for pickup trucks and compact cars), braking distance and visibility may also be applicable. Such information may be particularly useful for commercial vehicles (such as trucks and buses).

[0067] It may also be relevant to evaluate actions in context, such as by evaluating driving activities based on map data and perception information. Thus, the system may determine that some swings in the lane are due to construction that causes the lane to narrow or the absence of an adjacent shoulder. This may include the driver assistance system being aware of the construction due to received maps or traffic information. It may also include the driver assistance system obtaining real-time information about the driving environment from the analysis of data obtained by the perception system, such as obtaining real-time information about the driving environment from the detection of one or more construction vehicles, construction equipment or signs, debris or other indicia of the construction zone, or tailgaters present behind the vehicle. See, for example, the discussion of construction zones in U.S. Pat. No. 9,937,924 and the discussion of tailgate in U.S. Pat. No. 10,168,706, the entire disclosure of which is incorporated herein by reference. In these types of situations, the system may not consider the swing to be abnormal. In the case of lane departure, the driver's action may be due to another vehicle making an unexpected lane change or getting too close to the side or rear of the driver's vehicle. In view of this, the evaluation of driver behavior should be robust to a single event. That is, an otherwise normal driving profile with isolated instances of oscillation or hard (or soft) braking is more likely a response to road conditions, while another driving profile with such actions occurring every few minutes indicates potential damage.

[0068] Nevertheless, for fleet driving (such as for bus drivers, couriers, or tractor-trailer drivers), a longer evaluation period may be desired. In these cases, the system may aggregate driving data for individual drivers over hours, days, weeks, etc. It may also aggregate data for many drivers in a fleet (e.g., during a given time period (e.g., morning rush hour or evening rush hour), along the same route, etc.). In such cases, a new driver may be evaluated relative to a baseline of other drivers to determine if he or she is operating the vehicle in a manner that exceeds a threshold operating standard (e.g., speed, repeated braking, lane changes, etc.). Training and / or intervention may be provided based on this information.

[0069] Additionally, the complexity of the ADAS or autonomous driving system may play a role in how much context is needed to evaluate the driver's actions. As an example, a Level 2 type ADAS system that relies only on deceleration sensors may not have enough context as to why a human driver suddenly brakes when there is a potential cut in, a pedestrian entering the road from the sidewalk. Here, the ADAS system will most likely need to track deviations over a longer period of time, such as several minutes (or miles), to get a better signal-to-noise ratio to enable a determination of whether the action is abnormal. The longer the time frame, the less impact the local context has. For example, multiple braking events that are isolated in time (e.g., several minutes or hours apart) may indicate abnormal behavior without the need for the onboard system to evaluate why a particular event occurred.

[0070] In contrast, more robust partially or fully autonomous Level 3, Level 4, or Level 5 systems can better understand the reasons behind braking events. For example, in addition to detecting rapid deceleration, the perception system (e.g., lidar, radar, and / or camera sensors) can also detect that a vehicle in an adjacent lane has begun to maneuver into the driver's lane, or that a pedestrian is about to walk off the curb and into a crosswalk. As an example, another car that makes repeated lane changes or otherwise wiggles in and out of traffic may change how the driver reacts. Thus, if the driver is behind or near another vehicle at a traffic light, he or she may give the other vehicle more time to move away once the traffic light changes. Unless the onboard system understands some a priori information about the other vehicle, this delay may look like a slow reaction. In these cases, an onboard behavior prediction model can provide the probability that the other vehicle or pedestrian will move into the driver's lane of travel or take other actions. Here, a more sophisticated onboard system can distinguish between situations where braking or skidding is related to some other agent or event in the external environment and those that do not seem to have a clear cause. In such cases, the system can determine whether the driver's response falls within a predicted range or threshold range for actions for a single event.

[0071] One result of state tracking and behavioral analysis may be an error signal or other type of variability signal that indicates how well (or poorly) the driver performs various actions or responds to certain conditions. The system uses sensor information (e.g., camera and / or lidar information) to determine positioning within the lane. The deviation of the vehicle from the center of the lane is classified into two maneuvers: lane change and swing. In one scenario, lane change is classified by completely switching from one lane center to another, while swing is a movement back to the center of the lane where the movement originated. Swing can be classified according to the severity of the movement, for example, based on the amount of lateral distance from the center of the lane and / or lateral acceleration information about the amount of "jitter" during the swing. The classification can also take into account whether the swing exceeds a threshold. Alternatively or additionally, more severe swings can be weighted more heavily than less severe swings. Other scenarios can classify other driving actions, such as rapid speed changes (e.g., multiple speed increases of at least 5-10 miles per hour) or hard braking (e.g., a speed reduction of 10-15 miles per hour). Still other scenarios may classify inaction, such as failure to use a turn signal when changing lanes, failure to initiate braking when road conditions detected by sensors indicate that such braking should occur, etc.

[0072] In these scenarios, the system can maintain a driving log that can be indexed by time and distance traveled, where swings and other driving actions and / or inactions are labeled as events in the driving log. The driving log can be queried by incident. For example, the number of swings within a given amount of time or a given driving distance can be queried. This information can be stored in an onboard or remote database. As an example, this information can be stored in conjunction with a driver-specific behavior model. This information can be queried by an onboard system such as a drive planner or behavior prediction module.

[0073] Long-term analysis of driver actions (e.g., over weeks or months) can determine if certain actions of the driver have degraded, which may indicate other issues to address. For example, the system can assess whether the reaction time for a particular action (e.g., braking, lane change, continuing after a complete stop, etc.) has slowed down. In the long term, the system can also look for cyclical patterns. For example, in a commercial driving situation, the system can measure not only performance over time, but also profiles of driver performance within a shift or day and night, etc.

[0074] Once the system has determined that the driver's behavior is abnormal and exceeds the threshold or other range of permissible actions, the vehicle may take one or more different actions. One type of action includes warning the driver, such as directing the driver to pay more attention or pull over. Here, the driver may be provided with audible, visual and / or tactile feedback about individual actions (e.g., swinging within the lane or hard braking) or about repeated actions (e.g., multiple lane departures or failure to slow down at a yellow light). Another type of action involves autonomous operation of the vehicle. Here, the vehicle's speed may be limited, or even the vehicle may be slowed over time to a complete stop. Alternatively, depending on the vehicle's autonomous capabilities, it may take over some or all of the autonomous control of the driving.

[0075] Another type of action includes warning other drivers. Figure 7A-7B This can be done by turning on the hazard lights ( Fig. 7A 700) or change the headlight direction / mode ( Figure 7B Other vehicles can be alerted via the V2V communication system. Figure 8A-8B As shown in , remote systems and services (e.g., driver assistance, fleet management, law enforcement, etc.) can also be alerted. Logs of behavior-related information can be stored on the vehicle or in a remote system for future analysis, behavior model refinement, and other purposes.

[0076] In addition to the above types of actions, state tracking and behavior analysis can be used for real-time or offline driver training. For example, the information obtained can be used to train novice drivers or other drivers who may need retraining. This may be done via in-vehicle feedback (e.g., audible, visual and / or tactile feedback) or reports provided to the driver offline. Such training can use information such as local driving regulations. However, in-vehicle training may depend on how advanced the on-board driver assistance system is. For example, a partially autonomous system of the level 2 type may not be able to support such training, but a level 3 or level 4 type system can train the driver during certain manual driving operations.

[0077] Finally, the driving information obtained can be used to help calibrate or otherwise monitor the vehicle systems themselves. For example, if the system detects repeated swings to the left but not to the right in the lane, this may indicate that the positioning sensor (e.g., accelerometer or gyroscope) is not properly calibrated, the wheels are misaligned, or the camera external parameters are no longer adequately calibrated. Alternatively, if one brake sensor detects hard braking but other brake sensors and / or accelerometers do not, this may indicate that the brake sensor needs to be replaced. Driver behavior can also be used as another signal or tiebreaker when determining whether the problem is due to driving action, sensor failure, or other problems. For example, if a radar-based braking system continues to try to brake or otherwise slow the vehicle, but the camera-based system does not do so (or vice versa), the driver's actions can be used to determine which sensor may be at fault.

[0078] As mentioned above, the technology is applicable to various types of vehicles, including passenger cars, buses, RVs, and trucks or other cargo vehicles. In addition to using the driver operation information to operate the vehicle, the information can also be shared with other vehicles, such as vehicles that are part of a fleet.

[0079] Fig. 8A and Figure 8B An example of this is shown in FIG. Fig. 8A and Figure 8B 800 and a storage system 810. The system 800 also includes a plurality of computing devices 802, 804, 806, 808 and a storage system 810 connected via a network 816. Figure 1A-1B and Figure 1C-Figure 1D Vehicle 812 and vehicle 814 are configured the same or similarly to vehicles 100 and 150 in FIG. Vehicle 812 and / or vehicle 814 may be part of a fleet. Although only a small number of vehicles and computing devices are depicted for simplicity, a typical system may include significantly more vehicles and computing devices.

[0080] like Figure 8B As shown, each of computing device 802, computing device 804, computing device 806, and computing device 808 may include one or more processors, memories, data, and instructions. Such processors, memories, data, and instructions may be similar to those described above. Figure 2 Those described are configured similarly.

[0081] The various computing devices and vehicles may communicate via one or more networks, such as network 816. Network 816 and intermediate nodes may include various configurations and protocols, including short-range communication protocols such as Bluetooth. TM , Bluetooth LE TM, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi and HTTP, and various combinations of the above. Such communications may be facilitated by any device capable of transmitting and receiving data to and from other computing devices, such as modems and wireless interfaces.

[0082] In one example, computing device 802 may include one or more server computing devices (e.g., a load balancing server group) with multiple computing devices, and the server computing devices exchange information with different nodes of the network for the purpose of receiving data from other computing devices, processing data, and sending data to other computing devices and receiving data therefrom. For example, computing device 802 may include one or more server computing devices capable of communicating with computing devices of vehicle 812 and / or vehicle 814 and computing devices 804, computing devices 806, and computing devices 808 via network 816. For example, vehicle 812 and / or vehicle 814 may be part of a fleet that can be dispatched to various locations by a server computing device. In this regard, computing device 802 may be used as a dispatch server computing system that may be used to dispatch vehicles to different locations in order to pick up and drop off passengers or pick up and deliver goods. In addition, server computing device 802 may use network 816 to transmit and present information to a user of one of the other computing devices or a vehicle passenger. In this regard, computing devices 804, computing devices 806, and computing devices 808 may be considered client computing devices.

[0083] like Fig. 8A As shown, each client computing device 804, 806, and 808 can be a personal computing device intended for use by a corresponding user 818, and has all components commonly used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and internal hard drive) to store data and instructions, a display (e.g., a monitor with a screen, a touch screen, a projector, a television, or other devices such as a smart watch display that can be operated to display information), and a user input device (e.g., a mouse, keyboard, touch screen, or microphone). The client computing device may also include a camera for recording video streams, speakers, a network interface device, and all components for connecting these elements to each other.

[0084] Although each client computing device may include a full-sized personal computing device, they may alternatively include mobile computing devices capable of wirelessly exchanging data with a server over a network such as the Internet. By way of example only, client computing device 806 and client computing device 808 may be mobile phones or devices such as wireless-enabled PDAs, tablet PCs, wearable computing devices (e.g., smart watches), or netbooks capable of obtaining information via the Internet or other networks.

[0085] In some examples, the client computing device 804 may be a remote assistance workstation used by an administrator or operator to communicate with the driver of a dispatched vehicle. Figure 8A-8B Only a single remote assistant workstation 804 is shown in FIG, but any number of such workstations may be included in a given system. In addition, although the operation workstation is described as a desktop computer, the operation workstation may include various types of personal computing devices, such as laptop computers, netbooks, tablet computers, etc.

[0086] The storage system 810 may be any type of computerized storage device capable of storing information accessible by the server computing device 802, such as a hard drive, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a flash drive, and / or a tape drive. In addition, the storage system 810 may include a distributed storage system in which data is stored in a plurality of different storage devices that may be physically located in the same or different geographic locations. The storage system 810 may be connected to a computing device via a network 816 as shown in FIGS. 16A-16B , and / or may be directly connected to or incorporated into any computing device.

[0087] Storage system 810 can store various types of information. For example, storage system 810 can also store autonomous vehicle control software, which is used by vehicles such as vehicle 812 or vehicle 814 to operate such vehicles in autonomous driving mode. Storage system 810 can also store driver-specific or nominal driving models. Model information can be shared with specific vehicles or fleets on demand. It can be updated in real time, regularly updated, or offline when additional driving information is obtained. Storage system 810 can also include map information, route information, braking and / or acceleration profiles of vehicles 812 and 814, weather information, etc. This information can be shared with vehicles 812 and 814, for example, to help behavioral analysis performed by an on-board computer system during real-time driving of a specific driver.

[0088] The remote assistance workstation 804 can access the stored information and use it to assist in the operation of a single vehicle or a fleet of vehicles. Remote assistance can be used to communicate directly with the driver via the on-board user interface, or indirectly via the driver's client computing device. Here, for example, the driver can be provided with information about current driving operations, training, etc.

[0089] Fig. 9 An example method of operation 900 according to the above discussion is shown. As shown in box 902, the method performs state tracking of driving actions performed by a driver of a vehicle. In box 904, sensor information is received from a perception system of the vehicle. In box 906, the method determines one or more abnormal driving actions based on state tracking of driving actions during one or both of a given time frame or a distance traveled. In box 908, the determined abnormal driving actions are compared with a behavior model. In box 910, a variability signal is created based on the comparison. The variability signal indicates a deviation of the driver's performance relative to the behavior model. In box 912, in response to the variability signal, an action is selected from one or more classes of actions. And in box 914, the selected action is performed. The selected action may include autonomously taking corrective driving actions or providing a warning to the driver about abnormal driving actions.

[0090] Unless otherwise indicated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above-described features can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be understood by way of illustration, and not by way of limitation to the subject matter defined by the claims. In addition, the provision of examples described herein, as well as clauses with the wording "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures may identify the same or similar elements. Unless otherwise expressly indicated herein, including Fig. 9 The above processes or other operations of the methods shown in the drawings may be executed in different orders or simultaneously.

Claims

1. A method for controlling a vehicle, comprising: performing, by one or more processors, state tracking of driving maneuvers performed by a driver of the vehicle as the vehicle travels a given distance; receiving, by the one or more processors, sensor information from a perception system of the vehicle, the sensor information comprising information associated with an external environment of the vehicle; maintaining, by the one or more processors, a driving log based on state tracking of driving maneuvers and sensor information, the driving log indicating each time one of the driving maneuvers performed by the driver falls outside a threshold range and each time the driver fails to perform a driving maneuver; determining, by the one or more processors, whether the driver's behavior is abnormal based on the driving log; as well as When the driver's behavior is determined to be abnormal, the one or more processors control the vehicle to perform one or more actions when operating in the semi-automatic driving mode.

2. The method of claim 1, wherein: Determining whether the driver's behavior is abnormal includes comparing the driving action to a model of expected driver behavior, the model being an aggregate model based on information from a plurality of drivers other than the driver of the vehicle.

3. The method of claim 1, wherein: Determining whether the driver's behavior is abnormal includes comparing the driving action to a model of expected driver behavior that is specific to the driver of the vehicle based on his or her past driving history.

4. The method of claim 1, wherein: Determining whether the driver's behavior is abnormal includes evaluating at least one of a number of sensors of a perception system, an accuracy of each sensor of a perception system, a sensor field of view, or an arrangement of sensors around the vehicle.

5. The method of claim 1, wherein: Determining whether the driver's behavior is abnormal includes evaluating at least one of lane departure, swaying within a lane, a speed profile, a braking profile, or a turning profile.

6. The method of claim 1, wherein: Determining whether the driver's behavior is abnormal includes comparing the state tracking of the driving action with at least one of map information and received sensor information.

7. The method of claim 1, wherein: The one or more actions include providing tactile feedback to the driver to alert the driver that the driver's behavior is abnormal.

8. The method of claim 1, further comprising identifying, by the one or more processors based on the state tracking, whether one or more sensors of the perception system are calibrated.

9. The method of claim 1, wherein: The one or more actions include changing the orientation or pattern of one or more headlights of the vehicle to alert other vehicles of the abnormal behavior of the driver.

10. The method according to claim 1, wherein: The external environment includes at least one of (i) road conditions, or (ii) at least one other vehicle, and the road conditions include construction resulting in narrowing of one or more lanes or loss of an adjacent shoulder.

11. The method according to claim 1, wherein: The driver's behavior is abnormal due to at least one of the driver being intoxicated, drowsy, or inattentive.

12. The method according to claim 1, wherein: The driver's behavior is abnormal due to the driver's inexperience.

13. The method according to claim 1, wherein: The sensor information also includes information associated with at least one of an object or a condition within the vehicle.

14. The method according to claim 13, wherein: Sensor information indicates at least one of a positioning of the driver, whether the driver's hands are on a steering wheel of the vehicle, an eye gaze direction associated with the driver, or one or more factors associated with equipment of the vehicle.

15. The method according to claim 1, wherein: The one or more actions include limiting or reducing the speed of the vehicle.

16. A control system for a vehicle, the control system comprising: a memory configured to store the behavioral model; as well as One or more processors operably coupled to the memory, the one or more processors configured to: performing state tracking of driving maneuvers performed by a driver of the vehicle while the vehicle travels a given distance; receiving sensor information from a perception system of the vehicle, the sensor information comprising information associated with an external environment of the vehicle; maintaining a driving log based on state tracking of driving maneuvers and sensor information, the driving log indicating each time one of the driving maneuvers performed by the driver falls outside a threshold range and each time the driver fails to perform a driving maneuver; determining whether the driver's behavior is abnormal based on the driving log; as well as When the driver's behavior is determined to be abnormal, the vehicle is controlled to perform one or more actions when operating in the semi-automatic driving mode.

17. The control system of claim 16, wherein: Determining whether the driver's behavior is abnormal includes comparing the driving action to a model of expected driver behavior, the model being an aggregate model based on information from a plurality of drivers other than the driver of the vehicle.

18. The control system of claim 16, wherein: Determining whether the driver's behavior is abnormal includes comparing the driving action to a model of expected driver behavior that is specific to the driver of the vehicle based on his or her past driving history.

19. The control system of claim 16, wherein: Determining whether the driver's behavior is abnormal includes evaluating at least one of a number of sensors of a perception system, an accuracy of each sensor of a perception system, a sensor field of view, or an arrangement of sensors around the vehicle.

20. The control system of claim 16, wherein: Determining whether the driver's behavior is abnormal includes evaluating at least one of lane departure, swaying within a lane, a speed profile, a braking profile, or a turning profile.

21. The control system of claim 16, wherein: Determining whether the driver's behavior is abnormal includes comparing the state tracking of the driving action with at least one of map information and received sensor information.

22. The control system of claim 16, wherein: The one or more actions include providing tactile feedback to the driver to alert the driver that the driver's behavior is abnormal.

23. The control system of claim 16, wherein: The one or more processors are further configured to identify whether one or more sensors of the perception system are calibrated based on the state tracking.

24. The control system of claim 16, wherein: The one or more actions include changing the orientation or pattern of one or more headlights of the vehicle to alert other vehicles of the abnormal behavior of the driver.

25. A vehicle configured to evaluate a driver's abnormal driving maneuvers while operating in a semi-autonomous driving mode, the vehicle comprising the control system of claim 16, and a driving system and a perception system operably coupled to the control system.

Images