Vehicle operation with operator monitoring

Abstract

The invention provides a vehicle operation with operator monitoring. A system includes a computer including a processor and a memory. The memory includes instructions executable by the processor to determine an operator re-engagement score based on sensor data including a plurality of perceptual factors and a plurality of controllability factors. An operator re-engagement delay is adjusted based on the operator re-engagement score, and a vehicle component is actuated upon expiration of the operator re-engagement delay.

Description

Technical Field

[0001] The present disclosure relates to advanced driver assistance systems in vehicles. Background Art

[0002] For example, in the context of advanced driver assistance systems (ADAS) and the like, vehicles employ some form of steering wheel engagement verification. Many ADAS features are manual and are designed to assist with longitudinal and lateral control, such as lane centering, when the operator's hands are on the steering wheel. Additionally, hands-free features may encounter situations that require operator re-engagement. Summary of the Invention

[0003] The present disclosure provides techniques for using perception and controllability contexts as inputs to an operator re-engagement timing policy included in controlling vehicle features and / or systems. For example, perception factors can include path confidence, road type, and adjacent vehicles, and controllability factors can include lateral acceleration, road curvature, speed relative to limits, and operator engagement. A computer can include programming for determining an operator re-engagement score based on sensor data including perception factors and controllability factors. An operator re-engagement delay for providing an input to control one or more vehicle features and / or systems can be adjusted based on the re-engagement score. For example, the operator re-engagement delay can vary proportionally with a change in the re-engagement score between a minimum delay and a maximum delay. When the operator re-engagement delay expires, the computer can execute programming to actuate a feature and / or system, such as notifying the operator to place one or both hands in contact with the steering wheel. In one example, the operator re-engagement delay can be increased when conditions permit, such as low speed and a straight road. In other situations (such as a curved road and increased speed), the operator re-engagement delay can be reduced to zero or set to zero, thereby triggering an immediate re-engagement notification or actuation of other vehicle components, such as the vehicle braking system.

[0004] A system including a computer is disclosed herein. The computer has a processor and a memory. The memory includes instructions executable by the processor to determine an operator re-engagement score based on sensor data including a plurality of perception factors and a plurality of controllability factors. The instructions can include instructions for adjusting an operator re-engagement delay based on the operator re-engagement score and for actuating a vehicle component when the operator re-engagement delay expires.

[0005] Instructions for determining an operator re-engagement score can include instructions for multiplying each of the plurality of perception factors by a corresponding perception weight and then multiplying the weighted perception factors together.

[0006] The perception factors can include one or more of path confidence, road type, and adjacent vehicles.

[0007] Instructions for determining an operator re-engagement score can include instructions for multiplying each of a plurality of controllability factors by corresponding controllability weights and then multiplying the weighted controllability factors together.

[0008] The controllability factors can include one or more of lateral acceleration, road curvature, speed relative to a limit, and operator engagement.

[0009] Instructions for adjusting an operator re-engagement delay can include instructions for varying the operator re-engagement delay between a minimum delay and a maximum delay in proportion to a change in the operator re-engagement score.

[0010] Instructions for adjusting an operator re-engagement delay can include instructions for setting the operator re-engagement delay to zero when the operator re-engagement score is below a threshold.

[0011] The instructions can further include instructions for receiving an indication of detected deception and setting the operator re-engagement delay to the minimum delay in response to the indication.

[0012] Instructions for determining an operator re-engagement score can include instructions for multiplying each of a plurality of perception factors by corresponding perception weights; multiplying each of a plurality of controllability factors by corresponding controllability weights; and multiplying the weighted perception factors and the weighted controllability factors together.

[0013] The perception weights and the controllability weights can be determined empirically.

[0014] Instructions for adjusting an operator re-engagement delay can include instructions for varying the operator re-engagement delay between a minimum delay and a maximum delay in proportion to a change in the operator re-engagement score.

[0015] The actuated vehicle component can be an operator re-engagement indicator.

[0016] The actuated vehicle component can be a vehicle brake.

[0017] Disclosed herein is a method that includes determining an operator re-engagement score based on sensor data that includes a plurality of perception factors and a plurality of controllability factors. The method can include adjusting an operator re-engagement delay based on the operator re-engagement score and actuating a vehicle component when the operator re-engagement delay expires.

[0018] Determining an operator re-engagement score can include multiplying each of a plurality of sensed factors by a corresponding sensed weight and then multiplying the weighted sensed factors together.

[0019] Determining an operator re-engagement score can include multiplying each of a plurality of controllability factors by a corresponding controllability weight and then multiplying the weighted controllability factors.

[0020] Adjusting an operator re-engagement latency can include changing the operator re-engagement latency between a minimum latency and a maximum latency in proportion to a change in the re-engagement score.

[0021] An actuated vehicle component can be an operator re-engagement indicator.

[0022] The method can further include receiving an indication of detected spoofing and setting the latency to the minimum latency in response to the indication.

[0023] Adjusting an operator re-engagement latency can include setting the latency to zero when the re-engagement score is below a threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram of an example vehicle.

[0025] Figure 2 is a process flow diagram showing an example process for determining an operator re-engagement score.

[0026] Figure 3A is a process flow diagram showing an example process for adjusting operator re-engagement timing.

[0027] Figure 3B is a process flow diagram showing an alternative example process for adjusting operator re-engagement timing. DETAILED DESCRIPTION

[0028] Figure 1 is a block diagram of an example vehicle. As Figure 1 shown, system 100 includes vehicle 102, which includes computer 104 communicatively coupled via vehicle network 106 to various elements including sensors 108, subsystems or components 110 (such as steering, propulsion, and braking), human-machine interface (HMI) 112, and communication component 114. Computer 104 and server 118 discussed below include a processor and a memory. A memory such as that of computer 104 described herein includes one or more forms of non-transitory computer 104-readable media and can store instructions executable by computer 104 for performing various operations such that the vehicle computer is configured to perform various operations including those disclosed herein.

[0029] For example, computer 104 may include a general-purpose computer having a processor and memory as described above, and / or may include an electronic control unit (ECU) or controller for a specific function or set of functions, and / or a dedicated electronic circuit, the dedicated electronic circuit including an ASIC (application-specific integrated circuit) fabricated for a specific operation (e.g., an ASIC for processing data from sensors and / or transmitting data from sensor 108). In another example, computer 104 may include an FPGA (field-programmable gate array), which is an integrated circuit fabricated to be configurable by a user. In an example, a hardware description language such as VHDL (very high speed integrated circuit hardware description language) may be used in electronic design automation to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is fabricated based on VHDL programming provided before fabrication, while the logic components inside an FPGA may be configured based on, for example, VHDL programming stored in a memory electrically connected or coupled to the FPGA circuit. In some examples, a combination of a processor, an ASIC, and / or an FPGA circuit may be included in computer 104. Additionally, computer 104 may include multiple computers in a vehicle (e.g., multiple ECUs, etc.), which operate together to perform the operations attributed to computer 104 herein.

[0030] The memory of computer 104 may include any type, such as a hard disk drive, a solid-state drive, or any volatile or non-volatile medium. The memory may store the collected data transmitted by sensor 108. The memory may be a device separate from computer 104, and computer 104 may retrieve the information stored by the memory via a communication network in the vehicle (such as vehicle network 106) (e.g., via a controller area network (CAN) bus, a local interconnect network (LIN) bus, a wireless network, etc.). Alternatively or additionally, the memory may be part of computer 104, such as memory internal to computer 104.

[0031] Computer 104 may include or access instructions to operate one or more components 110, such as vehicle braking, propulsion (e.g., one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, infotainment, navigation, etc., and to determine whether and when computer 104 (as opposed to a human operator) controls such operations. Computer 104 may include more than one processor or be communicatively coupled to more than one processor, for example, via vehicle network 106, and the processors may be included in components 110 (such as sensors 108, ECUs, etc.) included in the vehicle for monitoring and / or controlling various vehicle components (e.g., a powertrain controller, a brake controller, a steering controller, etc.).

[0032] The vehicle sensor 108 may also include a torque sensor 122 that operates to measure the torque (i.e., torsional moment) applied to a steering element (e.g., the steering wheel 124) when an operator applies a rotational force to the steering wheel to control the heading of the vehicle 102. In an example, the torque sensor 122 is mounted to the steering column 126 to measure torque in the range of 0.02 Newton - meters (N - m) to 0.2 N - m. The torque sensor 122 may include a calibrated strain gauge, for example, to provide a voltage signal proportional to the torque applied to the steering wheel 124. In an example, when the vehicle 102 is moving along the path 150, the torque sensor 122 can always or almost always measure the torque with which the operator steers the vehicle 102 with at least one hand on the steering wheel 124. In response to the operator removing their hand from the steering wheel 124, the torque sensor 122 measures zero torque or other negligible amounts of torque transmitted to the steering column 126. It should be noted that the torque sensor 122 can measure torque corresponding to other objects placed in contact with the steering wheel 124, such as various foreign objects, e.g., a laptop computer, a water bottle, a coffee container, etc. It should be noted that although Figure 1 a steering wheel is shown, the techniques described herein can be applied to steering elements other than the steering wheel of a vehicle, such as a joystick, an aircraft control yoke, etc.

[0033] The steering wheel 124 may include additional sensors 108, such as capacitance sensors, mounted in, on, or near the steering wheel 124, for example, to determine whether the operator's hand is in contact with the steering wheel 124. Such sensors can operate to record a change in self - capacitance in response to the operator's hand approaching a capacitance sensor mounted on the steering wheel 124. However, it should also be noted that capacitance (e.g., self - capacitance (such as the capacitance of a single capacitance sensor), mutual capacitance (such as the capacitance between two or more capacitance sensors), etc.) may be affected by the proximity of certain foreign objects, such as a water bottle, a coffee container, etc. In one example, other sensors may be located on the steering wheel 124, such as sensors for measuring forces, torques, temperatures, and other properties in a global or local coordinate reference position and / or orientation that may be directly or indirectly related to the operator's input to the steering wheel.

[0034] Computer 104 can generally be arranged to communicate over vehicle network 106, which may include a communication bus in the vehicle, such as a Controller Area Network (CAN), and / or other wired and / or wireless mechanisms. Vehicle network 106 corresponds to a communication network that can facilitate message exchange among various in-vehicle devices (e.g., sensors 108, components 110, computer 104, and computers on vehicle 102). Computer 104 can generally be programmed to send and / or receive messages via vehicle network 106 to and / or from other devices in the vehicle (e.g., any one or all of an ECU, sensors 108, actuators, components 110, communication modules, HMI 112, etc.). For example, subsystems of various components 110 (e.g., component 110) can be controlled by corresponding ECUs.

[0035] In addition, in embodiments where computer 104 actually includes multiple devices, vehicle network 106 can be used for communication between the devices represented as computer 104 in this disclosure. For example, vehicle network 106 can provide communication capabilities via a wired bus (such as a CAN bus, LIN bus), or can utilize any type of wireless communication capabilities. Vehicle network 106 can include a network in which any other wired communication technology and / or wireless communication technology (e.g., Ethernet, etc.) is used to convey messages. Additional examples of protocols that can be used for communication over vehicle network 106 in some embodiments include, but are not limited to, Media Oriented Systems Transport (MOST), Time-Triggered Protocol (TTP), and FlexRay. In some embodiments, vehicle network 106 can represent a combination of multiple networks (possibly of different types) that support communication between devices on the vehicle. For example, vehicle network 106 can include: a CAN bus, through which some in-vehicle sensors and / or components communicate; and a wired or wireless local area network, in which a certain device in the vehicle communicates according to Ethernet, and / or Bluetooth communication protocols.

[0036] In addition to torque sensors, capacitance sensors, and other sensors related to determining whether an operator has placed a hand in contact with the steering wheel 124, the vehicle 102 generally also includes various sensors 108. The sensors 108 can include a set of devices that can obtain one or more measurements of one or more physical phenomena. Some sensors 108 detect variables characterizing the operating environment of the vehicle, such as vehicle speed settings, vehicle towing parameters, vehicle braking parameters, engine torque output, engine and transmission temperatures, battery temperature, vehicle steering parameters, and the like. Some sensors 108 detect variables characterizing the physical environment of the vehicle 102, such as ambient air temperature, humidity, weather conditions (e.g., rain, snow, etc.), parameters related to the inclination or slope of the road or other types of paths on which the vehicle is advancing, and the like. In an example, the sensors 108 can operate to detect the position or orientation of the vehicle using signals from, for example: satellite positioning systems (e.g., Global Positioning System or GPS); accelerometers, such as piezoelectric or microelectromechanical systems MEMS; gyroscopes, such as rate gyroscopes, ring laser gyroscopes, or fiber optic gyroscopes; inertial measurement units IMU; and magnetometers. In an example, the sensors 108 can include sensors for detecting aspects of the environment external to the vehicle 102, such as radar sensors, scanning lidar, cameras, and the like. The sensors 108 can also include light detection and ranging (lidar) sensors that detect the distance to an object by emitting laser pulses and measuring the time of flight of the pulses as they travel to and return from the object. The sensors 108 can include a controller and / or microprocessor that executes instructions to perform, for example, analog-to-digital conversion to convert sensed analog measurements and / or observations into input signals that can be provided to the computer 104 via, for example, the vehicle network 106.

[0037] The computer 104 can be configured to interface with devices external to the vehicle via vehicle-to-vehicle (V2V) communication using the communication component 114 and / or can, for example, interface with devices external to the vehicle via V2V communication through a wide area network (WAN) 116. The computer 104 can communicate outside of the vehicle 102, such as via vehicle-to-infrastructure (V2I) communication, vehicle-to-everything (V2X) communication, or V2X including cellular vehicle-to-everything C-V2X and / or wireless communication dedicated short range communication DSRC, etc. Communication outside of the vehicle 102 can be facilitated through direct radio frequency communication and / or via a network server 118. The communication component 114 can include one or more mechanisms by which the computer 104 communicates with vehicles external to the vehicle 102, including any desired combination of wireless (e.g., cellular, wireless, satellite, microwave, radio frequency) communication mechanisms and any one or more desired network topologies (when multiple communication mechanisms are used).

[0038] Vehicle 102 may include an HMI 112, such as one or more of an infotainment display, a touchscreen display, a microphone, a speaker, a haptic device, etc. A user, such as an operator of vehicle 102, may provide input to a device, such as computer 104, via the HMI 112. The HMI 112 may communicate with the computer 104 via the vehicle network 106. For example, the HMI 112 may send a message to the computer 104 that includes user input provided via a touchscreen, a microphone, a camera that captures gestures, etc., and / or may display an output via, for example, a display, a speaker, etc. Additionally, the operation of the HMI 112 may be performed by a portable user device (not shown), such as a smartphone, etc., that communicates with the computer 104 via, for example, Bluetooth, etc.

[0039] The WAN 116 may include one or more mechanisms through which the computer 104 may communicate with the server 118. The server 118 may include a device having one or more computing devices accessible via the WAN 116. For example, having a corresponding processor and memory and / or an associated data storage area. In an example, the vehicle 102 may include a wireless transceiver (i.e., a transmitter and / or a receiver) to send messages external to the vehicle 102 and receive messages external to the vehicle. Thus, the network may include one or more of a variety of wired or wireless communication mechanisms, including any desired combination of wired (e.g., cable and fiber optic) and / or wireless (e.g., cellular, wireless, satellite, microwave, and radio frequency) communication mechanisms and any one or more desired network topologies (when using multiple communication mechanisms). Exemplary communication networks include, for example, wireless communication networks that use Bluetooth, Bluetooth Low Energy BLE, IEEE 802.11, V2V or V2X (such as Cellular V2X CV2X, DSRC, etc.), local area networks, and / or the wide area network 116 that includes the Internet.

[0040] In an example implementation, computer 104 may utilize a camera mounted on the vehicle's dashboard to obtain an image of a portion inside vehicle 102, which portion may include an operator sitting in the operator's seat of vehicle 102. In some examples, the image of the operator includes, for example, the operator's head, neck, shoulders, chest, and / or upper arms, but may not include an image of the operator's hands. Thus, suitable image processing programming executed by computer 104 may extract parameters of the image to form a nodal model of the portion of the operator's body that is within the field of view of the camera. Computer 104 may additionally execute programming to estimate the position of features (e.g., hands) that are not included or are occluded outside the field of view of the camera. In this context, a nodal model refers to a representation of the operator by a system of nodes connected by lines to represent the operator's body parts or features. A nodal model is an example of a hands-off detection (HOOD). Any suitable HOOD technique may be used in conjunction with the techniques disclosed herein.

[0041] Computer 104 may also execute programming to determine whether the nodal model of the operator is consistent within a specified confidence level (e.g., greater than 95%) with a nodal model of the operator with one or both hands placed in contact with the steering wheel. Conversely, in response to determining that the nodal model of the operator is not consistent with a nodal model of the operator with one or both hands placed in contact with the steering wheel and / or that the confidence level is below the specified confidence level, computer 104 may execute programming to actuate or generate a signal indicating that one or both of the operator's hands are not in contact with the steering wheel 124.

[0042] A nodal model is an example of a HOOD system that can also be used to detect deception. When the system has determined that the operator's hands are not in contact with the steering wheel 124, the system may measure the steering wheel torque, which may represent the weight of an external object (e.g., a water bottle, a cup, a computing device, etc.) placed on the steering wheel that may be considered a potential deception.

[0043] The disclosed systems and methods provide for adjusting the operator re-engagement delay based on perception factors such as path confidence, road type, and / or adjacent vehicles and controllability factors such as lateral acceleration, road curvature, speed relative to the limit, and / or operator engagement. Thus, the operator re-engagement delay may be adjusted and / or may change over time (i.e., based on such factors). The operator re-engagement delay refers to the amount of time between the last detection of the operator's hands on the steering wheel and a subsequent operator re-engagement notification and / or other vehicle action based on the time elapsed since the last detection of the operator's hands on the steering wheel.

[0044] In response to confirmation by the operator's hand on the steering wheel, the operator re - engagement delay can be reset. For example, the detection of a hand on the steering wheel can be based on a steering wheel capacitance sensor, steering wheel button interactions, and / or a specified amount of torque applied to the steering wheel.

[0045] A perception factor refers to a factor derived from data (including sensor data) characterizing the environment in which the vehicle is operating. A controllability factor refers to a factor derived from data (including sensor data) characterizing the operating dynamics of the vehicle. It can be understood that the environment in which the vehicle is operating and the operating dynamics of the vehicle may affect which ADAS features are available and, if the environment and / or operating dynamics change, may affect the time at which the vehicle's operator will re - engage.

[0046] In one example, an operator re - engagement score is determined based on the perception factor and the controllability factor. For example, the operator re - engagement delay can change (e.g., increase) proportionally to the change in the re - engagement score between a minimum delay (e.g., 15 seconds) and a maximum delay (e.g., 30 seconds). For example, a base delay (e.g., 15 seconds) can be multiplied by the operator re - engagement score.

[0047] In another example, if the operator re - engagement score is below a threshold, the system can set the delay to the minimum delay or set it to zero.

[0048] In another example, the delay can default to a maximum value, e.g., 30 seconds, and if a certain control behavior is detected, the delay is set to zero. Examples of control behaviors that cause zero delay can include entering a curve, an environmental change that causes a loss of perception, drifting within a lane, and / or drifting out of a lane due to limited torque authority.

[0049] As described above, an operator re - engagement score is determined based on multiple perception factors and multiple controllability factors. The perception factors can include one or more of path confidence, road type, and adjacent vehicle factor. The controllability factors can include one or more of lateral acceleration, road curvature, speed relative to the limit, and operator engagement factor. Each of these factors can include one or more variables derived from data, which can include sensor data, such as data available on the vehicle network 106 (e.g., CAN bus). Representative examples of these variables and the techniques for deriving these variables from various sensors and / or data sources are provided below.

[0050] Path confidence can be quantified by scaling or ranking data related to, for example, lane line quality, weather conditions, and map data. For example, lane line quality can be determined based on data captured by a vehicle camera, weather conditions can be determined based on location (e.g., GPS)-based weather information, and map data can be determined based on an in-vehicle navigation system. For example, path confidence can be provided by an ADAS ECU. In one example, values can be assigned to each of these variables based on scaling or ranking. For example, weather conditions such as ice, snow, wet, and dry can be ranked from 1 to 4. The most favorable condition (i.e., dry) is assigned the highest value in the scale. Lane line quality and map data can be similarly quantified. All of these variables can be combined (e.g., summed and / or weighted) to derive a path confidence factor.

[0051] Road type can be quantified based on data identifying different types of roads. For example, camera data, computer vision, and / or GPS data can be used to identify surface streets with two-way traffic, multiple lanes, and limited-access highways. For example, these different road types can be ranked from 1 to 3, where surface streets are the least favorable and limited-access highways are the most favorable.

[0052] For example, adjacent vehicles can be quantified by the number of surrounding vehicles, their relative positions, and relative speeds. For example, radar sensors, scanning lidar, cameras, ultrasonic sensors, and lidar can be used to determine these variables. Each of these variables can be assigned a relative value that has the highest value under favorable conditions such as low speed, large spacing, and uncongested traffic. All of these variables can be combined (e.g., summed and / or weighted) to derive an adjacent vehicle factor.

[0053] For example, lateral acceleration can be quantified based on vehicle accelerometer data. When a vehicle is passing through a sharp curve, the steering torque used to keep the vehicle centered in the lane may exceed the characteristic torque limit. Therefore, a low value can be assigned to high lateral acceleration.

[0054] Road curvature can also be quantified based on vehicle accelerometer data and / or vehicle camera data. While lateral acceleration focuses on smaller radii of curvature, more gentle curves can also be considered. Therefore, the gentler the curve, the higher its ranking. In another example, a road curvature factor can be calculated based on the measured radius of the curve.

[0055] The speed relative to the speed limit can be quantified as the difference between the vehicle speed and the posted speed limit. The higher the vehicle speed above the speed limit, the lower the assigned value. If the vehicle's speed is at or below the posted limit, the speed factor relative to the speed limit can be set to a base value of, for example, 50. The difference between the vehicle speed and the posted speed limit can be subtracted from the base value.

[0056] Operator engagement can be determined by an operator-facing camera to track the operator's eye position and / or monitor vehicle hands-free calls as an indicator of operator engagement. In some examples, if a favorable state is detected, operator engagement can be quantified by assigning a value to each of these variables (e.g., eye position, call, etc.). For example, if it is determined that the operator's gaze is on the road, a relatively high value is assigned, and if the operator's gaze is on, for example, the infotainment system, a relatively low value is assigned. In another example, a driver engagement score can be derived from a driver state monitoring (DSM) system.

[0057] The operator re-engagement score (ORS) can be determined by multiplying each of a plurality of perception factors (PF) by a corresponding perception weight (PW); multiplying each of a plurality of controllability factors (CF) by a corresponding controllability weight (CW), and then multiplying the weighted perception factors and the weighted controllability factors together according to Equation 1:

[0058] ORS = (PF 1 *PW 1 ) X (PF 2 *PW 2 )…(PF N *PW N ) X (CF 1 *CW 1 ) X

[0059] (CF 2 *CW 2 )…(CF N *CW N )(1)

[0060] In an alternative example, the weighted perception factors and the weighted controllability factors can be summed together according to Equation 2:

[0061] ORS = PF 1 *PW 1 + PF 2 *PW 2 …PF N *PW N + CF 1 *CW 1 + CF 2*CW 2 …

[0062] CF N *CW N (2)

[0063] In one example, the perception weight and the controllability weight can be determined empirically. The perception weight and the controllability weight can also be selected based on the relative importance of each factor, and the relative importance can also be determined empirically through testing. For example, test data on how much time is required for a driver to re-engage can be collected for multiple drivers in different situations for different factors (such as road type, path confidence (e.g., may be in different environmental conditions), etc.). These factors can then be weighted accordingly in the operator re-engagement score algorithm.

[0064] Figure 2 FIG. 200 is a process flow diagram showing an example process for determining an operator re-engagement score. Process 200 can be implemented in computer 104 included in vehicle 102. Process 200 includes a plurality of blocks that can be executed in the order shown. Alternatively or additionally, process 200 can include fewer blocks, or can include blocks executed in a different order.

[0065] Process 200 can start at block 202, such as in response to receiving data from, for example, block 302 of process 300 to adjust the operator re-engagement timing policy using perception and controllability contexts ( Figure 3A ).

[0066] At block 202, computer 104 quantifies a plurality of perception factors, and at block 204, computer 104 quantifies a plurality of controllability factors, as explained above.

[0067] At block 206, computer 104 multiplies each of the plurality of perception factors by a corresponding perception weight, and at block 208, computer 104 multiplies each of the plurality of controllability factors by a corresponding controllability weight.

[0068] At block 210, computer 104 multiplies the weighted perception factors and the weighted controllability factors together.

[0069] At block 212, computer 104 outputs an operator re-engagement score for adjusting, for example, the operator re-engagement timing policy in process 300.

[0070] In some examples, the operator re-engagement delay can be adjusted based on the operator's engagement history (e.g., reaction time of placing their hands on the steering wheel). The system can build a historical profile regarding both the frequency at which the operator must be notified to re-engage the steering wheel and the speed at which the operator responds. Responsive operators (e.g., infrequent notifications, fast re-engagement) can have an extended maximum delay value. Less-responsive operators (e.g., frequent notifications, slow re-engagement) can have a shortened maximum delay value.

[0071] Many operator identification systems are key fob-based (e.g., key fob or phone). These systems typically do not account for operators sharing keys. In one example, when adjusting the operator re-engagement delay based on the operator's engagement history, biometric verification can be used to confirm the operator's identity. Using the vehicle's operator-facing camera, a face recognition check can be performed to access the correct operator profile. Alternatively, for example, voice recognition can be used via the HMI to prompt the operator to confirm their identity.

[0072] An operator may attempt to deceive the HOOD system by means of an artificial torque source (such as adding weight to the steering wheel). Deception detection of hands on the steering wheel can be performed by observing the lack of change in wheel torque input and / or by means of the operator-facing camera and node model discussed above. For example, when deception is detected, the delay can be set to the minimum delay or zero. If an operator has a history of deceiving the system, their maximum delay time can be shortened.

[0073] In some examples, re-execution learning can be used to intelligently model the operator re-engagement score. A base model can be deployed to infer whether the re-engagement delay will expire, and then a second model can be deployed to infer how long re-engagement will take. These models are then rewarded by the HOOD system. A positive reward (e.g., time of re-execution inference) is given to minimize the difference between the inferred time of a predicted event and the actual time.

[0074] Figure 3A FIG. 300 is a process flow diagram showing an example process for using sensed and controllability contexts to adjust an operator re-engagement timing policy. Process 300 can be implemented in computer 104 included in vehicle 102. Process 300 includes a number of blocks that can be executed in the order shown. Alternatively or additionally, process 300 can include fewer blocks, or can include blocks executed in a different order.

[0075] For example, process 300 may begin at block 302, such as in response to vehicle 102 being placed in an on state or in a "driving" state to operate on a road. In one example, process 300 may begin at block 302, such as in response to detecting that an operator's hand is not in contact with the steering wheel.

[0076] At block 302, computer 104 receives data including sensor data from, for example, vehicle sensors 108, and the data may be used to determine a plurality of perception factors and a plurality of controllability factors. Sensors may include cameras, accelerometers, GPS, radar sensors, scanning lidar, lidar, and torque sensors, to name just a few examples.

[0077] At block 200( Figure 2 ), computer 104 determines an operator re-engagement score based on data from block 302 including perception factors and controllability factors. The perception factors and controllability factors are quantified based on the data and weighted with corresponding weighting factors. Then the weighted perception factors and weighted controllability factors are multiplied together to determine the operator re-engagement score.

[0078] At decision block 304, computer 104 determines whether deception has been detected. In response to an indication that deception has been detected, the operator re-engagement delay is set to a minimum delay at block 306. Otherwise, process 300 proceeds to block 308.

[0079] At block 308, computer 104 adjusts the operator re-engagement delay based on the re-engagement score. The operator re-engagement delay may be adjusted proportionally to the change in the re-engagement score between a minimum delay and a maximum delay. For example, the operator engagement delay may be multiplied by the re-engagement score from block 200. In another example, if the minimum delay is 15 seconds and the maximum delay is 30 seconds and the re-engagement score is determined to be 50 on a scale of, for example, 0 - 100, the operator re-engagement delay may be increased from 15 seconds to approximately 22.5 seconds, which is 50% of the delay range.

[0080] At decision block 310, computer 104 determines whether the operator re-engagement delay has expired. If the delay has expired, vehicle components are actuated at block 312. Otherwise, the process returns to decision block 310 until the delay expires.

[0081] At block 312, computer 104 actuates vehicle components when the operator re-engagement delay expires. The vehicle components may be operator re-engagement indicators, such as auditory or visual indicators and / or tactile indicators. In one example, the vehicle components may be the vehicle braking system and / or the steering system.

[0082] After block 312, process 300 ends.

[0083] Figure 3B FIG. 320 is a process flow diagram illustrating an alternative example process for using sensed and controllability contexts to adjust an operator re-engagement timing strategy. Process 320 may be implemented in computer 104 included in vehicle 102. Process 320 includes a plurality of blocks that may be executed in the order shown. Alternatively or additionally, process 320 may include fewer blocks, or include blocks executed in a different order.

[0084] For example, process 320 may begin at block 322, such as in response to vehicle 102 being placed in an on state or in a “driving” state to operate on a road. In one example, process 320 may begin at block 322, such as in response to detecting that an operator's hand is not in contact with the steering wheel.

[0085] At block 322, computer 104 receives data including sensor data from, for example, vehicle sensors 108, which may be used to determine a plurality of sensed factors and a plurality of controllability factors. Sensors may include cameras, accelerometers, GPS, radar sensors, scanning lidar, lidar, and torque sensors, to name just a few examples.

[0086] At block 200( Figure 2 ), computer 104 determines an operator re-engagement score based on data from block 322 including sensed factors and controllability factors. The sensed factors and controllability factors are quantified based on the data and weighted with corresponding weighting factors. The weighted sensed factors and weighted controllability factors are then multiplied together to determine the operator re-engagement score.

[0087] At decision block 324, computer 104 determines whether the re-engagement score from block 200 is less than a threshold. If the re-engagement score is less than the threshold, the operator re-engagement delay is set to zero at block 326, and the process proceeds to decision block 332. Otherwise, process 320 proceeds to decision block 328. In some examples, the threshold is determined empirically through testing and / or simulation.

[0088] At decision block 328, computer 104 determines whether spoofing has been detected. In response to an indication that spoofing has been detected, the operator re-engagement delay is set to a minimum delay at block 330. Otherwise, process 320 proceeds to decision block 332.

[0089] At decision block 332, computer 104 determines whether the operator re-engagement delay has expired. If the delay has expired, vehicle components are actuated at block 334. Otherwise, the process returns to decision block 332 until the delay expires.

[0090] At block 334, computer 104 actuates a vehicle component when an operator re-engagement delay expires. The vehicle component can be an operator re-engagement indicator, such as an audible or visual indicator and / or a tactile indicator. In one example, the vehicle component can be a vehicle braking system and / or a steering system.

[0091] After block 334, process 320 ends.

[0092] The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0093] The present disclosure has been described in an illustrative manner, and it is to be understood that the terms used are of a descriptive nature and not of a limiting nature. Given the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure may be practiced in other ways than specifically described.

[0094] In the drawings, like reference numerals indicate like elements. Additionally, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it is to be understood that while the steps of such processes, etc. have been described as occurring in a particular order, such processes may be practiced with the steps being performed in an order other than that described herein, unless otherwise stated or apparent from the context. Also, it is to be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the processes herein is provided for the purpose of illustrating certain examples and should in no way be construed as limiting the claims.

[0095] The adjectives first and second are used as identifiers throughout this document and are not intended to denote importance, order, or quantity unless otherwise expressly stated.

[0096] The term exemplary is used herein in the sense of representing an example; for example, a reference to an exemplary widget should be construed as referring only to an example of a widget.

[0097] The use of "responsive to," "based on," and "when determining..." herein indicates a causal relationship and not merely a temporal relationship.

[0098] Computer-executable instructions can be compiled or interpreted by computer programs created using a variety of programming languages and / or technologies, including but not limited to Java, C, C++, Visual Basic, Java Script, Perl, Python, HTML, etc., either alone or in combination. Generally, a processor (e.g., a microprocessor) receives instructions, for example, from a memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using a variety of computer-readable media. Files in a networked device are typically a collection of data stored on a computer-readable medium (such as a storage medium, random access memory, etc.). A computer-readable medium includes any medium that participates in providing data (e.g., instructions) that can be read by a computer. Such media can take many forms, including but not limited to non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including fiber optics, wires, wireless communication, including internal components that make up the system bus coupling to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0099] According to the present invention, a system is provided that has a computer including a processor and a memory, the memory including instructions executable by the processor to: determine an operator re-engagement score based on sensor data including a plurality of perception factors and a plurality of controllability factors; adjust an operator re-engagement delay based on the operator re-engagement score; and actuate a vehicle component when the operator re-engagement delay expires.

[0100] According to one embodiment, the instructions for determining the operator re-engagement score include instructions for: multiplying each of the plurality of perception factors by a corresponding perception weight, and then multiplying the weighted perception factors together.

[0101] According to one embodiment, the perception factors include one or more of path confidence, road type, and adjacent vehicles.

[0102] According to one embodiment, the instructions for determining the operator re-engagement score include instructions for: multiplying each of the plurality of controllability factors by a corresponding controllability weight, and then multiplying the weighted controllability factors together.

[0103] According to one embodiment, the controllability factors include one or more of lateral acceleration, road curvature, speed relative to a limit, and operator engagement.

[0104] According to one embodiment, instructions for adjusting an operator re - engagement latency include instructions for: varying the operator re - engagement latency between a minimum latency and a maximum latency in proportion to a change in an operator re - engagement score.

[0105] According to one embodiment, instructions for adjusting an operator re - engagement latency include instructions for: setting the operator re - engagement latency to zero when the operator re - engagement score is below a threshold.

[0106] According to one embodiment, the instructions further include instructions for: receiving an indication of detected spoofing and, in response to the indication, setting the operator re - engagement latency to the minimum latency.

[0107] According to one embodiment, instructions for determining an operator re - engagement score include instructions for: multiplying each of a plurality of perception factors by a corresponding perception weight; multiplying each of a plurality of controllability factors by a corresponding controllability weight; and multiplying the weighted perception factors together with the weighted controllability factors.

[0108] According to one embodiment, the perception weights and the controllability weights are determined empirically.

[0109] According to one embodiment, instructions for adjusting an operator re - engagement latency include instructions for: varying the operator re - engagement latency between a minimum latency and a maximum latency in proportion to a change in an operator re - engagement score.

[0110] According to one embodiment, the vehicle component is an operator re - engagement indicator.

[0111] According to one embodiment, the vehicle component is a vehicle brake.

[0112] According to the present invention, a method includes: determining an operator re - engagement score based on sensor data including a plurality of perception factors and a plurality of controllability factors; adjusting an operator re - engagement latency based on the operator re - engagement score; and actuating a vehicle component when the operator re - engagement latency expires.

[0113] In one aspect of the present invention, determining the operator re - engagement score includes: multiplying each of a plurality of perception factors by a corresponding perception weight and then multiplying the weighted perception factors together.

[0114] In one aspect of the present invention, determining the operator re - engagement score includes: multiplying each of a plurality of controllability factors by a corresponding controllability weight and then multiplying the weighted controllability factors together.

[0115] In one aspect of the present invention, adjusting an operator re-engagement latency includes varying the operator re-engagement latency between a minimum latency and a maximum latency in proportion to a change in a re-engagement fraction.

[0116] In one aspect of the present invention, a vehicle component is an operator re-engagement indicator.

[0117] In one aspect of the present invention, the method includes receiving an indication of detected spoofing and setting a latency to a minimum latency in response to the indication.

[0118] In one aspect of the present invention, adjusting an operator re-engagement latency includes setting the latency to zero when the re-engagement fraction is below a threshold.

Claims

1. A system, comprising: A computer comprising a processor and a memory, the memory comprising instructions executable by the processor to: determining an operator reengagement score based on sensor data including a plurality of perception factors and a plurality of controllability factors; adjusting an operator re-engagement delay based on the operator re-engagement score; and Upon expiration of the operator reengagement delay, actuating a vehicle component.

2. The system of claim 1, wherein the instructions for determining the operator re-engagement score include instructions for multiplying each of the plurality of perception factors by a corresponding perception weight, and then multiplying the weighted perception factors together.

3. The system of claim 2, wherein the perception factor comprises one or more of path confidence, road type, and neighboring vehicles.

4. The system of claim 1 , wherein the instructions for determining the operator re-engagement score include instructions for multiplying each of the plurality of controllability factors by a corresponding controllability weight, and then multiplying the weighted controllability factors together.

5. The system of claim 4, wherein the controllability factors include one or more of lateral acceleration, road curvature, speed relative to a limit, and operator involvement.

6. The system of claim 1 , wherein the instructions for adjusting the operator re-engagement delay comprise instructions for changing the operator re-engagement delay between a minimum delay and a maximum delay in proportion to a change in the operator re-engagement score.

7. The system of claim 1, wherein the instructions for adjusting the operator re-engagement delay include instructions for setting the operator re-engagement delay to zero when the operator re-engagement score is below a threshold.

8. The system of claim 7, wherein the instructions further comprise instructions for receiving an indication that fraud was detected, and setting the operator re-engagement delay to the minimum delay in response to the indication.

9. The system of claim 1 , wherein the instructions for determining the operator re-engagement score include instructions for: multiplying each of the plurality of perceptual factors by a corresponding perceptual weight; multiplying each of the plurality of controllability factors by a corresponding controllability weight; and The weighted perceptual factor and the weighted controllability factor are multiplied together.

10. The system of claim 9, wherein the perception weight and the controllability weight are determined empirically.

11. The system of claim 9, wherein the instructions for adjusting the operator re-engagement delay include instructions for changing the operator re-engagement delay between a minimum delay and a maximum delay in proportion to a change in the operator re-engagement score.

12. The system of any one of claims 1 to 11, wherein the vehicle component is an operator re-engagement indicator.

13. The system of claim 1, wherein the vehicle component is a vehicle brake.

14. A method comprising: determining an operator reengagement score based on sensor data including a plurality of perception factors and a plurality of controllability factors; adjusting an operator re-engagement delay based on the operator re-engagement score; as well as Upon expiration of the operator reengagement delay, actuating a vehicle component.

15. The method of claim 14, wherein adjusting the operator re-engagement delay comprises: The operator re-engagement delay is varied between a minimum delay and a maximum delay in proportion to a change in the operator re-engagement score.

Images