Vehicle system and method for autonomous operation using driver attention

Abstract

Vehicles and related systems and methods for controlling vehicles in autonomous operating modes are provided. One method involves a controller associated with a vehicle: identifying a visual attention state associated with a driver of the vehicle based at least in part on output of an imaging device on the vehicle; determining a driver lane preference corresponding to a neighboring lane of a current travel lane of the vehicle relative to the vehicle in a direction of the visual attention based at least in part on the visual attention state; adjusting a priority associated with the neighboring lane corresponding to the driver lane preference; and autonomously operating one or more actuators on the vehicle to initiate maneuvering the vehicle from the current lane in a manner influenced by the adjusted priority associated with the neighboring lane.

Description

Technical Field

[0001] The technology field generally relates to vehicle systems, and more specifically to the autonomous operation of vehicles, which uses the driver’s visual attention to increase confidence in the execution of maneuvers. Background Technology

[0002] Autonomous vehicles are vehicles capable of sensing their environment and navigating with little or no user input. Autonomous vehicles use sensing devices such as radar, lidar, and image sensors to perceive their environment. Autonomous vehicle systems also use information from Global Positioning System (GPS) technology, navigation systems, vehicle-to-vehicle communication, vehicle-to-infrastructure technology, and / or drive-by-wire systems to navigate the vehicle.

[0003] Vehicle automation has been categorized into numerical levels ranging from Level 0 (corresponding to non-automation with complete human control) to Level 5 (corresponding to full automation without human control). Various automated driver assistance systems (such as cruise control, adaptive cruise control, and parking assistance systems) correspond to lower levels of automation, while truly "driverless" vehicles correspond to higher levels of automation.

[0004] Because of the numerous variables present in real-world environments, autonomous vehicle control systems may encounter environments or scenarios requiring assistance. For example, they may encounter traffic, road conditions, and other obstacles or scenarios that impair autonomous operation. In lower levels of automation (e.g., Level 3 or below), in some cases, such scenarios may require manual control or operation of the vehicle by the driver or other vehicle occupants, placing a burden on them and contradicting the intent of automation to some extent. Therefore, it is desirable to provide vehicle control systems and methods that can autonomously analyze scenarios and arrive at satisfactory solutions on how to autonomously operate the vehicle while reducing the driver's burden to improve the user experience. Other desirable features and characteristics of the invention will become apparent from the following detailed description and appended claims, taking into account the accompanying drawings and the foregoing technical and background information. Summary of the Invention

[0005] Vehicle apparatus and related methods for controlling a vehicle in an autonomous operating mode are provided. One method for controlling a vehicle in an autonomous operating mode involves a controller associated with the vehicle: identifying a visual attention state associated with the driver of the vehicle based at least in part on the output of an imaging device on the vehicle; determining a driver lane preference corresponding to an adjacent lane in the visual attention direction relative to the vehicle's current driving lane, the visual attention direction corresponding to the visual attention state, based at least in part on the visual attention state; adjusting the priority associated with the adjacent lane in accordance with the driver lane preference; and autonomously operating one or more actuators on the vehicle to initiate vehicle maneuvering from the current lane in a manner influenced by the adjusted priority associated with the adjacent lane.

[0006] On one hand, identifying the visual attention state involves identifying the driver's visual attention directed towards the area associated with one side of the current driving lane in the direction of visual attention; determining the driver's lane preference involves identifying the adjacent lane on the side of the current driving lane in the direction of visual attention as the preferred lane; adjusting priority involves increasing the priority associated with the adjacent lane, thereby generating an increased priority associated with the adjacent lane; and autonomously operating one or more actuators involves autonomously operating one or more actuators to initiate a lane change from the current lane to the adjacent lane based on the increased priority. On another hand, identifying the driver's visual attention directed towards the area associated with one side of the current driving lane involves identifying the driver's visual attention directed towards the rearview mirror on the side of the vehicle in the direction of visual attention. On yet another hand, autonomously operating one or more actuators to initiate a lane change from the current lane to the adjacent lane based on the increased priority involves providing instructions to the motion planning module to generate motion planning to execute the pending lane change in the direction of visual attention.

[0007] On one hand, adjusting priority involves reducing the priority associated with the adjacent lane when the visual attention direction corresponds to the first side opposite the second side of the current lane corresponding to the adjacent lane, thereby generating a reduced priority associated with the adjacent lane; and autonomously operating one or more actuators involves autonomously operating one or more actuators to delay a lane change from the current lane to the adjacent lane according to the reduced priority. On the other hand, adjusting priority involves reducing the priority associated with the adjacent lane when the visual attention direction corresponds to the first side opposite the second side of the current lane corresponding to the adjacent lane, thereby generating a reduced priority associated with the adjacent lane; and autonomously operating one or more actuators involves autonomously operating one or more actuators to initiate a lane change from the current lane to the second adjacent lane on the first side of the current lane according to the reduced priority associated with the adjacent lane. On the other hand, autonomously operating one or more actuators involves autonomously operating one or more actuators to cancel or delay a pending lane change. On the other hand, determining the visual attention state involves determining at least one of the frequency and duration of visual attention in the visual attention direction within the previous time window; determining the driver's lane preference involves determining an adjacent lane as a preferred lane when at least one of the frequency and duration of visual attention in the visual attention direction within the previous time window is greater than a threshold; and adjusting the priority involves increasing the priority associated with the adjacent lane in response to determining that the adjacent lane is a preferred lane, thereby generating an increased priority associated with the adjacent lane.

[0008] In one or more embodiments, a vehicle is provided, the vehicle including an imaging device; one or more actuators on the vehicle; and a controller coupled to the imaging device and the one or more actuators. The controller, via a processor, identifies a visual attention state associated with the driver of the vehicle based at least in part on the output of the imaging device; determines a driver lane preference corresponding to an adjacent lane in the visual attention direction relative to the vehicle's current driving lane, the visual attention direction corresponding to the visual attention state; adjusts the priority associated with the adjacent lane in accordance with the driver lane preference; and autonomously operates the one or more actuators to initiate vehicle maneuvering from the current lane in a manner influenced by the adjusted priority associated with the adjacent lane. In one aspect, the imaging device includes a camera oriented to capture an image of the driver while the driver is maneuvering the vehicle. In another aspect, the visual attention state is the driver's visual attention directed toward an area associated with one side of the current driving lane in the visual attention direction, and the driver lane preference is an adjacent lane on the side of the current driving lane in the visual attention direction. In one or more embodiments, the area is a rearview mirror on the side of the vehicle in the visual attention direction.

[0009] In one or more embodiments, a non-transitory computer-readable medium having executable instructions stored thereon is also provided. When executed by a processor, the instructions cause the processor to: identify a visual attention state associated with the driver of the vehicle, at least in part based on the output of an imaging device on the vehicle; determine a driver lane preference corresponding to an adjacent lane in a visual attention direction relative to the vehicle's current lane of travel, at least in part based on the visual attention state, the visual attention direction corresponding to the visual attention state; adjust the priority associated with the adjacent lane in accordance with the driver lane preference; and autonomously operate one or more actuators on the vehicle to initiate vehicle maneuvering from the current lane in a manner influenced by the adjusted priority associated with the adjacent lane.

[0010] On one hand, identifying the visual attention state involves identifying the driver's visual attention directed towards an area associated with the side of the current driving lane in the direction of visual attention; determining the driver's lane preference involves identifying the adjacent lane on the side of the current driving lane in the direction of visual attention as the preferred lane; adjusting priority involves increasing the priority associated with the adjacent lane; and autonomously operating one or more actuators involves autonomously operating one or more actuators to initiate a lane change from the current lane to an adjacent lane based on the increased priority. In one or more embodiments, the area is a rearview mirror on the side of the vehicle in the direction of visual attention.

[0011] On the other hand, autonomously operating one or more actuators to initiate a lane change from the current lane to an adjacent lane according to an adjusted priority involves providing instructions to the motion planning module to generate motion planning to execute the pending lane change in the visual attention direction. In yet another aspect, adjusting the priority involves reducing the priority associated with the adjacent lane when the visual attention direction corresponds to a first side opposite the second side of the current lane corresponding to the adjacent lane; and autonomously operating one or more actuators involves autonomously operating one or more actuators to delay the lane change from the current lane to the adjacent lane according to the reduced priority. On yet another aspect, adjusting the priority involves reducing the priority associated with the adjacent lane when the visual attention direction corresponds to a first side opposite the second side of the current lane corresponding to the adjacent lane; and autonomously operating one or more actuators involves autonomously operating one or more actuators to initiate a lane change from the current lane to a second adjacent lane on the first side of the current lane according to the reduced priority associated with the adjacent lane. On yet another aspect, autonomously operating one or more actuators involves autonomously operating one or more actuators to cancel or delay the pending lane change. In another aspect, determining the visual attention state involves determining at least one of the frequency and duration of visual attention in the visual attention direction within the previous time window; determining the driver's lane preference involves determining an adjacent lane as a preferred lane when at least one of the frequency and duration of visual attention in the visual attention direction within the previous time window is greater than a threshold; and adjusting the priority involves increasing the priority associated with the adjacent lane in response to determining that the adjacent lane is a preferred lane, thereby generating an increased priority associated with the adjacent lane. Attached Figure Description

[0012] Exemplary aspects will be described below with reference to the following figures, wherein like reference numerals denote like elements, and wherein:

[0013] Figure 1 This is a block diagram illustrating an autonomous vehicle control system for a vehicle according to various embodiments;

[0014] Figure 2 It is suitable for various implementation methods Figure 1 A block diagram of the autonomous vehicle control system (ADS) implemented by the vehicle.

[0015] Figure 3 A block diagram of an autonomous vehicle control system according to one or more aspects described herein is depicted, the autonomous vehicle control system including components adapted to... Figure 2 together with ADS Figure 1 The driver monitoring system used in the autonomous vehicle control system;

[0016] Figure 4Depicting the suitability of one or more aspects according to the description herein by Figure 2 ADS in Figure 1 A flowchart of the visual lane priority ranking process implemented in an autonomous vehicle control system; and

[0017] Figures 5 to 6 Depicting one or more aspects according to the description in this article Figure 4 An exemplary scenario of an example implementation of the visual lane priority sorting process. Detailed Implementation

[0018] The following detailed description is merely exemplary in nature and is not intended to limit application or use. Furthermore, it is not intended to be bound by any express or implied theory set forth in the foregoing introduction, summary of the invention, or the following detailed description. As used herein, the term "module" refers individually or in any combination of any hardware, software, firmware, electronic control components, processing logic, and / or processor devices, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0019] Now for reference Figure 1 According to one or more embodiments, the autonomous vehicle control system 100 determines a plan for autonomously operating the vehicle 10 along a route in a manner that takes into account objects or obstacles detected by the onboard sensors 28, 40, as described in more detail below. In this regard, the control module on the vehicle 10 calibrates the different types of onboard sensors 28, 40 relative to each other and / or the vehicle 10, thereby allowing data from those different types of onboard sensors 28, 40 to be spatially or otherwise correlated with each other based on this calibration for object detection, object classification, and the resulting autonomous operation of the vehicle 10.

[0020] like Figure 1 As shown, vehicle 10 typically includes a chassis, body 14, and front wheels 16 and rear wheels 18, which are rotatably coupled to the chassis near corresponding corners of body 14. Body 14 is mounted on chassis and substantially surrounds the components of vehicle 10, and body 14 and chassis may together form a frame.

[0021] In an exemplary embodiment, vehicle 10 is an autonomous vehicle or otherwise configured to support one or more autonomous operating modes, and control system 100 is integrated into vehicle 10 (hereinafter referred to as vehicle 10). In the illustrated embodiment, vehicle 10 is depicted as a passenger car; however, it should be understood that any other means of transportation, including motorcycles, trucks, sports utility vehicles (SUVs), recreational vehicles (RVs), ships, aircraft, etc., may also be used. In an exemplary embodiment, vehicle 10 is a so-called Level 2 automation system. Level 2 system stands for "partial driving automation," referring to a specific driving mode performance where the automated driving system controls steering, acceleration, and braking in a particular scenario, while the driver remains alert and actively supervises the automated driving system and can provide driver support to control the primary driving tasks.

[0022] As shown, vehicle 10 typically includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the wheels 16, 18 according to selectable speed ratios. According to various embodiments, the transmission system 22 may include a graded automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. The braking system 26 is configured to provide braking torque to the wheels 16, 18. In various embodiments, the braking system 26 may include friction brakes, brake-by-wire brakes, regenerative braking systems such as electric motors, and / or other suitable braking systems. The steering system 24 affects the position of the wheels 16, 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 24 may not include a steering wheel.

[0023] Sensor system 28 includes one or more sensing devices 40a to 40n for sensing observable conditions of the external and / or internal environment of vehicle 10. Sensing devices 40a to 40n may include, but are not limited to, radar, lidar, GPS, optical cameras, thermal imagers, ultrasonic sensors, and / or other sensors. Actuator system 30 includes one or more actuator devices 42a to 42n that control one or more vehicle features, such as, but not limited to, propulsion system 20, transmission system 22, steering system 24, and braking system 26. In various embodiments, vehicle features may also include internal and / or external vehicle features, such as, but not limited to, doors, trunk, and cabin features (unnumbered) such as air, music, and lighting.

[0024] Data storage device 32 stores data used for automatically controlling vehicle 10. In various embodiments, data storage device 32 stores a defined map of the navigable environment. In various embodiments, the defined map may be predefined by and obtained from a remote system. For example, the defined map may be assembled by a remote system and transmitted (wirelessly and / or via wired means) to vehicle 10 and stored in data storage device 32. It is understood that data storage device 32 may be part of controller 34, separate from controller 34, or part of controller 34 and a separate system.

[0025] The controller 34 includes at least one processor 44 and a computer-readable storage device or medium 46. The processor 44 may be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among several processors associated with the controller 34, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, any combination thereof, or any device typically used to execute instructions. The computer-readable storage device or medium 46 may include, for example, volatile and non-volatile memory in the form of read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operational variables when the processor 44 is powered off. The computer-readable storage device or medium 46 may be implemented using any of many known memory devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combined memory device capable of storing data, some of which represents executable instructions used by the controller 34 to control the vehicle 10.

[0026] The instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by processor 44, the instructions receive and process signals from sensor system 28, execute logic, calculations, methods, and / or algorithms for automatically controlling components of vehicle 10, and generate control signals to actuator system 30 to automatically control components of vehicle 10 based on logic, calculations, methods, and / or algorithms. Although Figure 1 Only one controller 34 is shown, but embodiments of vehicle 10 may include any number of controllers 34 that communicate and cooperate via any suitable communication medium or combination of communication media to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of vehicle 10.

[0027] In various embodiments, one or more instructions of controller 34 are embodied in control system 100 (e.g., in data storage element 46) that, when executed by processor 44, enable processor 44 to acquire data captured or generated from imaging and ranging equipment 40 and utilize the captured environmental data to determine commands for autonomously maneuvering vehicle 10, as described in more detail below. In one or more exemplary embodiments, data storage element 46 maintains a lookup table of lateral planning information that can be used to determine the corresponding lateral reference trajectory for lateral maneuvering into adjacent lanes. The lateral planning information and the resulting reference lateral trajectory are utilized by processor 44 or otherwise referenced to determine commands for autonomously maneuvering vehicle 10 when the normal vehicle guidance or control scheme supported by processor 44 encounters a time-sensitive lateral maneuver deadline or other time constraints, thus avoiding the need to solve the commanded vehicle path within a limited time period.

[0028] Still referencing Figure 1 In an exemplary embodiment, communication system 36 is configured to wirelessly transmit information to and from other entities 48 via a communication network, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems, and / or personal devices. In an exemplary embodiment, communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels, are also considered within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium range wireless communication channel specifically designed for automotive use, and a corresponding set of protocols and standards.

[0029] The communication network utilized by communication system 36 may include a wireless carrier system, such as a cellular telephone system, comprising multiple cell towers (not shown), one or more mobile switching centers (MSCs) (not shown), and any other networking components required to connect the wireless carrier system to the terrestrial communication system. The wireless carrier system may implement any suitable communication technology, including, for example, digital technologies such as CDMA (e.g., CDMA2000), LTE (e.g., 4G LTE or 5G LTE), GSM / GPRS, or other current or emerging wireless technologies. Alternatively or concurrently, a second wireless carrier system in the form of a satellite communication system may be utilized to provide one-way or two-way communication using one or more communication satellites (not shown) and uplink transmitting stations (not shown), including but not limited to satellite radio services, satellite telephone services, etc. Some implementations may utilize terrestrial communication systems, such as conventional terrestrial telecommunications networks, including the Public Switched Telephone Network (PSTN) for providing hard-wired telephone, packet-switched data communications, and Internet infrastructure. One or more segments of a terrestrial communication network may be implemented using standard wired networks, fiber optic or other optical networks, cable networks, power lines, other wireless networks (such as wireless local area networks (WLANs)) or networks providing broadband wireless access (BWA), or any combination thereof.

[0030] Now for reference Figure 2 According to various implementations, the controller 34 implements an autonomous driving system (ADS) 70. That is, it utilizes suitable software and / or hardware components of the controller 34 (e.g., processor 44 and computer-readable storage device 46) to provide an autonomous driving system 70 for use in conjunction with the vehicle 10, for example, to automatically control various actuators 30, and thereby control the vehicle's acceleration, steering, and braking respectively, without human intervention.

[0031] In various implementations, the instructions of the autonomous driving system 70 can be organized by function or system. For example, such as Figure 2 As shown, the autonomous driving system 70 may include a sensor fusion system 74, a positioning system 76, a guidance system 78, and a vehicle control system 80. It will be understood that, in various embodiments, since this disclosure is not limited to this example, instructions can be organized into any number of systems (e.g., combined, further divided, etc.).

[0032] In various embodiments, sensor fusion system 74 synthesizes and processes sensor data and predicts the presence, location, classification, and / or path of objects and features in the environment of vehicle 10. In various embodiments, sensor fusion system 74 may combine information from multiple sensors (including, but not limited to, cameras, lidar, radar, and / or any number of other types of sensors). In one or more exemplary embodiments described herein, sensor fusion system 74 correlates image data with lidar point cloud data, a vehicle reference frame, or some other reference coordinate system using calibrated transformation parameter values ​​associated with a pair of corresponding cameras and reference frames to correlate lidar points with pixel locations, assign depth to image data, identify objects in one or more of the image data and lidar data, or otherwise synthesize associated image data and lidar data. In other words, sensor outputs from sensor fusion system 74 provided to vehicle control system 80 (e.g., labels of detected objects and / or their positions relative to vehicle 10) reflect, or are otherwise influenced by, calibration and correlation between camera images, lidar point cloud data, etc.

[0033] The positioning system 76 processes sensor data and other data to determine the position of vehicle 10 relative to its environment (e.g., local position relative to a map, precise position relative to a road lane, vehicle heading, speed, etc.). The guidance system 78 processes sensor data and other data to determine the path to be followed by vehicle 10 given current sensor data and vehicle attitude. The vehicle control system 80 then generates control signals for controlling vehicle 10 based on the determined path. In various embodiments, the controller 34 implements machine learning techniques to assist its functions, such as feature detection / classification, obstacle mitigation, route traversal, mapping, sensor integration, and ground condition determination.

[0034] In one or more embodiments, the guidance system 78 includes a motion planning module that generates motion plans for controlling the vehicle as it traverses a route. The motion planning module includes a longitudinal solver module that generates a longitudinal motion planning output for controlling the vehicle's movement along the route in the overall direction of travel, for example, by causing the vehicle to accelerate or decelerate at one or more locations along the route in the future to maintain a desired speed or rate. The motion planning module also includes a lateral solver module that generates a lateral motion planning output for controlling the vehicle's lateral movement along the route to change the overall direction of travel, for example, by causing the vehicle to steer at one or more locations along the route in the future (e.g., to keep the vehicle centered in a lane, change lanes, etc.). The longitudinal and lateral planning outputs correspond to (or planned) path outputs of commands provided to the vehicle control system 80 for controlling the vehicle actuators 30 to move the vehicle 10 along the routes corresponding to the longitudinal and lateral plans.

[0035] During normal operation, the longitudinal solver module attempts to optimize the vehicle speed (or rate) in the direction of travel, the vehicle acceleration in the direction of travel, and the derivative of the vehicle acceleration in the direction of travel (which may alternatively be referred to herein as the vehicle's longitudinal jerk), while the lateral solver module attempts to optimize one or more of the steering angle, the rate of change of the steering angle, and the acceleration or second derivative of the steering angle (which may alternatively be referred to herein as the vehicle's lateral jerk). In this regard, the steering angle may be related to the curvature of the path or route, and any one of the steering angle, the rate of change of the steering angle, and the acceleration or second derivative of the steering angle can be optimized individually or in combination by the lateral solver module.

[0036] In an exemplary embodiment, the longitudinal solver module receives or otherwise obtains the vehicle's current or instantaneous attitude, including the vehicle's current position or orientation, current orientation, current speed or rate, and current acceleration. Using the vehicle's current position or orientation, the longitudinal solver module also retrieves or otherwise obtains route information, including information about the route the vehicle is traveling given its current attitude, plus some additional buffer distance or time period (e.g., the next 12 seconds), such as current and future road slope or inclination, current and future road curvature, current and future lane information (e.g., lane type, boundaries, and other constraints or limitations), and other constraints or limitations associated with the road (e.g., minimum and maximum speed limits, height or weight limits, etc.). The route information can be obtained from, for example, onboard data storage element 32, an online database, or other entities. In one or more embodiments, lateral route information may include a lateral path planning command output by the lateral solver module, wherein the longitudinal solver module and the lateral solver module iteratively derive an optimal driving plan along the route.

[0037] The longitudinal solver module also receives or otherwise obtains current obstacle data relating to the vehicle's route and current attitude, which may include, for example, the location or position, size, orientation or heading, velocity, acceleration, and other characteristics of objects or obstacles near the vehicle or future route. The longitudinal solver module also receives or otherwise obtains longitudinal vehicle constraint data, which characterizes or otherwise defines the vehicle's kinematic or physical capabilities for longitudinal movement, such as, for example, maximum acceleration and maximum longitudinal jerk, maximum deceleration, etc. The longitudinal vehicle constraint data may be vehicle-specific and may be obtained from onboard data storage element 32 or from a networked database or other entities 48, 52, 54. In some embodiments, the longitudinal vehicle constraint data may be dynamically or substantially in real-time calculated or otherwise determined based on the vehicle's current mass, current fuel level on the vehicle, the vehicle's historical or recent performance, and / or potential other factors. In one or more embodiments, the longitudinal vehicle constraint data is calculated or determined relative to the lateral path, lateral vehicle constraint data, and / or determinations made by the lateral solver module. For example, by calculating the maximum longitudinal speed as a function of path curvature and maximum lateral acceleration (which itself may be constrained by occupant preferences or vehicle dynamics), the maximum longitudinal speed can be constrained by path curvature and maximum lateral acceleration at specific locations. In this respect, at locations with relatively high levels of path curvature (e.g., sharp turns), the maximum longitudinal speed can be correspondingly limited to maintain comfortable or achievable lateral acceleration along the curve.

[0038] Using various inputs to the longitudinal solver module, the module optimizes some longitudinal cost variables or combinations thereof (e.g., minimizing travel time, fuel consumption, jerk, etc.) by changing the vehicle's speed or rate from its current posture in a way that ensures the vehicle conforms as closely as possible to longitudinal riding preference information while also adhering to lane boundaries or other route constraints and avoiding collisions with objects or obstacles. The longitudinal solver module calculates or otherwise determines a longitudinal plan (e.g., future planned speed, acceleration, and jerk values ​​as a function of time) for traveling along the route within a prediction range (e.g., 12 seconds). In this regard, in many situations, the resulting longitudinal plan generated by the longitudinal solver module does not violate the user-associated maximum vehicle speed, maximum vehicle acceleration, maximum deceleration, and maximum longitudinal jerk settings, while also conforming to the user-associated follow distance or buffer. That is, in some scenarios, it may be necessary to violate one or more longitudinal riding preference settings to avoid collisions, obey traffic signals, etc. In such cases, the longitudinal solver module may attempt to maintain compliance with as many user-specific longitudinal riding preference settings as possible. Therefore, the resulting longitudinal plan generally conforms to the user's longitudinal riding preference information, but not necessarily strictly.

[0039] In a similar manner, the lateral solver module receives or otherwise obtains current vehicle attitude and relevant route information, as well as obstacle data, to determine lateral driving planning schemes within the prediction range. The lateral solver module also receives or otherwise obtains lateral vehicle constraint data, which characterizes or otherwise defines the vehicle's kinematic or physical capabilities for lateral movement, such as, for example, maximum steering angle or steering angle range, minimum turning radius, maximum rate of change of steering angle, etc. Lateral vehicle constraint data can also be specific to each particular vehicle and can be obtained from onboard data storage element 32 or from a networked database or other entities 48, 52, 54. The lateral solver module may also receive or otherwise obtain user-specific lateral riding preference information, including, for example, user-specific steering rate values ​​or settings (e.g., maximum rate of change of steering angle, maximum acceleration of steering angle, etc.), lateral jerk, etc. Lateral riding preference information may also include user-specific distances or buffers, such as, for example, minimum and / or maximum distances from lane boundaries, minimum lateral buffer or lateral spacing distances between objects or obstacles, etc., and potential other user-specific lane preferences (e.g., preferred driving lanes).

[0040] Using various inputs from the lateral solver module, the lateral solver module calculates or otherwise determines lateral planning for driving along a route at a future location within a certain prediction range (e.g., minimizing deviation from the road center, minimizing path curvature, minimizing lateral jerk, etc.) by adjusting steering angles or wheel angles in a way that ensures the vehicle follows lateral riding preference information as much as possible while also adhering to lane boundaries or other route constraints and avoiding collisions with objects or obstacles.

[0041] During normal operation, the lateral solver module can utilize the longitudinal driving plan from the longitudinal solver module, along with route information and obstacle data, to determine how to steer the vehicle from its current posture within the prediction range, while attempting to conform to lateral riding preference information. In this regard, the resulting longitudinal and lateral driving plans, ultimately output by the motion planning module, conform to as many user riding preferences as possible, while optimizing cost variables and avoiding collisions by altering one or more of the vehicle's rate, acceleration / deceleration (longitudinal and / or lateral), jerk (longitudinal and / or lateral), steering angle, and rate of change of steering angle. The longitudinal driving plan output by the motion planning module includes a series of planned rate and acceleration commands with respect to time for maneuvering the vehicle within the longitudinal prediction range (e.g., rate planning for the next 12 seconds), and similarly, the lateral driving plan output by the motion planning module includes a series of planned steering angles and steering rates with respect to distance or position for steering the vehicle within the lateral prediction range while operating according to the longitudinal driving plan (e.g., steering planning for the next 50 meters). The longitudinal planning output and the lateral planning output are provided to the vehicle control system 80, which can use the vehicle positioning information and adopt its own control scheme to generate control output. The control output adjusts the vehicle positioning information to longitudinal planning and lateral planning by changing the speed and steering commands provided to the actuator 30, thereby changing the speed and steering of the vehicle 10 to simulate or otherwise realize longitudinal planning and lateral planning.

[0042] In an exemplary embodiment, the guidance system 78 supports a manual-free autonomous operation mode, which, when enabled and operated, controls steering, acceleration, and braking to provide lane centering using current sensor data (or obstacle data) provided by the sensor fusion system 74 and the current vehicle attitude provided by the positioning system 76, while attempting to maintain a driver-selected speed and / or following distance (or gap time) relative to other vehicles. In the autonomous operation mode, the guidance system 78 includes or otherwise implements a lane change coordinator that analyzes route information (if available) in addition to analyzing data or other information from the sensor fusion system 74, the positioning system 76, and potentially other modules or systems, to determine whether to initiate and execute a lane change from the current lane to an adjacent lane, for example, based on the presence of slow-moving traffic in the current lane ahead of the vehicle (e.g., to overtake or pass another vehicle), whether the current lane is terminating or merging into an adjacent lane, whether a lane change is needed to maintain travel along the desired route, etc. In this regard, the lane change coordinator can autonomously determine when to initiate a lane change and autonomously configure the lateral solver module and / or motion planning module to generate the corresponding lateral plan to change lanes in the desired manner and provide the lateral plan to the vehicle control system 80, which autonomously generates the corresponding control signal to autonomously control the vehicle actuator 30 to manipulate the vehicle 10 and execute the lane change.

[0043] Now for reference Figure 3 And continue to refer to Figures 1 to 2 In an exemplary embodiment, the vehicle control system 300 includes a driver monitoring system 302 that analyzes image data output by the imaging device 304 to classify or otherwise determine the visual attention state associated with the driver of the vehicle, and analyzes the driver's visual attention state to identify or otherwise determine whether the driver has a lane preference for an alternative lane to the current driving lane. When the driver monitoring system 302 determines that a driver lane preference exists, it identifies or otherwise designates an adjacent lane in the driver's visual attention direction as the preferred lane, and provides a corresponding indication of the preferred lane to the lane change coordination system 306 at the guidance system 308 (e.g., guidance system 78) to influence lateral maneuvering of the vehicle from the current driving lane.

[0044] In an exemplary embodiment, imaging device 304 is implemented as a camera located in the interior compartment or passenger cabin of the vehicle and positioned or otherwise oriented toward the driver's seat to capture video or images of the driver during vehicle operation. For example, imaging device 304 may be integrated or otherwise incorporated into the steering wheel such that the field of view or line of sight of imaging device 304 is aligned with the central axis of the steering wheel to capture an image of the driver's face or head while the driver is seated and looking forward toward the windshield of the vehicle. That is, it should be understood that the subject matter described herein is not limited to any particular type of imaging device 304 or any particular arrangement, configuration, or encapsulation of imaging device 304.

[0045] In an exemplary embodiment, the driver monitoring system 302 is implemented or otherwise realized using at least one processor and a computer-readable storage device or medium capable of storing data or executable instructions, which cause the at least one processor to execute, generate, or otherwise provide a visual attention classification module 310 and a visual lane priority ranking module 312. The visual attention classification module 310 is configured to receive video or other image data output by the imaging device 304 and analyze the image data to identify or otherwise determine the direction in which the driver's head or eyes are looking. It should be understood that any number of different image and video processing techniques can be used to detect or otherwise identify the orientation of a driver's gaze (or its focus), and these techniques are not closely related to this disclosure, and the subject matter described herein is not limited to any particular manner or technique for estimating or determining the orientation of a driver's gaze using captured image data.

[0046] In one or more exemplary embodiments, the visual attention classification module 310 analyzes the direction or orientation of the driver's gaze relative to the imaging device 304 to classify or otherwise assign the driver's visual attention to one of a plurality of defined visual attention states. For example, when the direction or orientation of the driver's gaze indicates that the driver's visual attention is directed toward or otherwise contained within a driver-side area of ​​the vehicle, including the driver-side rearview mirror and the driver-side window, the visual attention classification module 310 may determine that the driver's visual attention direction corresponds to the driver's side of the vehicle (or, in the United States, the left side of the vehicle). Conversely, when the direction or orientation of the driver's gaze indicates that the driver's visual attention is directed toward or otherwise contained within a passenger-side area of ​​the vehicle, including or otherwise contained within the passenger-side rearview mirror and the passenger-side window, the visual attention classification module 310 may determine that the driver's visual attention direction corresponds to the passenger side of the vehicle (or, in the United States, the right side of the vehicle). In addition, other areas of the driver's line of sight or focus can be classified into different visual attention states. For example, when the direction or orientation of the driver's line of sight indicates that the driver's visual attention is directed toward an area including the rearview mirror or otherwise contained in that area, the visual attention classification module 310 can determine that the driver's visual attention direction corresponds to the rear of the vehicle, while other forward-looking areas (e.g., the dashboard, center console, etc.) can be assigned different visual attention classifications.

[0047] The visual lane priority ranking module 312 is coupled to the visual attention classification module 310 to receive a marker of the driver's visual attention state and dynamically determine the driver's lane preference and dynamically adjust the priorities associated with different potential driving lanes to reflect the driver's current lane preference substantially in real time. For example, the visual lane priority ranking module 312 can monitor, track, or otherwise determine the frequency or number of occurrences of a specific visual attention state within a previous time window and / or the duration of occurrence of a specific visual attention state within a previous time window. In this regard, when the driver's visual attention state corresponding to the driver's visual attention direction toward a specific side of the vehicle occurs more than a threshold number of times and / or continues to exceed a threshold duration within a previous monitoring window, the visual lane priority ranking module 312 can identify the adjacent lane on the side of the vehicle corresponding to the driver's visual attention direction as the driver's preferred driving lane and automatically increase the priority of the adjacent lane assigned to that direction, because the driver's visual attention is directed toward that lane. At the same time, the visual lane priority ranking module 312 can automatically decrease the priority of the relatively adjacent lane on the other side of the vehicle opposite to the driver's visual attention direction to reflect the driver's visual attention being directed away from that lane.

[0048] Still referencing Figure 3The visual lane priority ranking module 312 outputs or otherwise provides a marker of the current visual attention state and the corresponding lane priority ranking or preference to the lane change coordination system 306 at the guidance system 308. The lane change coordination system 306 uses the driver's current visual attention state within the previous monitoring window and the lane priority ranking influenced by the driver's visual attention state to determine whether to initiate a lane change or maintain the vehicle in the current lane. In this respect, the driver's visual attention state and lane priority ranking can be used to enhance automatic lane change determination and increase the confidence of desired or undesired automatic lane changes based on whether the driver's visual attention state and / or lane preference are consistent with automatic lane change determinations based on other variables. For example, as described above, the lane change coordination system 306 of the guidance system 308 can receive data or other information indicating nearby traffic or other obstacles from a vehicle-associated sensor system 314 (e.g., sensor fusion system 74) to automatically determine whether the vehicle should change lanes to overtake or otherwise pass a slower-moving vehicle in front of the main vehicle to maintain a pre-set speed (or cruising speed) that may be defined by the driver or otherwise desired. Additionally, the lane change coordination system 306 can receive data or information from the positioning system 316 (e.g., positioning system 76) indicating the current vehicle posture, which can be combined with route information 318 (if available) to determine whether the vehicle should change lanes to maintain its desired route.

[0049] In one or more exemplary embodiments, when the driver's visual attention direction corresponds to the same side of the vehicle as the automatic lane change, the lane change coordination system 306 utilizes the driver's current visual attention state to further increase the priority associated with performing the automatic lane change. For example, based on sensor data from sensor system 314 indicating that a slower-moving vehicle in the path ahead of the primary vehicle is within a threshold distance of the primary vehicle, the lane change coordination system 306 can automatically determine to initiate a lane change from the current lane to the adjacent lane on the driver's side of the primary vehicle. When the driver's visual attention direction corresponds to the driver's side of the primary vehicle and / or the adjacent lane on the driver's side of the primary vehicle is assigned a higher priority by the visual lane priority ranking module 312, the lane change coordination system 306 determines that the driver's visual attention direction and the automatic lane change direction are consistent, and accordingly provides commands, signals, or other instructions (e.g., by setting an acceleration flag to a specific value) to the motion planning module to expedite the implementation of the automatic lane change from the current lane to the adjacent lane on the driver's side. In response, the motion planning module can automatically determine a motion plan configured to laterally maneuver the vehicle from the current lane to an adjacent lane on the driver's side, and provide the motion plan (or corresponding command signal) to the vehicle control system 320 (e.g., vehicle control system 80), which autonomously operates one or more actuators on the vehicle to initiate the maneuver from the current lane to the adjacent lane on the driver's side. In this regard, when the lane change coordination system 306 indicates a desire to accelerate the lane change based on the driver's visual attention confirmation of the automatic lane change, in some embodiments, the guidance system 308 and / or the vehicle control system 320 can automatically initiate an autonomous lane change operation without waiting for further driver confirmation or providing the user with a pending lane change notification, because the driver's visual attention aligns with or otherwise matches the automatic lane change.

[0050] On the other hand, in other scenarios, when the driver's visual attention direction does not match or otherwise correspond to the side of the vehicle that is the same as the automatic lane change, the lane change coordination system 306 can utilize the driver's current visual attention state to change, delay, cancel, or otherwise reduce the priority associated with performing the automatic lane change. For example, when the lane change coordination system 306 automatically determines to initiate an automatic lane change from the current lane to the adjacent lane on the driver's side of the main vehicle, but the driver's visual attention direction corresponds to the adjacent lane on the relative passenger side of the vehicle and / or the passenger side of the main vehicle being assigned a higher priority by the visual lane priority ranking module 312, the lane change coordination system 306 determines that the driver's visual attention direction and the automatic lane change direction do not match or are inconsistent. In some embodiments, the lane change coordination system 306 utilizes the driver's visual attention direction to overtake or otherwise enhance the automatic lane change to a preferred or higher priority lane derived from the driver's visual attention state, for example, by modifying the automatic lane change to the adjacent lane on the passenger side of the main vehicle instead of the driver's side lane. In other embodiments, the lane change coordination system 306 may reduce the priority associated with automatic lane change or otherwise delay the execution of automatic lane change, for example by requiring the driver to confirm the pending automatic lane change to the driver's side when the driver's attention is not directed toward the driver's side of the primary vehicle or is inconsistent with the driver's side of the primary vehicle.

[0051] In some implementations, when no pending automatic lane change is currently available, the lane change coordination system 306 utilizes the driver's visual attention state and corresponding lane priority ranking or preference to influence subsequent automatic lane changes that may be determined based on adjacent traffic, route information, etc. For example, when the driver's visual attention state indicates that the driver's visual attention is preferentially directed towards the passenger side of the vehicle, the priority associated with the adjacent lane on the passenger side can be increased, such that in response to a subsequent determination to initiate an automatic lane change to overtake or pass a slower-moving vehicle in front of the main vehicle, the lane change coordination system 306 can determine to initiate an automatic lane change from the current driving lane to the adjacent lane on the passenger side of the vehicle instead of defaulting to the adjacent lane on the driver side of the vehicle. In this regard, the driver's visual attention state and corresponding lane preference can be used to dynamically increase or decrease the relative priority of different potential driving lanes and / or dynamically increase or decrease different timers or thresholds used to initiate automatic lane changes starting from the current driving lane.

[0052] For example, if the driver's visual attention is directed toward a specific side of the vehicle for more than a threshold duration and / or more than a threshold number of times during the previous monitoring window, the lane change coordination system 306 may gradually increase the priority associated with the adjacent lane on that side of the vehicle and / or gradually decrease the timer or threshold used to initiate a lane change to that side of the vehicle to increase the likelihood of initiating an automatic lane change from the current lane to the adjacent lane on that side of the vehicle. In this regard, some implementations of the lane change coordination system 306 utilize the driver's visual attention state and the corresponding visual lane preference or priority as input variables. These input variables fuse or otherwise combine the output of the driver monitoring system 302 with the outputs from other onboard systems or components analyzing different aspects of vehicle operation and the surrounding environment, and provide corresponding lane preferences or priorities as a weighted combination of different inputs to finally determine whether to initiate an automatic lane change in a specific direction.

[0053] It should be noted that Figure 3 A simplified representation of the vehicle control system 300 is depicted for illustrative purposes and is not intended to be limiting. In this respect, although Figure 3 The driver monitoring system 302 is depicted as a separate or independent system distinct from the guidance system 308. However, in other embodiments, the driver monitoring system 302 (or portions thereof) may be incorporated into, integrated into, or otherwise combined with the guidance system 308 or another system on the vehicle. For example, the visual lane priority module 312 may be implemented by or on the guidance system 308, and / or the visual attention classification module 310 may be implemented by or on the sensor system 314 (e.g., where image data 304 output by an imaging device is input to the sensor system 314). Furthermore, the lane change coordination system 306 may include any number of different sub-components or subsystems configurable to determine whether automatic lane changes are likely to be desired and to assign lane priorities to automatic lane changes. In such implementations, the lane change coordination system 306 arbitrates outputs from different subcomponents or subsystems in accordance with the driver visual attention state outputs from the driver monitoring system 302 and / or the visual lane priority module 312, thereby determining whether to initiate an automatic lane change in a manner that takes into account the driver's visual lane preference and any number of other different factors or variables during vehicle operation (e.g., route information, lane closure or lane termination, road construction, traffic, detected objects or obstacles, driver preferences or other vehicle settings, etc.).

[0054] Figure 4An exemplary embodiment of a visual lane priority process 400 suitable for implementation by one or more control modules on a vehicle (e.g., by a driver monitoring system 302 in conjunction with an ADS 70 supported by a controller 34 in vehicle 10) is described, which autonomously operates one or more actuators on the vehicle to change lanes in a manner influenced by driver lane preferences derived from analysis of the driver's visual attention state. For illustrative purposes, the following description may refer to the above in conjunction with... Figures 1 to 3 The components described. Although the various parts of the visual lane priority process 400 can be performed by different components of the vehicle system, for illustrative purposes, this paper can describe the subject primarily in the context of the visual lane priority process 400 being performed by the guidance system 78, 308 of the ADS 70, which is mainly implemented by the driver monitoring system 302 and the controller 34 associated with the vehicle 10.

[0055] In an exemplary embodiment, the visual lane prioritization process 400 begins at 402 by identifying or otherwise determining the driver's visual attention state. (As stated above...) Figure 3 As described in the context of the exemplary embodiment, in the driver monitoring system 302, the visual attention classification module 310 continuously analyzes the image data output by the imaging device 304 to classify the orientation of the driver's line of sight or focus relative to the position of the imaging device 304 into a specific visual attention state corresponding to the direction of the driver's visual attention. For example, when the visual attention classification module 310 determines that the driver's visual attention is directed towards the driver's side rearview mirror or otherwise further directed towards the driver's side of the vehicle (e.g., outside the driver's side window), the visual attention classification module 310 can classify the driver's visual attention as the adjacent lane on the driver's side of the vehicle. On the other hand, when the visual attention classification module 310 determines that the driver's visual attention is directed towards the passenger's side rearview mirror or otherwise further directed towards the passenger side of the vehicle (e.g., outside the passenger's side window), the visual attention classification module 310 can classify the driver's visual attention as the adjacent lane on the passenger side of the vehicle. Similarly, when the visual attention classification module 310 determines that the driver's visual attention is directed towards the rearview mirror, center console, dashboard, or other areas within the vehicle, the visual attention classification module 310 can classify the driver's visual attention into a corresponding visual attention state associated with that area. In this regard, in some embodiments, specific areas can be assigned to visual attention states for negating, canceling, or otherwise controlling automatic lane changes, as described in more detail below.

[0056] After determining the driver's visual attention state, the visual lane priority process 400 identifies or otherwise determines a driver lane preference metric at 404 based on the driver's visual attention state. In an exemplary embodiment, the visual lane priority process 400 identifies or otherwise determines the driver lane preference metric based on the driver's visual attention state during the previous monitoring window period. For example, the visual attention classification module 310 may periodically sample or otherwise analyze image data from the imaging device 304 and output a label of the classified visual attention state associated with the corresponding sampling period. The visual lane priority module 312 receives an indication of the current visual attention state and calculates or otherwise determines a corresponding metric indicating the driver's relative preference for different potential lanes based on the visual attention state label within the previous monitoring window.

[0057] In an exemplary embodiment, for a potential visual attention state corresponding to a specific lane, the visual lane priority ranking module 312 may implement a counter or similar feature to count the number of times the driver's visual attention state corresponds to that specific lane in the previous monitoring window. For example, the visual lane priority ranking module 312 may count the number of times the driver's visual attention state was classified as pointing towards the adjacent lane on the driver's side of the vehicle in the previous minute, thereby tracking or otherwise monitoring the frequency with which the driver looks at the driver's side lane in the previous minute. Similarly, the visual lane priority ranking module 312 may count the number of times the driver's visual attention state was classified as pointing towards the adjacent lane on the passenger side of the vehicle in the previous minute, thereby tracking the frequency with which the driver looks at the passenger side lane in the previous minute. Likewise, in embodiments where different areas of the driver's visual attention are classified as specific visual attention states of the current driving lane (e.g., when the driver's visual attention is directed towards the windshield), the visual lane priority ranking module 312 may count the number of times the driver's visual attention state is classified as pointing towards the current lane.

[0058] In one or more exemplary embodiments, the visual lane priority ranking module 312 also implements a timer or similar feature to count the duration for which the driver's visual attention state remains constant within consecutive sampling periods while being directed toward a specific lane in the previous monitoring window. For example, when the visual attention classification module 310 classifies the driver's visual attention state as being directed toward the adjacent lane on the driver's side of the vehicle in consecutive samples, the visual lane priority ranking module 312 can use the relative difference between the timestamps of consecutive samples to determine the corresponding duration for which the driver's visual attention is focused on the driver's side lane, and increment the cumulative duration for which the driver's visual attention is focused on the driver's side lane in the previous monitoring window. In this respect, by tracking or monitoring the duration for which the driver's attention is directed toward a specific lane, while also tracking or monitoring the frequency or number of times the driver's attention is directed toward that specific lane, the resulting metric for different potential lanes indicates the driver's relative preference for a specific lane among the available potential lanes.

[0059] Still referencing Figure 4 After determining the driver's current visual attention state and the corresponding driver lane preference metric, the visual lane priority process 400 identifies or otherwise determines at 406 whether an automatic lane change is pending. If so, it determines at 408 whether the pending automatic lane change is in the same direction as the driver's visual attention. In this regard, the lane change coordination system 306 receives an indication of the driver's current visual attention state from the driver monitoring system 302 and determines whether the driver's current visual attention state is classified into an adjacent lane that matches or otherwise corresponds to a pending automatic lane change determined based on traffic, obstacles, route information, etc. When the visual lane priority process 400 confirms that the driver's current visual attention state matches or otherwise corresponds to a pending automatic lane change, the visual lane priority process 400 automatically increases the priority associated with that specific lane at 410 to expedite the initiation of an automatic lane change in that direction or otherwise increases the likelihood of executing an automatic lane change in that direction.

[0060] For example, when an automatic lane change to the adjacent lane on the driver's side is previously determined by the lane change coordination system 306 to overtake slower-moving traffic in the vehicle path and wait for the guidance systems 78, 308 and / or vehicle control systems 80, 320 to execute, and the driver's current visual attention state is classified as oriented towards the driver's side lane, the lane change coordination system 306 can automatically increase the priority value associated with the pending lane change and / or the driver's side lane to increase the likelihood that the motion planning module of the guidance systems 78, 308 will determine the corresponding motion plan to change lanes from the current driving lane to the driver's side lane. Alternatively, in some embodiments, the lane change coordination system 306 may set a flag or provide other indications in conjunction with the automatic lane change request provided to the guidance systems 78, 308 to indicate that the guidance systems 78, 308 should attempt to accelerate the pending lane change to the driver's side lane. Therefore, since the driver's visual attention aligns with or otherwise confirms the automatic lane change and provides increased confidence in the expectation of the automatic lane change, the guidance systems 78, 308 and / or vehicle control systems 80, 320 can respond to the driver's visual attention state by accelerating the autonomous operation of the vehicle to change lanes in the direction of the driver's visual attention, so as to behave more in line with the driver's expectations indicated by the driver's visual attention, thereby improving the user experience.

[0061] On the other hand, when the visual lane priority ranking process 400 determines that the driver's current visual attention state is in a different or opposite direction from the proposed automatic lane change, the visual lane priority ranking process 400 automatically reduces the priority associated with that particular lane at 412 to delay initiating an automatic lane change in that direction or otherwise reduce the likelihood of performing an automatic lane change in that direction. For example, when an automatic lane change to the adjacent lane on the driver's side is previously determined by the lane change coordination system 306 to overtake slower-moving traffic in the vehicle path, but the driver's current visual attention state is classified as pointing to the passenger-side lane on the opposite side of the vehicle, the lane change coordination system 306 may automatically reduce the priority value associated with the proposed lane change and / or the driver-side lane to reduce the likelihood that the motion planning module of the guidance systems 78, 308 will determine a corresponding motion plan to change lanes from the current driving lane to the driver-side lane. Alternatively, in some embodiments, the lane change coordination system 306 may set a flag or provide other indications in conjunction with the automatic lane change request provided to the guidance systems 78, 308 to indicate that the guidance systems 78, 308 should suspend or delay the pending lane change to the driver's side lane. In other embodiments, the lane change coordination system 306 may cancel the pending automatic lane change request when the driver's visual attention state or direction does not match the direction of the pending automatic lane change. Therefore, when the driver's visual attention is inconsistent with the automatic lane change and reduces confidence in the expectation of the automatic lane change, the guidance systems 78, 308 and / or the vehicle control systems 80, 320 can respond to the driver's visual attention state by maintaining the autonomous operation of the vehicle in the current driving lane to avoid behaving inconsistent with the driver's expectations indicated by the driver's visual attention, thereby improving the user experience.

[0062] Still referencing Figure 4When no pending automatic lane change is detected at 406, the visual lane priority process 400 analyzes lane preference metrics determined based on the driver's visual attention state at 414 to detect or otherwise identify when a corresponding value of one or more of the driver's lane preference metrics exceeds a threshold indicating a driver lane preference for another lane. When the driver's lane preference metrics indicate a preference for another lane, the visual lane priority process 400 automatically increases the priority associated with adjacent lanes in the direction of visual attention and automatically decreases the priority associated with adjacent lanes in the opposite direction at 416, thereby accelerating the initiation of an automatic lane change in the driver's preferred direction or otherwise increasing the likelihood of an automatic lane change in the driver's preferred direction. In this regard, the lane preference threshold can be selected to effectively filter or otherwise ignore transient fluctuations in the driver's visual attention state that may be merely coincidental or unintentional and unlikely to reflect the driver's preference or intention. Therefore, when the driver’s lane preference metric does not indicate any particular lane preference, the visual lane priority ranking process 400 exits and repeats after the next sampling of the image data output by the imaging device 304 to dynamically respond to the driver’s visual attention state in essentially real time.

[0063] It should be noted that while this document describes the subject matter in a context where both the frequency and duration of the driver's visual attention being directed in a specific direction are required to confirm lane preference in an inclusive implementation, other implementations may be independent, requiring only that either the frequency or duration of the driver's visual attention being directed in a specific direction exceeds a corresponding threshold in order to assign lane preference. Furthermore, this subject matter is not limited to any particular type, number, or combination of visual lane preference metrics and corresponding thresholds that can be used to prioritize different lanes or otherwise assign relative preferences based on the driver's visual attention state.

[0064] Still referencing Figure 4 In an exemplary embodiment, at 414, the visual lane priority module 312 verifies or otherwise confirms that the frequency or number of times the driver's visual attention is directed toward the adjacent lane during the previous monitoring window is greater than a threshold number of times, and also verifies or otherwise confirms that the cumulative duration of the driver's visual attention being directed toward the adjacent lane during the previous monitoring window is greater than a threshold duration. In this way, the visual lane priority module 312 infers that the driver is more likely to prefer driving in that adjacent lane and / or that the driver has a better situational awareness of that adjacent lane based on the driver's visual attention being preferentially focused in a specific direction toward that particular adjacent lane, thereby improving safety and user experience when initiating automatic lane changes in that direction.

[0065] For example, if a driver's gaze at the adjacent lane on the passenger side exceeds a threshold number of times for a duration exceeding the threshold during the previous minute, the visual lane priority module 312 can determine that the driver prefers to change lanes to the passenger side and provide a corresponding instruction to the lane change coordination system 306 to increase the priority associated with the passenger side (and decrease the priority associated with the driver side). As a result, when an automatic lane change request is subsequently generated (e.g., determined by an overtaking evaluator to require a lane change to overtake traffic in a slower-moving path), the lane change coordination system 306 can utilize the increased priority associated with the passenger side to result in an automatic lane change to the passenger side instead of the driver side of the vehicle and / or accelerate the automatic lane change to the passenger side. In this way, by taking into account the driver's previously observed visual attention state when prioritizing potential driving lanes relative to each other, the visual lane priority process 400 at 408 reduces the likelihood of a mismatch between subsequent automatic lane changes and the driver's visual attention state.

[0066] Still referencing Figure 4 In an exemplary embodiment, the visual lane prioritization process 400 is repeated in response to each updated sample of image data from the imaging device 304 to continuously analyze the driver's visual attention state, dynamically updating and adjusting the relative priorities or preferences associated with different potential driving lanes to reflect the driver's visual attention most recently observed in the previous monitoring window. In this way, the direction of automatic lane changes can more intuitively follow the driver's expectations and align with the driver's situational awareness of the target lane for the automatic lane change by prioritizing directions most consistent with the driver's visual attention. Furthermore, by reducing the priority or delaying automatic lane changes that are far from the driver's visual attention, the visual lane prioritization process 400 reduces the likelihood of manual driver interaction to overtake or cancel automatic lane changes.

[0067] Some implementations of the visual lane priority process 400 allow a driver to visually and effectively cancel or deny an automatic lane change by keeping his or her gaze in the opposite direction. This delays the automatic lane change and / or lowers the priority of the target lane for that automatic lane change until the guidance systems 78, 308, and / or the lane change coordination system 306 no longer require the automatic lane change. In this regard, in some implementations, an automatic lane change cancellation zone (e.g., a specific area on the dashboard) can be defined such that when the driver's gaze or focus is within the automatic lane change cancellation zone and the driver's visual attention state is classified as an automatic lane change cancellation state, the visual lane priority module 312 and / or the lane change coordination system 306 can automatically overtake and cancel the automatic lane change. This allows the driver to effectively provide visual input to cancel the automatic lane change without requiring another manual or physical input (e.g., actuating a turn signal lever in the opposite direction).

[0068] Additionally, in some embodiments, the visual lane prioritization process 400 can be configured to support a specific sequence of driver visual attention states corresponding to predefined patterns for initiating or accelerating automatic lane changes. In this regard, the driver monitoring system 302 can be configured to detect or otherwise identify when a specific sequence of driver visual attention states corresponds to a predefined pattern for automatic lane changes in a particular direction, and in response, provide a corresponding marker to the lane change coordination system 306 to increase the priority or preference for that particular direction to initiate or accelerate the desired lane change. For example, a sequence of driver visual attention states can be specified or otherwise assigned to on-demand lane changes to the passenger side, followed by the center console, and then the passenger side rearview mirror, wherein in response to detecting the driver visual attention state sequence, the driver monitoring system 302 provides a corresponding marker to the lane change coordination system 306, which causes the guidance systems 78, 308 to automatically initiate the desired lane change from the current lane to the adjacent lane on the passenger side of the vehicle. It should be understood that many different potential sequences or patterns can be assigned to specific actions or operations of the vehicle, and the subject matter described herein is not intended to be limited to any particular pattern or sequence.

[0069] Figure 5 Depicting according to one or more embodiments Figure 4 An exemplary scenario of the visual lane priority sorting process 400. Figure 5 The text describes an initial state 500 in which vehicle 502 (e.g., vehicle 10) is traveling in the center lane 520 of a road, behind another vehicle 504, while operating in an autonomous cruise control mode or another secondary autonomous operating mode. This other secondary autonomous operating mode attempts to maintain vehicle 502 substantially centered along the lane centerline within the current lane 520 at a user-defined speed, subject to other user-defined or user-configurable constraints (e.g., distance from other vehicles). As described above... Figure 3 As described in the context, vehicle 502 includes an imaging device 506 (e.g., imaging device 304) located within the passenger compartment, oriented toward a driver's position 508 of vehicle 502 to capture analyzable video or other images of the driver seated at that position 508 (e.g., analyzed at 402 by visual attention classification module 310) to classify the driver's visual attention state, for example, by capturing video or other images of the driver's face or eyes that can be used to identify the relative orientation or direction of the driver's gaze 510. In this respect, Figure 5 The text describes a scenario where the driver's visual attention is categorized as directed toward the adjacent lane 540 on the passenger side (or right side) of the vehicle 502 because the driver's gaze 510 is directed toward the passenger-side rearview mirror or otherwise toward the passenger-side window.

[0070] In the initial state 500, the sensor systems 74 and 314 on vehicles 10 and 502 capture or otherwise acquire sensor data associated with other vehicles 504 within the current driving lane 520, and the guidance systems 78 and 308 (or their associated overtaking evaluators) determine, based on the distance between vehicles 502 and 504 and the current speed of the primary vehicle 502 relative to vehicles 504 in the path, that an automatic lane change should be performed to allow the primary vehicle 502 to overtake another vehicle 504. Based on the driver's visual attention state being categorized as pointing towards the passenger-side lane 540, the visual lane priority module 312 and / or the lane change coordination system 306 can increase the priority associated with the passenger-side lane 540 to expedite the guidance systems 78 and 308 in determining the corresponding motion plan (indicated by trajectory 512) to initiate and autonomously execute an automatic lane change from the current driving lane 520 to the adjacent passenger-side lane 540. In this respect, because Figure 4 The visual lane priority process 400 takes into account the driver's visual attention being directed toward the passenger-side lane 540 by increasing the priority associated with the passenger-side lane 540 and / or decreasing the priority associated with the driver-side lane 530. It can perform automatic lane changes to the passenger-side lane 540 to align with the driver's visual attention direction (indicated by line of sight 510), where the driver may have better situational awareness to better meet the driver's expectations, rather than defaulting to automatically changing lanes to the driver-side lane 530 to overtake vehicles 504 in the path without taking into account the driver's visual attention.

[0071] Figure 6 Depicting according to one or more embodiments Figure 4 An exemplary scenario of the visual lane priority sorting process 400. Figure 6 The text describes an initial state 600 in which vehicle 502 (e.g., vehicle 10) is traveling in the center lane 520 of a road behind another vehicle 504, operating in an autonomous cruise control mode or another secondary autonomous operating mode. This other secondary autonomous operating mode attempts to maintain vehicle 502 substantially centered along the lane centerline within the current lane 520 at a user-defined speed, subject to other user-defined or user-configurable constraints (e.g., distance from other vehicles). In this respect, Figure 6 The text describes a scenario where the driver's visual attention is categorized as directed toward the adjacent lane 530 on the driver's side of the vehicle 502 because the driver's gaze 610 is directed toward the driver's side rearview mirror or otherwise toward the driver's side window.

[0072] exist Figure 6In this scenario, guidance systems 78, 308 (or their associated overtaking evaluators) can determine that an automatic lane change is not required at the current time point based on the distance between vehicles 502, 504 and the current speed of the primary vehicle 502 relative to vehicle 504 in the path. However, based on the driver's visual attention being directed toward the driver's side lane 530 with sufficient frequency and duration (e.g., at 414), the visual lane priority module 312 and / or lane change coordination system 306 can increase the priority associated with the driver's side lane 530 to expedite the guidance systems 78, 308 in determining the corresponding motion plan (indicated by trajectory 612) to initiate and autonomously execute an automatic lane change from the current driving lane 520 to the adjacent driver's side lane 530. In this respect, because the visual lane prioritization process 400 takes into account the driver's visual attention and increases the priority associated with the driver's side lane 530, rather than waiting for the main vehicle 502 to continue decreasing the buffer distance with the vehicle 504 in the path until it overtakes another component of the evaluator or guidance system 78, 308 to request an automatic lane change, automatic lane changes can be accelerated and preemptively executed (e.g., once the priority associated with the driver's side lane 530 exceeds a threshold or exceeds the priority assigned to the current lane 520) to align with the driver's visual attention, which indicates that the driver may expect or anticipate that vehicle 502 will automatically change lanes to overtake vehicle 504. Therefore, by allowing the driver to effectively provide visual input to initiate automatic lane changes, the driver does not need to provide manual or physical input (e.g., actuating a turn signal lever) to manually initiate lane changes on demand.

[0073] Refer again Figures 1 to 4In one or more embodiments, the vehicle control system 300 and the visual lane priority process 400 can be co-configured to support self-learning to provide personalized, driver-specific automatic lane change behavior by adjusting the threshold used to initiate or cancel automatic lane changes in a manner that combines the driver's visual attention state with other interactions between the driver and the system. For example, when the driver cancels an automatic lane change through physical interaction with the system (e.g., by actuating a turn signal stalk) or by visual cancellation or overtaking, the lane change coordination system 306 and / or the visual lane priority module 312 can automatically increase the threshold used at 414 to reduce the sensitivity of the visual lane priority process 400 and require more frequent and / or longer visual attention to that side before initiating or accelerating an automatic lane change to that side of the vehicle. Conversely, when a driver manually initiates a lane change on demand via physical interaction with the system, the lane change coordination system 306 and / or the visual lane priority module 312 can automatically reduce the threshold used at 414 to improve the sensitivity of the visual lane priority process 400, requiring less frequent and / or shorter visual attention to that side before initiating or accelerating an automatic lane change to that side of the vehicle. Therefore, over time, the threshold can be adjusted in a user-specific manner to achieve more intuitive automatic lane change behavior that aligns with each individual driver's driving or riding preferences, while reducing the amount of physical or other manual interaction, thereby improving the user experience.

[0074] While at least one exemplary aspect has been presented in the foregoing detailed description, it should be understood that many variations exist. It should also be understood that one or more exemplary aspects are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary aspects. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A method for controlling a vehicle in an autonomous operation mode, the method comprising: The driver's gaze orientation is classified into a plurality of different visual attention states by a controller associated with the vehicle, based at least in part on the output of an imaging device on the vehicle. The visual attention state indicates the driver's visual attention to an area associated with the current driving lane on one side of the visual attention direction. The controller determines, at least in part, the driver's lane preference corresponding to the adjacent lane on the side of the current driving lane relative to the vehicle in the visual attention direction, based on the visual attention state. The controller increases the priority associated with the adjacent lane in accordance with the driver's lane preference, thereby generating an increased priority associated with the adjacent lane; as well as The controller autonomously operates one or more actuators on the vehicle to initiate a lane change from the current lane to the adjacent lane in a manner influenced by the increased priority associated with the adjacent lane.

2. The method according to claim 1, wherein, Autonomously operating the one or more actuators to initiate a lane change from the current driving lane to the adjacent lane according to the increased priority includes: providing instructions to the motion planning module to generate a motion plan to execute the pending lane change in the visual attention direction.

3. The method according to claim 1, further comprising: The priority associated with the second adjacent lane on the second side opposite to the side of the current driving lane that corresponds to the adjacent lane is reduced, thereby producing a reduced priority associated with the second adjacent lane; as well as Autonomously operating the one or more actuators includes: autonomously operating the one or more actuators to delay a second lane change from the current driving lane to the second adjacent lane according to the reduced priority.

4. The method according to claim 1, wherein, Autonomously operating the one or more actuators includes: autonomously operating the one or more actuators to cancel or delay a planned lane change to a second adjacent lane on a second side opposite to the side of the current driving lane that corresponds to the adjacent lane.

5. The method according to claim 1, wherein: Classifying the orientation of the driver's gaze in the vehicle as the visual attention state includes: determining at least one of the frequency and duration of visual attention in the visual attention direction within a previous time window; and Determining the driver's lane preference includes: determining the adjacent lane as the preferred lane when at least one of the frequency and the duration of visual attention in the visual attention direction during the previous time window is greater than a threshold.

6. A vehicle comprising: Imaging equipment; One or more actuators on the vehicle; and A controller, coupled to the imaging device and the one or more actuators, wherein the controller, through a processor, classifies the orientation of the driver's gaze in the vehicle into a visual attention state among a plurality of different visual attention states, based at least in part on the output of the imaging device, the visual attention state indicating the driver's visual attention to an area associated with one side of the current driving lane in the visual attention direction; The driver lane preference is determined, at least in part, based on the visual attention state, corresponding to the adjacent lane on the side of the current driving lane relative to the vehicle in the visual attention direction. The priority associated with the adjacent lane is increased in accordance with the driver's lane preference, thereby generating an increased priority associated with the adjacent lane; And autonomously operate the one or more actuators to initiate a lane change from the current driving lane to the adjacent lane in a manner influenced by the increased priority associated with the adjacent lane.

7. The vehicle according to claim 6, wherein: The imaging device includes a camera that is oriented to capture an image of the driver as the driver operates the vehicle.

Images