Context-sensitive user interface for enhanced vehicle operation

Abstract

Systems and methods for context-sensitive user interfaces for enhanced vehicle operation. One example method is a method implemented by a system of one or more processors, the system being included in a vehicle. The method includes causing presentation, via a display of the vehicle, of a unified user interface including a combined view aggregating autonomous visualizations and map information, the autonomous visualizations and map information being associated with a zoom level, the unified user interface being generated based on first context information indicative of operation of the vehicle. Second context information indicative of a subsequent operation of the vehicle is accessed. The unified user interface is updated based on the second context information, the combined view is adjusted, the adjusting of the combined view including adjusting a size of the combined view in the unified user interface or adjusting the zoom level.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 938842, filed November 21, 2019, entitled “CONTEXT SENSITIVE USER INTERFACEFOR ENHANCED VEHICLE OPERATION”, and U.S. Provisional Application No. 62 / 938769, filed November 21, 2019, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates to user interfaces, and more specifically to user interfaces for vehicle operation. Background Technology

[0004] Displays for presenting digital user interfaces are increasingly being included in vehicles for vehicle control and / or operation. For example, a display can be included in the vehicle's dashboard. In this example, the display can present a user interface including the vehicle's current speed, total distance traveled, outside temperature, etc. As another example, a display can be included in the center console of the vehicle. This display can be used to present navigation information, control the air conditioning, etc. Such digital user interfaces can therefore replace mechanical displays, buttons, knobs, etc.

[0005] However, as digital user interfaces are expanded to include more control and / or operation of the vehicle, their complexity is increasing. For some digital user interfaces, this complexity reduces the usability associated with the interface. Therefore, the user experience associated with vehicle operation may be diminished. For example, some functions may be hidden in menus at different levels. As another example, users may need to remember complex sequences of user input to navigate the digital user interface. Therefore, providing an intuitive digital user interface can enhance the user's enjoyment of operating the vehicle. Attached Figure Description

[0006] The foregoing aspects and many accompanying advantages will become more readily understood when taken in conjunction with the following detailed description, in which the accompanying drawings are shown:

[0007] Figure 1A A block diagram of an example contextual user interface system that presents the user interface is shown.

[0008] Figure 1BA block diagram of an example contextual user interface system is shown, which presents an updated user interface based on received user input.

[0009] Figure 2A This is a flowchart of an example process for updating a unified user interface.

[0010] Figure 2B This is a flowchart of an example process for updating the unified user interface based on the vehicle's location.

[0011] Figure 2C This is a flowchart of an example process for updating the unified user interface based on vehicle function selection.

[0012] Figure 3A This is a flowchart of an example process for multitasking control using a unified user interface.

[0013] Figure 3B The user interface depicting the vehicle is shown, along with the music user interface.

[0014] Figure 3C The user interface depicting the vehicle, along with the quick control user interface, is shown.

[0015] Figure 3D The user interface is shown, displaying the climate user interface.

[0016] Figure 4 This is a flowchart illustrating an example process for adjusting the icon to the driver's or passenger's side.

[0017] Figure 5A A sample user interface for parking vehicles is shown.

[0018] Figure 5B An example of a user interface that instructs users to park their vehicles while they are charging is shown.

[0019] Figure 6A A sample user interface for the driving view is shown.

[0020] Figure 6B An example user interface for the driving view is shown, where the icons are adjusted for the passenger view.

[0021] Figure 6C An example user interface for a driving view with today's user interface is shown.

[0022] Figure 6D An example user interface for a driving view with an air conditioning user interface is shown.

[0023] Figure 7A This shows a sample user interface for driving a vehicle in an "off-road" environment.

[0024] Figure 7B An example user interface showing the driving view when the vehicle is driving in an "off-road" environment is displayed, and powertrain information is also presented.

[0025] Figure 8A A sample user interface for a vehicle used to tow a truck trailer is shown.

[0026] Figure 8B This shows a sample user interface for the vehicle's driving view when towing a trailer.

[0027] Figure 9 An example user interface for a vehicle in camping mode is shown.

[0028] Figure 10A A sample user interface for a driving view used for vehicle navigation is shown.

[0029] Figure 10B Another example user interface for the driving view is shown.

[0030] Figure 10C Another example user interface for the driving view is shown.

[0031] Figures 11A-11C A user interface for selecting navigation information is shown.

[0032] Figure 12 This is a flowchart of an example process for associating passenger preference information with a specific passenger.

[0033] Figure 13A This is a flowchart of an example process for adjusting air conditioning based on passenger tracking.

[0034] Figures 13B-13D An example user interface for air conditioning tracking passengers is shown.

[0035] Figure 14 This is a flowchart of an example process for adjusting a reflector based on passenger eye tracking. Detailed Implementation

[0036] This specification describes, with reference to several embodiments, an enhanced user interface that can be used to control and / or operate a vehicle. The user interface described herein can be an example of a digital user interface rendered by a processor or system and presented via a display located within the vehicle. In some embodiments, the display may be included in the central portion of the vehicle. For example, the display may be larger than a certain diagonal size (e.g., 10 inches, 13 inches, 15 inches, 17 inches, 21 inches, etc.). Because the display may be centrally located and larger than a certain diagonal size in some embodiments, it can provide a centralized interface where vehicle control and / or operation takes place. Therefore, ensuring that the centralized interface is easy to understand and use can improve vehicle usability. As will be described, the enhanced user interface dynamically unifies (e.g., aggregates) autonomous visualizations (e.g., visualizations of the vehicle generated from sensors), map information, navigation information, and / or vehicle function controls.

[0037] One embodiment of the invention is a vehicle having a user interface (referred to herein as a unified user interface) that can be dynamically updated based on contextual information related to the operation of the vehicle. For example, the contextual information may indicate that the vehicle is parked or being driven. As another example, the contextual information may indicate that the vehicle is navigating towards a destination and / or is being autonomously or semi-autonomously controlled. As another example, the contextual information may indicate that the vehicle is towing a trailer, is off-road driving, etc. As another example, the contextual information may be related to the control of vehicle functions. As will be described, the unified user interface includes easily understandable interactive elements that respond to the current contextual information.

[0038] Regarding vehicles in parking positions, the unified user interface can present autonomous visualizations that include a graphical depiction of the vehicle selectable using a touchscreen interface. Vehicle controls can be included in the graphical depiction, allowing end users to open the trunk, open / close windows, disconnect or connect trailers, etc. In some embodiments, the autonomous visualization can include a graphical representation of the real-world environment in which the vehicle is located. Regarding autonomously or semi-autonomously controlled vehicles, the unified user interface can present autonomous visualizations overlaid on map information. For example, the autonomous visualization can include a representation of the surface on which the vehicle is driving. In this example, a graphical depiction of the vehicle in a specific lane of a road, along with graphical representations of other vehicles or objects approaching the vehicle, can be shown. As an example, this can include representations of lane lines, exit or entrance ramps, pedestrians, hazards, signs, or other objects. The map information can include a graphical representation of a portion of a map in which the vehicle is being driven. For example, the map information can indicate the number of lane lines on the current road, other roads approaching the current road, or other map-based information. For example, the road can be represented along with its identifier.

[0039] For vehicles navigating towards a destination, a unified user interface can include autonomous visualization, map information, and navigation information. As will be described, the unified user interface can graphically depict navigation events identified in the navigation information (e.g., an imminent turn, exit ramp, entry ramp, etc.). For example, the unified user interface can graphically depict a vehicle moving from its current lane to a different lane. Furthermore, the unified user interface can zoom in or out on the presented information based on contextual information. For example, zoom levels can be set that cause adjustments to the size of autonomous visualization, map information, and navigation information.

[0040] As an example, the information presented can be scaled down when a vehicle exits a highway ramp. In this example, autonomous visualizations (e.g., a graphical depiction of the vehicle) might become smaller, while portions of the map information representing larger real-world geographical areas (e.g., depicting more roads) become relatively larger. Imminent navigation events (such as navigation routes) can be similarly scaled down to show a larger portion of the route. In this way, an upcoming turn on a ground-level road can be presented to the end user. After the user exits the exit ramp and is driving on a ground-level road, the presented information can be selectively scaled up.

[0041] Additional contextual information may relate to the control of vehicle functions, such as playing music, adjusting the air conditioning, streaming video, and overlaying video from one or more cameras. As will be understood, control of such vehicle functions may consume different parts of the user interface. For example, to adjust air conditioning settings, the user interface may present air conditioning controls. As will be described, a unified user interface can ensure that autonomous visualizations, map information, and / or navigation information remain unobstructed. For example, an end user can select user interface elements to control air conditioning settings while navigating to a destination. In this example, the unified user interface can adjust the presentation of autonomous visualizations, map information, and / or navigation information, and includes air conditioning controls, or the overlay of air conditioning controls on a navigation display. Example techniques for adjusting the presentation of information to passengers may include moving autonomous visualizations, map information, and / or navigation information to different parts of the unified user interface (e.g., away from the end user). Another example technique is to overlay certain controls on top of map or navigation information.

[0042] Contextual information can therefore be related to the operation of the vehicle. Example contextual information is described herein; however, it should be understood that other information may be used and falls within the scope of this disclosure. For example, contextual information may indicate that the vehicle is navigating to a location. In this example, the contextual information may reflect the vehicle's position relative to that location. For example, the location may indicate an upcoming or previous driving event (e.g., turning, lane changing, etc.). Therefore, when the vehicle navigates to that location, the contextual information can be updated based on that location. In this way, as will be described, the user interface can be updated based on that location (e.g., the presented map may be zoomed out, etc.).

[0043] While the above description focuses on the user interface presented on the display, in some embodiments, multiple displays may be used. For example, the first display may be located near or in front of the vehicle driver. In this example, the second display may be located in the central part of the vehicle or away from the driver. As an example, the vehicle may represent a semi-truck with two or more displays.

[0044] First, a sample user interface is described. This sample user interface provides centralized control of the vehicle. However, during routine use of the user interface, it may obscure important information, such as a map depicting the vehicle's location and / or navigation information. As will be described, a unified user interface can improve upon this sample user interface. It should be recognized that the selection of controls or other information on the user interface can be performed via touch interfaces or other well-known user interface mechanisms.

[0045] Example User Interface

[0046] An example user interface for centralized control of a vehicle can be divided into a first part and a second part. For example, the first part may include a graphical representation of the vehicle, which the user interface can update substantially in real time based on vehicle operation. For instance, if the vehicle is braking, the graphical representation can be updated to depict the brake lights. As another example, if the vehicle's headlights, hazard lights, etc., are activated, the graphical representation can be updated accordingly. The second part may include a graphical representation of a map. For example, the map may include icons, such as arrows, indicating the vehicle's position on the map. In this example, when the vehicle is being driven, the icons can be updated to reflect the vehicle's basic real-time position.

[0047] Therefore, the first section can provide a concise view of the real-time situation associated with vehicle operation. As an example, the user interface can indicate lane markings in this first section. Thus, users can quickly check whether they are correctly following the lane markings. Furthermore, the vehicle can be allowed to operate autonomously. Therefore, users can refer to the first section to ensure that the vehicle correctly recognizes lane markings in autonomous mode.

[0048] The second section can be used to present information about the area associated with operating the vehicle. For example, the second section could depict traffic conditions on roads included in the map. In this example, the user can quickly determine if there are any accidents, traffic congestion, etc., on a highway or expressway. The second section can also present navigation information. For example, the route to a specific destination can be displayed on the map, along with instructions for the next turn. In this way, the user can quickly identify how to reach the destination.

[0049] This example user interface may additionally include the display of multiple icons or other graphic markers associated with different vehicle functions controllable via the user interface. Example vehicle functions may include air conditioning, music, monitoring of energy usage for electric vehicles, driving control, automatic driving control, etc. To adjust vehicle functions, such as adjusting air conditioning settings, the user can select an icon. As an example, the user interface can be presented via a touchscreen display. In this example, the user can touch a portion of the display depicting the icons. As another example, the user interface may respond to verbal commands. In this example, the user can provide a verbal command indicating the type of vehicle function (e.g., "Set the air conditioning").

[0050] When a user selects an icon, a menu or other user interface associated with the type of vehicle function can be presented. Since the user interface is divided into two parts, the presented menu can be overlaid on at least one of these parts. For example, if the user selects an icon associated with music, the user interface can update to a music selection menu. It is understood that the menu can obscure at least one of the two parts. For example, the menu can be overlaid on a map. Therefore, at least a portion of the map can be obscured by this music menu. The user can be prompted to remove the menu to view the entire map. For example, the user can swipe down on the menu.

[0051] Therefore, during routine operation of the example user interface, controlling vehicle functions can cause information included in the first and / or second sections to be obscured. This can be disadvantageous during vehicle navigation. As an example, the second section may indicate a route to follow on the presented map. If the user prefers to select updated air conditioning settings, adjust the music, view energy tracking information, etc., at least a portion of the indicated route may become hidden.

[0052] Unified User Interface

[0053] As will be described, a system or processor rendering a unified user interface can combine autonomous visualizations (e.g., a graphical depiction of a vehicle) and map information. This combination may be referred to herein as a combined view and can be presented as a consistent view that provides the user with at least the autonomous visualizations and map information. This view can advantageously remain unobstructed by menus or user interfaces associated with vehicle function types. As will be described, menus for vehicle functions can optionally be dynamically presented separately from menus of the combined autonomous visualizations and map information.

[0054] Additionally, the system or processor can dynamically update the composite view based on upcoming driving events. For example, the system can adjust the zoom level associated with the composite view. In this example, the zoom level can indicate the rendered or virtual camera associated with the composite view. For instance, a higher zoom level could indicate that the rendered or virtual camera contains map information for a larger area. Therefore, the size of the autonomous visualization can be reduced accordingly.

[0055] Unified User Interface – Combining Autonomous Visualization and Mapping Information

[0056] In some embodiments, the system or processor can update the unified user interface based on current context information associated with vehicle operation. For example, if the vehicle is parked, the unified user interface can depict a large graphical representation of the vehicle. Thus, the graphical representation can substantially fill the unified user interface. This graphical representation may include interactive options associated with vehicle control. For example, the vehicle may be an electric truck, and the options may include suspension adjustment, opening / closing the charging port, opening / closing the tailgate, opening / closing the tailgate, etc. If the user selects to pause adjustment, the unified user interface can be updated to reflect the different pause levels that can be selected.

[0057] The aforementioned large graphical representation of the vehicle can be included in a zoomed-in view of the map. For example, the unified user interface can optionally reflect environmental information approaching the vehicle. The graphical representation can optionally reflect a third-person view of the vehicle, such as reflecting a virtual or rendered camera positioned forward from the rear of the vehicle. In some embodiments, the graphical representation can be presented in a street view of the map. In some embodiments, the graphical representation can be presented on a representation of roads, optionally along with the names of the roads and / or adjacent roads. Users of the unified user interface can provide user input to zoom out of the map. For example, users can use pinch-to-zoom technology to zoom out of the map. Since the graphical representation of the vehicle is located in the map, the size of the graphical representation can decrease as the map is zoomed out. In this way, users can view a larger map depicting a larger area around the vehicle. Furthermore, users can induce translations of map information via user input. Example user input may include swiping in a specific direction, which causes the map to adjust the area depicted. Additionally, if the graphical representation of the vehicle is not included in the adjusted area, the graphical representation of the vehicle can be removed from the combined view. Optionally, the map can be zoomed out such that the combined view includes the area enclosed by the area depicted graphically including the vehicle (e.g., based on its location) and the area translated to (e.g., the adjusted area).

[0058] Contextual information can be updated by the user based on the map. For example, the user can select a location on the map. In this example, the unified user interface can present a route that the vehicle can follow to reach that location. Therefore, the unified user interface can present the route as a graphical depiction extending from the vehicle to the location.

[0059] If the user puts the vehicle in motion, the system or processor can update the unified user interface to present a larger view of the vehicle's graphical representation. Therefore, contextual information can be updated based on navigation information to reflect that the user is driving. For example, as described below, the unified user interface can present autonomous visualizations that include a driving view of the vehicle traversing a real-world environment. The driving view can reflect the vehicle's forward and backward views as it travels through the environment toward that location. For example, lane markings, other vehicles, etc., can be rendered in the unified user interface. The unified user interface can also present individual turning directions. As an example, a left-turn arrow can be included before the user is instructed to turn left on a street.

[0060] In this way, compared to the example user interface described above, the graphical representation of the vehicle can be combined with map information. Therefore, the unified user interface can dynamically adjust the presentation of the graphical representation and / or map information, essentially in real time. For example, the map can be dynamically zoomed in and out, etc.

[0061] The vehicle may optionally include cameras or sensors that can be used to determine the positions of other vehicles, lanes, exit ramps, entrance ramps, stop signs, traffic lights, etc., in near real-time. Therefore, during navigation, a unified user interface can instruct the user to navigate through a certain number of lanes before turning, exiting ramps, etc.

[0062] In one embodiment, the unified user interface can present an autonomous visualization of obstacles included in the exit ramp. Optionally, obstacles can be highlighted or otherwise brought up to ensure the user notices them. In some embodiments, the unified user interface can be updated to present an updated navigation route that avoids the exit ramp.

[0063] While the above description focuses on navigation, it should be understood that such obstacles can also be presented in the unified user interface during normal vehicle operation. For example, the unified user interface can display a larger graphical representation of the vehicle when it is parked. If the user puts the vehicle into motion, the context information can be updated to reflect that the vehicle is being driven. Therefore, the unified user interface can be updated to present an autonomous visualization reflecting the vehicle's driving view. For example, the driving view could include a rear view of the vehicle traversing streets or off-road terrain. The driving view can also indicate lane markings, other vehicles, stop signs, streetlights, pedestrians, potholes, rocks, obstacles, or hazards. Furthermore, the unified user interface can include map information such as street names, the location of stop signs or traffic lights, and the names of nearby businesses or locations. Users can similarly zoom out on the map, such as via touch input or verbal commands, to have the unified user interface display a zoomed-out view.

[0064] Unified User Interface – Vehicle Function User Interface

[0065] The unified user interface can additionally include icons associated with vehicle function types. For example, icons could enable selections such as air conditioning functions, music functions, etc., as described herein. Compared to the example user interface described above, the unified user interface ensures that the combined autonomous visualization and map information is not obscured. For example, if the end user selects the air conditioning icon, a menu can be presented. Advantageously, the unified user interface can reduce the size associated with the combined autonomous visualization and map information. Furthermore, the unified user interface can present menus within the remaining portion of the unified user interface.

[0066] In some embodiments, the aforementioned dynamic scaling can be based on current context information. For example, if a user is navigating a complex series of streets, the unified user interface can increase the size associated with a combination of autonomous visualizations and map information. In this way, the user can clearly see upcoming turns. Similarly, if there is an impending hazard, the unified user interface can increase the size of the map and the depicted hazard to ensure that the user can easily see the hazard.

[0067] Unified User Interface – Driver / Passenger Side Adjustment

[0068] For some vehicles, such as trucks, the centralized display may be positioned beyond a threshold distance from the driver and passenger seats. Therefore, the driver can easily access a portion of the display closest to the driver (e.g., the left side). Conversely, the driver may have more difficulty accessing the more distant portion of the display (e.g., the right side). Similarly, passengers may easily access the right side of the display but less so the left side (in cars designed for left-hand drive). In some embodiments, the unified user interface may be updated based on whether the driver or passenger is interacting with it.

[0069] As an example, icons associated with function types can be presented on the driver's side (e.g., along the left side of the display from top to bottom). This allows the driver easy access to these icons. When an icon is selected, the associated menu or user interface can be presented on the driver's side. Conversely, a unified user interface can present icons on the passenger's side. When an icon is selected, the associated menu or user interface can be presented on the passenger's side.

[0070] As will be described, in some embodiments, the unified user interface (UI) may be updated based on whether a driver or passenger is currently or will be interacting with the UI. For example, the system or processor may identify whether a driver or passenger is currently or will be interacting with the UI. The system may optionally acquire information from an infrared emitter or a projector. These may indicate whether the end user's hand has moved from a particular side of the display to the UI. Thus, after detecting a user's hand moving toward the display, the UI may be updated based on that particular side. The system may optionally acquire information based on one or more cameras located inside the vehicle. For example, the system may analyze video or images to determine whether a driver or passenger is using or will be using the UI.

[0071] The above-described embodiments and other embodiments of the unified user interface will be described in more detail below.

[0072] Example diagram

[0073] Figure 1AA block diagram of an example contextual user interface system 100 that causes the presentation of user interface 102 (e.g., the unified user interface described herein) is shown. Contextual user interface system 100 may be a system of one or more processors, an application-specific integrated circuit (ASIC), etc. Contextual user interface system 100 may be included in a vehicle such as an electric vehicle and may cause a display to present user interface 102. The display may be, for example, a touch-sensitive display included in the front of the vehicle. For example, the display may be included in the central front of the vehicle.

[0074] The context-based user interface system 100 can receive user input 104 provided by the user of the user interface 102. Example user input 104 may include touch-based user input, verbal commands, etc. In this way, the user of the user interface 102 can interact with the user interface 102. For example, the user can provide user input 104 to operate different aspects of the vehicle. In this example, the user can select from icons 108A-108H. Each icon can enable control of a type of vehicle function. For example, icon 108A can enable control of a driving function (e.g., control of steering sensitivity, acceleration characteristics, etc.). As another example, icon 108H can enable control of a music streaming application.

[0075] Therefore, the context-based user interface system 100 can output vehicle operation information 106. For example, information 106 can be provided to systems, modules, applications, etc., that regulate vehicle operation. As an example, a user can adjust steering sensitivity. The vehicle operation information 106 can thus reflect the adjustment, allowing the systems, modules, software, and applications associated with steering control to be updated accordingly.

[0076] As shown, the user interface 102 includes autonomous visualizations (e.g., a graphical depiction of the vehicle 110A) in a combined view along with map information 110B. As described above, the graphical depiction 110A can reflect real-time operational information associated with the vehicle. For example, if the vehicle's lights are on, the graphical depiction 110A can be updated to show that the lights are on. The map information 110B can represent a map of the vehicle's location. As will be described, this map information can be updated based on the current context associated with vehicle operation. For example, when the vehicle is being driven, the map can optionally be zoomed in to present a driving view. As another example, during navigation, the map can optionally indicate the route the vehicle is to follow.

[0077] In some embodiments, map information 110B may be used at least in part to render the autonomous visualization. For example, the autonomous visualization may include a graphical representation of the vehicle's exterior (e.g., the real-world environment). In this example, sensor information (e.g., images from an image sensor) may be analyzed by system 100 or another processor or system in the vehicle to render the graphical representation. The map information may be used to determine the physical features or characteristics of the external environment. For example, the number of lanes may be identified based on the map information. As another example, roundabouts, upcoming lane movements or changes, upcoming highway interchanges, etc., may be identified. In some embodiments, these physical features or characteristics may be used to inform the generation of the autonomous visualization. For example, the autonomous visualization may include the precise path of an upcoming section of road that may not yet be visible or may be occluded. As another example, the combined view may be zoomed out or panned, and the map information may inform the generation of the autonomous visualization. For example, if the user adjusts the combined view to pan forward one mile or zoom up, these parts of the external environment may not yet be visible. Advantageously, the map information may be used to present their appearance. For example, representations of buildings, etc., may be included in the combined view based on the map information.

[0078] exist Figure 1A In this context, user interface 102 presents navigation user interface 112. Users can use this interface to indicate the location they wish to navigate to. Advantageously, the navigation user interface 112 is positioned so as not to obstruct the aforementioned combined views 110A-110B. Furthermore, the navigation user interface 112 can be located on the driver's side, allowing the driver easy interaction with it.

[0079] Figure 1B A block diagram of an example contextual user interface system 100 is shown, which presents an updated user interface 102 based on received user input 104. In the illustrated embodiment, the user of user interface 102 has provided user input 104 to enable navigation to a location. Therefore, navigation user interface 112 has been updated to specify the direction toward that location. Furthermore, route 114A is presented in combined views 110A-110B. This route can be depicted on map 110B as leading to location 114B.

[0080] In some embodiments, as described below, the navigation user interface 112 can be optionally removed by the user interface 102 when the vehicle is being driven. For example, the size of the combined views 110A-110B can be dynamically increased. In this example, the combined views 110A-110B may include more detailed autonomous visualizations (e.g., a driving view).

[0081] As an example, the driving view can represent a rear-up view of a graphical depiction of the vehicle 110A. As an example, the driving view can resemble a driving view from a camera or drone located at a threshold distance behind and above the vehicle. The system or processor can use sensor data, such as images from image sensors, combined with one or more machine learning models (e.g., convolutional neural networks or other computer vision techniques) to generate information to be included in the driving view. For example, the machine learning model can analyze input images from image sensors located around the vehicle. In some implementations, the image sensors can provide a 360-degree view of the vehicle's surroundings. These input images can be analyzed to classify vehicles (e.g., cars, trucks, motorcycles), objects (hazards, potholes, speed bumps, pedestrians, parking lights, traffic lights), etc., depicted in the images. Optionally, the input images can be stitched together to provide a consistent (e.g., 360-degree) view of the real-world environment in which the vehicle is located (e.g., via a machine learning model, such as a neural network). Images or other sensor data can be received at a specific frequency (e.g., 30 frames per second, 60 frames per second, 120 frames per second, etc.). These images can be analyzed to update the information included in the driving view.

[0082] In some embodiments, the user interface system 100 can access models associated with vehicles, objects, etc. The user interface system 100 can then render the models to generate a driving view based on received sensor data. For example, the user interface system 100 can receive information indicating the current location associated with a particular type of neighboring vehicle (e.g., a truck). In this example, the user interface system 100 can render a model of the truck in the driving view that corresponds to the current location. As will be understood, the position of the rendered model in the driving view can be determined using camera parameters from one or more image sensors that acquire corresponding images of the truck. Thus, the position of the real-world truck can be converted to its position within the driving view. Furthermore, in implementations where images are stitched together, the resulting stitch can represent a map indicating the position, size (e.g., bounding box), etc., of vehicles and / or objects. This map can be used, for example, by system 100 or a different system or processor to identify the position of the truck within the driving view for rendering. Although the use of models has been described above, in some embodiments, physical characteristics of vehicles or objects can be extracted from one or more images. As an example, the system or processor can generate the appearance of a vehicle or object to be rendered in user interface 102.

[0083] Therefore, the driving view can reflect a graphical representation of the real-world environment in which the vehicle is located. Graphical depictions of other vehicles can be rendered as moving around the vehicle, consistent with their actual positions in the real-world environment. Furthermore, road markings, signs, etc., can be presented in the driving view. The user interface system 100 can update the user interface 102 in real time (e.g., substantially in real time) to render vehicles and / or objects included in the real-world environment. For example, images can be received, analyzed, and used to update the user interface 102.

[0084] As will be described in more detail below, regarding Figure 2B The user interface system 100 can additionally update the user interface 102 to reflect navigation information. Therefore, the combined view described herein can include autonomous visualization, map information, and navigation information. For example, as the vehicle is traveling along a route, the user interface 102 can be updated to indicate navigation events. In this example, the user interface 102 can instruct the user to move in one or more lanes, turn, etc.

[0085] In some implementations, vehicles can operate in autonomous or semi-autonomous modes. Therefore, a driving view can provide insight into the real-world environment as determined by the systems or processors included in the vehicle. Users can review the driving view to ensure the real-world environment is interpreted correctly. Furthermore, the use of navigation information can proactively project actions the vehicle is about to perform onto the user.

[0086] Therefore, the user interface 102 can be dynamically updated based on the context associated with vehicle operation. For example, the navigation user interface 112 is presented based on user input 104 indicating the selection of navigation functions. Figure 1A Therefore, the dimensions of the combined views 110A-110B are dynamically reduced to fit the navigation user interface 112. Figure 1B In this context, the navigation user interface 112 displays the direction toward that location. As described above, the navigation user interface 112 can be dynamically removed as the user begins driving. In this way, the user can focus on the combined views 110A-110B, which can be updated to include navigation information (e.g., upcoming navigation events).

[0087] Example Flowchart

[0088] Figure 2A This is a flowchart of an example process 200 for updating a unified user interface. For convenience, process 200 will be described as being executed by a system with one or more processors (e.g., context user interface system 100).

[0089] In box 202, the system presents a unified user interface depicting the vehicle. For example... Figure 1A-Figure 1BAs shown, the unified user interface can include autonomous visualizations (e.g., a graphical depiction of the vehicle). Furthermore, the unified user interface can include map information. The degree to which the map information is zoomed in or out can be based on the current context associated with vehicle operation. For example, if the vehicle is parked, the unified user interface can present a zoomed-in view of the map. In this example, the graphical depiction of the vehicle can substantially fill the display showing the unified user interface.

[0090] As another example, if the vehicle is being driven, the map information can be further zoomed out. As an example, the map information can be rendered as a street view in a unified user interface. In this example, the unified user interface can include a graphical depiction of the vehicle driving on a rendered version of the street. As another example, the view of the map information can represent a substantially top-down view. For example, during navigation, the unified user interface can cause the map to zoom out to indicate an upcoming turn. In this example, the zooming out can occur based on identifying a series of rapidly approaching turns. Therefore, the unified user interface can advantageously depict a sequence of turns on the map information. As another example, the zooming out can occur based on the type of upcoming driving event. For example, exiting an exit ramp onto a surface road might cause zooming out. As another example, the zooming out can occur based on the detection of a hazard. For example, a hazard can be detected using images from the vehicle's image sensors. In this example, the route can be updated to avoid the hazard, and the map information can be zoomed out to depict at least a portion of the updated route.

[0091] The unified user interface may additionally include menus or user interfaces associated with vehicle function types. As described herein, this menu or user interface can be positioned next to the map information. Therefore, users of the unified user interface can interact with the menus while maintaining a view of the map information.

[0092] In box 204, the system receives information to update context information. Users of the unified user interface can provide user input to update the context associated with vehicle operation. For example, a user can instruct the unified user interface to present navigation information toward a specific location. As another example, a user can provide user input to select an icon associated with a vehicle function type.

[0093] Furthermore, the received information may not be based on user input. For example, cameras or sensors positioned around the vehicle can be used to identify hazards, traffic lights, stop signs, environmental conditions, etc. This information can be used to update the unified user interface. For instance, if the vehicle is driving in an off-road environment, environmental conditions could reflect fast-moving water ahead or tilt angles greater than a threshold. Similarly, the received information can also be related to the vehicle's positioning or location. Figure 2BAs described herein, the system can update the unified user interface based on the vehicle's position relative to the navigation route. For example, the combined view described herein can be zoomed in, zoomed out, etc., based on position.

[0094] In box 206, the system updates the unified user interface based on this information. Regarding user input associated with an icon, the unified user interface can be updated to include the menu or user interface associated with that icon. Similarly, if the user enters navigation mode, the unified user interface can be updated to include navigation directions, etc., on a combined view of the vehicle and the map.

[0095] The unified user interface (UI) can be updated to reflect information determined from image sensors or other sensors (e.g., radar, ultrasound). For example, if a stop sign is approaching, the UI can highlight the detected stop sign. As another example, the vehicle can automatically stop at a stop sign, and information from the UI can be included to reflect this stop. If a hazard such as a pothole is detected, the UI can be updated to depict the hazard in the combined view. Regarding the driving view described above, the UI can indicate the location of the pothole on the portion of the street the vehicle is approaching. Optionally, the UI can present information instructing the user to adjust their lane.

[0096] Advantageously, when updating the unified user interface, the system can optionally ensure that the combined view of autonomous visualizations (e.g., depictions of vehicles and optionally nearby vehicles and / or objects) and maps is not obscured. For example, the system can cause navigation information to be overlaid on the combined view. As another example, the system can include information determined using cameras and / or sensors in the combined view. As yet another example, the system can present menus or user interfaces associated with vehicle functions separately from the menus or user interface of the combined view. In this example, the system can dynamically resize and / or adjust the position of the combined view. For example, the size of the combined view can be reduced, and if the driver is operating the user interface, the combined view can be positioned on the passenger (e.g., right) side of the user interface.

[0097] Optionally, the unified user interface can present images or videos from cameras located on the vehicle. As will be described in more detail below, the unified user interface can present images or videos that approximate a combined view. These can be triggered to present, for example, if the vehicle is towing something. These can also be triggered if the vehicle is in off-road mode. Examples of presenting images or videos from cameras are included below.

[0098] Figure 2BThis is a flowchart of an example process 210 for updating the unified user interface based on the vehicle's location. For convenience, process 210 will be described as being executed by a system with one or more processors (e.g., context user interface system 100).

[0099] In box 212, the system presents a unified user interface via a display. In box 214, the system receives user input to select navigation controls. As described above, regarding... Figure 1A-Figure 1B A unified user interface enables navigation functionality. In box 216, the system receives a selection of a destination. The user can provide user input indicating the location to navigate to. For example, the user can specify a name associated with the location. The user can also use the search user interface to search for locations.

[0100] In box 218, the system updates the unified user interface to depict a combined view of autonomous visualizations (e.g., graphical representations of vehicles) included on the graphical representation of the map. Figure 1B As shown, the user interface can be updated to reflect route information associated with the route to that location. Therefore, a composite view can reflect a substantially top-down view of a map in which vehicles are depicted. For example, a composite view could represent a rendered view of a real-world environment where a rendered or virtual camera is positioned at a threshold distance above the vehicle.

[0101] In box 220, the system updates the unified user interface based on the vehicle's movement. As described above, the system dynamically adjusts the combined view as the vehicle moves. For example, the unified user interface could depict a driving view in which the rear of the vehicle is shown for navigation along a route. Thus, the driving view could represent a rendered view of a real-world environment in which a rendered or virtual camera is located at the rear of the vehicle.

[0102] The system can update the unified user interface based on updated contextual information, such as updating the vehicle's position relative to the route. For example, when the number of upcoming navigation events is less than a threshold, the system can maintain a driving view of the rear of the vehicle. As an example, if the vehicle is traveling along a highway without any upcoming turns or lane changes, the system can complete the driving view so that it covers most of the display. As another example, if the route indicates a threshold number of upcoming driving events, the system can update the combined view to a zoomed-out state. For example, autonomous visualizations and map information can be zoomed out. In some embodiments, when zooming out autonomous visualizations, certain visualizations (e.g., visualizations of other vehicles, pedestrians, signs, road markings, etc.) can be removed while retaining a graphical depiction of the vehicle. Upcoming navigation events can be identified on the map information. For example, route instructions can be graphically included for these navigation events.

[0103] In some embodiments, the system can zoom in or out based on the vehicle's location and one or more complexity metrics associated with the route. As an example, the system can determine the complexity metric of the upcoming portion of the route within a threshold distance of the vehicle's location or within a threshold travel time of the vehicle's location. Based on the complexity metric, the system can update the combined view. For example, if the upcoming portion of the route has a higher complexity metric, the system can zoom out to depict a greater number of navigation events.

[0104] Complexity metrics can be correlated with the number of upcoming turns, the number of upcoming lane changes, the number of upcoming driving maneuvers (e.g., U-turns), etc. In some embodiments, the above information can be associated with different weights. Thus, the number of upcoming turns can be associated with a higher complexity metric than the number of lane changes. Furthermore, complexity metrics can be correlated with the probability of missing upcoming navigation events. For example, probability can be correlated with historical information. In this example, the complexity metric can increase if the previous driver has missed one or more of the upcoming navigation events.

[0105] Advantageously, if a user selects a vehicle function, such as via a selection icon, the menu associated with that function can be displayed alongside the driving view. In this way, the user can control the function without obstructing the driving view.

[0106] Figure 2C This is a flowchart of an example process 230 for updating the unified user interface based on the selection of vehicle functions. For convenience, process 230 will be described as being executed by a system with one or more processors (e.g., context user interface system 100).

[0107] In box 232, the system presents a unified user interface via a display. In box 234, the system receives user input indicating a specific vehicle function. For example, the unified user interface may include multiple icons associated with different vehicle controls (e.g., air conditioning, music, etc., as described herein). The user can select a specific icon to adjust the corresponding vehicle control. For example, the user can provide touch input to the display. As another example, the user can provide verbal commands for the system or another process or system (e.g., included in or wirelessly communicating with the vehicle) to interpret. As described herein, contextual information can be updated to reflect the selection of a specific vehicle function.

[0108] In box 236, the system adjusts autonomous visualizations (e.g., a graphical representation of the vehicle) within the unified user interface. Certain vehicle functions can adjust the graphical representation of the vehicle. For example, a user can adjust the lights on the vehicle (e.g., fog lights, high beams, etc.). In this example, the system can update the graphical representation accordingly. As another example, a user can adjust the suspension on a truck. In this example, the graphical representation can indicate whether the truck is raised or lowered. As another example, a user can cause a wiper to start, and the representation can show the movement of the wiper. As yet another example, a user can cause the trunk lid to open or close, and the representation can depict this opening and closing.

[0109] In box 238, the system updates the unified user interface to include menus or user interfaces associated with functions. For example, it may include a control-type user interface. As described herein, the menu or user interface may be presented such that it does not obscure the depiction of vehicles and the map. For example, the depiction of vehicles and the map may be resized (e.g., reduced in size). Therefore, the user can interact with the menu or user interface while maintaining a view of the map.

[0110] Figure 3A This is a flowchart of an example process 300 for multitasking control using a unified user interface. For convenience, process 300 will be described as being executed by a system with one or more processors (e.g., context user interface system 100).

[0111] In box 302, the system presents a unified user interface. In box 304, the system receives user input indicating a selection of a first control type. For example, the user of the user interface can select an icon associated with a vehicle function. In box 306, the system receives a selection of a second control type. In response, the system can remove the menu or user interface associated with the first control type. Conversely, it can include a menu or user interface associated with the second control type.

[0112] In box 308, the system updates the unified user interface to include multitasking control. This control can enable quick switching between recently selected vehicle functions. For example, a user can switch between a first control type and a second control type by manipulating the multitasking control. In some embodiments, the multitasking control may represent one or more lines included along the same axis (e.g., horizontal direction). The user can use a swipe gesture to move between different menus or user interfaces associated with the first and second control types.

[0113] In box 310, the system receives user input for multitasking control. User input may include, for example, selection of multitasking control on a touchscreen display. For instance, a user may slide along one or more axes (e.g., a horizontal axis). User input may also include verbal commands associated with multitasking control. As described above, a user can quickly switch between a first control type and a second control type.

[0114] Now refer to Figures 3B-3D , Figures 3B-3D The user interface associated with multitasking control is shown. Figure 3B A user interface 320 is shown, depicting a vehicle 322 along with a music user interface 324. For example, the vehicle may be in parking mode. If the vehicle is moving or navigating, the following description may also apply. For example, a combined view of autonomous visualization and map information may be presented in the user interface 320.

[0115] Figure 3C A user interface 320 is shown, depicting the vehicle 322 together with a quick control user interface 326. For example, the user may have already interacted with the music user interface 324 and either closed it or selected an icon associated with the user interface 326.

[0116] like Figure 3C As shown, this includes multitasking control 328. Each line of the multitasking control line can represent a different user interface. Therefore, the selection of a line (e.g., via input to the line or via swiping) may result in the presentation of different user interfaces.

[0117] For example, the leftmost line could correspond to the quick control user interface 326. As another example, the rightmost line could correspond to the music user interface 324. In some embodiments, the most recently presented user interface could be the user closest to user interface 320. In this example, the driver might be operating user interface 320. Therefore, the most recently used user interface could be the leftmost line. If a passenger is operating user interface 320, the most recently used user interface could be the rightmost line. As shown, the leftmost line is highlighted to indicate that the associated user interface (e.g., user interface 326) is being presented.

[0118] Figure 3DUser interface 320 is shown, in which climate user interface 330 is presented. As shown, multitasking control 328 includes three lines. In some embodiments, user interface 320 may limit the number of lines. Thus, the user can select between a threshold number of recently used vehicle functions. In some embodiments, user interface 320 may include a live view or representation of the relevant vehicle functions above the lines. For example, a live view of the quick control 326 user interface may be included above the center line. In some embodiments, the live view may be presented based on the system identifying that the user is about to interact with user interface 320. For example, an infrared emitter, camera, or other sensor may be used to determine that the user is about to or is interacting with user interface 320.

[0119] Figure 4 This is a flowchart of an example process 400 for adjusting an icon to the driver or passenger side. For convenience, process 400 will be described as being executed by a system with one or more processors (e.g., context user interface system 100).

[0120] At box 402, the system presents a unified user interface via a display. Icons associated with vehicle functions may be displayed on the driver side of the display presenting the unified user interface. In some embodiments, the icons may be displayed on the driver side when the display is on. For example, the display may be on when the driver enters the vehicle. In some embodiments, when the display is on, the system may determine whether the driver or a passenger is in the vehicle. For example, if only a passenger enters the vehicle, the icon may be placed on the passenger side. In this example, the system may use sensors in the seat to determine whether the passenger is seated. The system may also identify that a door on the passenger side is being used. The system may also use cameras or other sensors facing inwards to identify that a passenger is in the vehicle. Similarly, the system may use cameras facing outwards to identify that a passenger has entered or is about to enter the vehicle.

[0121] At box 404, the system identifies whether user input has been received or will be received from the driver or passenger. The system can use an infrared emitter or projector to determine if a hand is moving towards the display. The system can also use a camera to determine hand movement. In some embodiments, machine learning models can be used to distinguish between movements toward the display and movements toward the center of the vehicle (e.g., to access the center console or a beverage).

[0122] In box 406, the system updates the icon to the passenger side. If the system identifies a passenger as providing user input, it can update the icon to their side. Conversely, in box 408, if the user input is provided from the driver side, the system can maintain the icon on the driver side.

[0123] As described above, the selection of the icon can result in the presentation of the associated vehicle control user interface. This user interface can be presented so as not to obscure the combined view of autonomous visualization and map information. For example, the size associated with the combined view can be reduced, and the combined view can be positioned further away from the passenger or driver. As an example, if the system determines that a passenger is providing user input, the system can present the vehicle control user interface on the passenger side. Similarly, the system can present the combined view closer to the driver side.

[0124] Example User Interface

[0125] Example embodiments of a unified user interface are described below. It will be understood that each user interface may describe one or more embodiments, and certain aspects or features of these user interfaces may be adapted and fall within the scope of the disclosure herein. These user interfaces may be presented via a display included in a vehicle. In some embodiments, the display may be a touch-sensitive display. The user interface may be presented by a system or processor included in the vehicle, such as the user interface system 100 described above.

[0126] Figure 5A An example user interface 500 is shown, illustrating a combined view of a parked vehicle. As shown, a graphical depiction 502 of the vehicle is included in the user interface. Additionally, a representation 504 of the environment is included. For example, an autonomous visualization including the graphical depiction 502 and the representation 504 can be rendered. Furthermore, the user interface 500 may optionally include map information. For example, it may include an indication of the street name where the vehicle is parked. As another example, shops, restaurants, etc., may be depicted in the representation 504. These can be obtained, for example, from street view information. Therefore, in some embodiments, contextual information may reflect that the vehicle is parked.

[0127] Optionally, the combined view may include terrain information. For example, a vehicle may be parked while off-roading. In this example, autonomous visualization may include a graphical representation of the terrain surrounding the vehicle. For example, sensor information (e.g., images) of the terrain may be acquired. Map information may be used to inform the terrain information. For example, a terrain map for the vehicle's location may be accessible. User interface 500 may therefore include a terrain representation based on the terrain map. Optionally, sensor information and terrain maps may be combined. As an example, the terrain map may be used to determine the shape associated with the vehicle's location. Sensor information may then inform the shape or other fine-grained information. For example, sensor information may indicate that a part of the terrain is steeper than indicated on the terrain map. This could be helpful for a user driving the vehicle off-road. Therefore, representation 504 may be based on terrain information of the surrounding area.

[0128] As described above, representation 504 may include information determined using cameras or sensors positioned around the vehicle. For example, pedestrians, other vehicles, cones, animals, lights, stop signs, etc., may be determined by systems included in the vehicle. These can be rendered in user interface 500. Similarly, representation 504 may include rocks, slopes, inclines, etc., in the off-road area surrounding the vehicle. In some embodiments, a drone may be stored in the vehicle. The drone can fly out from the vehicle and perform mapping operations on the surrounding terrain. For example, photogrammetry techniques can be used. This information can be provided to the vehicle and used to render representation 504. In some embodiments, the vehicle may at least partially use Simultaneous Localization and Mapping (SLAM) techniques to render representation 504.

[0129] The vehicle depiction 502 includes optional options 506A-506E. Each optional option can be used to operate the vehicle. For example, option 506A can cause the front van to open. Option 506E can cause the trunk lid to open. Similarly, icons 508A-508H can be used to present menus or user interfaces associated with different vehicle functions. For example, icon 508H can cause the presentation of a menu or user interface associated with streaming music. As described above, if icon 508H is selected, the user interface 500 can be updated to reduce the size associated with the vehicle depiction 502 and the associated environment 504. In this way, the user of the user interface can interact with the music streaming application.

[0130] Figure 5B A user interface 510 of a parked vehicle being charged is shown. As illustrated, the vehicle depiction 502 reflects the vehicle's charging status. For example, charging information 510 is included in the user interface. Therefore, in some embodiments, contextual information may reflect that the vehicle is charging.

[0131] Figure 6A An example user interface 600 for a driving view is shown. In this example, autonomous visualization is included in the user interface 600. This visualization may include a graphical depiction 602 of the vehicle along with a representation of an example environment 604 in which the vehicle is being driven. As mentioned above, this example environment 604 may be rendered based on map information (e.g., satellite information, topographic maps, etc.). Environment 604 may also be rendered using cameras and / or sensors located on the vehicle. Therefore, the user of user interface 600 can view impending hazards, changes in gradient, etc.

[0132] In the example shown, the user interface 600 includes routes 605A-605B that the user can follow. In off-road situations, it may be advantageous for the user to quickly view example routes to follow. In this way, these routes can indicate to the user that they should avoid a hazard if one arises. The routes can be determined using sensors and / or cameras on the vehicle. For example, routes 605A-605B can be determined based on existing road markings (e.g., tire markings). As another example, routes 605A-605B can be determined based on the vehicle's width and terrain information. In this example, routes 605A-605B can be positioned such that it causes the vehicle to stop at a distance greater than a threshold distance from a raised or walled section of the terrain. Therefore, routes 605A-605B can indicate a generally flat or drivable section of the upcoming terrain.

[0133] User interface 600 also includes a quick control user interface 606. This quick control user interface 606 can be presented based on the user's selection of a relevant icon 608. While the quick control user interface 606 is being presented, the user interface 606 can be updated to resize the combined view described herein. For example, as shown, the combined view is located in the right-hand portion of the user interface 606 and its size is reduced (e.g., its width is reduced). The user can cause the quick control user interface 606 to disappear, for example, when selecting information within the user interface 606. The combined view can then dynamically increase its size to include more of the user interface 600.

[0134] Although Figure 6A The example shows an off-road environment, but environment 604 can also represent a road environment. For example, it can depict a city road or highway on which the vehicle is traveling.

[0135] Figure 6B An example user interface 600 is shown, where icons 610 are adjusted for the passenger view. At least as in Figure 4 As described above, the position of icon 610 can be updated based on whether a passenger or driver is accessing user interface 600. In the illustrated embodiment, quick control user interface 606 has been moved along with icon 610 to the passenger (e.g., right) side of user interface 600. Similarly, autonomous visualizations (e.g., vehicle 602 and environment 604) have been moved to the driver side.

[0136] Figure 6C An example user interface 600 with a Today user interface 620 is shown. The Today user interface 620 can display weather information, calendar information, etc., as shown in the figure.

[0137] Figure 6D An example user interface 600 with an air conditioning user interface 630 is shown.

[0138] Figure 7A An example user interface 700 for a driving view associated with off-roading is shown. In this user interface 700, autonomous visualizations are shown (e.g., a depiction of the vehicle 702A and the environment 704 in which the vehicle is driving). Additionally, the user interface 700 may include map information informing the topology of the environment 704. In some embodiments, the unified user interface described herein may automatically present off-road information based on detected road conditions. For example, a system or processor included in the vehicle may determine when an off-road condition is detected and cause the user interface 700 to present off-road information. In this example, the off-road condition may be detected based on detected driving conditions. Therefore, in some embodiments, contextual information may reflect the off-road condition.

[0139] For example, driving conditions can be based on metrics associated with vibration, changes in angle associated with the vehicle, tire slippage, etc. Furthermore, sensor information (e.g., image sensors) can be used to determine the type of surface the vehicle is driving on. Therefore, if the surface type is rock, sand, etc., off-road conditions can be detected. As another example, the system can use the vehicle's location to inform the presentation of off-road information. For example, a map can indicate whether the vehicle is outside a known road or on a known remote road associated with off-road conditions.

[0140] Example off-road information may include roll measurements associated with the vehicle. For instance, depiction 702A could be illustrated as navigating around environment 704, as described above regarding autonomous visualization. However, for off-road purposes, depiction 702A may also indicate roll measurements (e.g., "8 degrees"). Figure 7A As shown, this tilt can be depicted graphically. Additionally, a 702B pitch metric can be included. Figure 7A In the example, the spacing 702B is 14 degrees. Sensors in the vehicle can be used to inform these measured angles.

[0141] On the driver's side of the user interface 700, video or images from the image sensor 706 are displayed. The user of the user interface 700 may have selected an icon that triggers the presentation of the image or video. Advantageously, the image or video can be presented without obstructing the driving view. As shown, information related to rock climbing and / or the powertrain can be presented.

[0142] Figure 7B An example user interface 700 is shown, presenting powertrain information 708. This includes, for example, information associated with tire pressure, torque, suspension, etc., for each wheel.

[0143] Figure 8AAn example user interface 800 for a truck trailer is shown. As described herein, the vehicle can be a truck. Advantageously, the unified user interface can present information related to towing the truck trailer. For example, this information can be presented based on the user selecting a towing mode via user interface 800. This information can also be presented based on cameras or sensors in the truck detecting the truck trailer's approach (e.g., behind the truck). This information can also be presented based on the truck trailer providing wireless communication to the vehicle. Therefore, contextual information can indicate that the truck is about to tow the trailer or otherwise connect to it.

[0144] In the example shown, autonomous visualization is included. For example, a graphical depiction 802 of the vehicle is included. Because the current context is associated with the attached trailer, depiction 802 is presented as a top-down view of the vehicle. A representation of trailer 804 is also included. This representation of trailer 804 can be generated using a camera and / or sensors. For example, a model can be generated based on the view of the trailer. Optionally, user interface 800 can provide a pre-rendered version of the trailer. For example, a system included in the vehicle (e.g., system 100) can store a model of the trailer. Therefore, user interface 800 can reflect the position of trailer 804.

[0145] The user interface 800 also includes attachment points 806A-806B for the vehicle and trailer. In some embodiments, a user of the user interface 800 can back up and arrange these attachment points 806A-806B. In some embodiments, the vehicle can be autonomously driven and can automatically reverse. As shown, a trailer can be automatically detected (e.g., “TAILER DETECTED”) and the trailer can also indicate the distance between the vehicle and the trailer (e.g., “4FT AWAY”). This distance can be determined via radar, ultrasonic, and / or image sensors. In some embodiments, a user of the user interface can select an “ATTACH” selectable object to initiate or otherwise cause attachment.

[0146] Images or videos from camera 808 are also included in user interface 800. These can be displayed based on user input using icons included in user interface 800. They can also be displayed when the vehicle is towing or about to tow a trailer.

[0147] Figure 8BAn example user interface 800 including a driving view for towing a trailer is shown. As described herein, contextual information can indicate that the trailer is attached to the vehicle and that the vehicle is being driven. In the illustrated example, autonomous visualization is included. The visualization includes a graphical depiction 802 of the vehicle together with the trailer 804. This depiction can be adjusted in real time. For example, cameras and / or sensors in the vehicle can monitor the trailer. Therefore, any tilting, pushing, sliding, etc., can be presented in the user interface 800. In some embodiments, the trailer may include sensors and / or cameras that can be used to monitor its status. This can be received and used to inform the rendering of the trailer 804 and / or the environment. For example, image sensors behind the trailer can be used to identify vehicles and / or objects behind the trailer.

[0148] In this example, a highway environment 810 is rendered. This includes, for example, highway lanes along with other vehicles. In some embodiments, the size of the trailer may be known, allowing the driver to rely on the user interface 800 to view proximity to other vehicles and / or objects.

[0149] Camera information from image sensor 814 is also included in user interface 800 for easy viewing. As described herein, this camera information may deviate from the driving view of vehicle 802 and environment 810. Also included is towing information 812, such as trailer weight.

[0150] Figure 9 An example user interface 900 for camping mode is shown. In some embodiments, a vehicle can be placed in camping mode. Therefore, contextual information can reflect the camping mode. The unified user interface described herein can be updated to reflect a graphical depiction of vehicle 902 and a graphical depiction of campervan 904. Additionally, vehicle accessories can be graphically displayed. In some embodiments, these accessories can be automatically detected by systems included in the vehicle. For example, they can be detected using cameras on the vehicle. Accessories may also provide wireless information identifying their function or purpose.

[0151] exist Figure 9 In the example, the example accessory is the oven top 906. Through the user interface 900, the user can control various aspects of the oven top 906. For example, the user can instruct the initiation of food cooking. As another example, the user can specify the temperature or measurement associated with heat (e.g., high, medium, etc.). Control information can be routed to the oven top 906 via wired or wireless communication.

[0152] Figure 10AAn example user interface 1000 for vehicle navigation is shown. In this example, contextual information may indicate that the vehicle is navigating towards a location. The example user interface 1000 can be used by a user to follow the navigation direction to a location. The user interface 1000 can also be presented when the vehicle is in autonomous or semi-autonomous mode (e.g., autonomous driving mode).

[0153] The user interface 1000 includes autonomous visualizations (e.g., a graphical depiction of the vehicle 1002 and a highway environment 1004). As shown, environment 1004 indicates carpool lanes, dual lanes, other vehicles, etc. Furthermore, map information is included in the user interface. For example, map information may indicate street names (e.g., "Crenshaw Blvd" 1006), lane directionality, etc. This combined view of autonomous visualizations and map information can represent the driving pattern described above.

[0154] Information 1006 indicating the vehicle's next turn is included in the user interface 1000. At least as... Figure 1A-Figure 1B As described above, during navigation, the unified user interface may optionally present a combined view indicating navigation events. Therefore, the combined view can indicate the next turn the vehicle is about to make. For complex turns, the environment 1004 may optionally zoom out to illustrate a larger portion of the top-down view, such as... Figure 2B As described in [the text].

[0155] Because navigation technology can rely on sensors and / or cameras, the systems included in the vehicle can have precise knowledge of the vehicle's position. For example, the system can identify the exact lane the vehicle is traveling in. Therefore, the user interface 1000 can instruct the vehicle to move into the next lane 1008. As shown, the user interface 1000 illustrates (e.g., animates) the vehicle moving into the next lane 1008. Similarly, the user interface 1000 can indicate the presence of other vehicles. Therefore, if there is a vehicle in the next lane 1008, the user interface 1000 can instruct the vehicle to wait until it is safe to move. In automatic driving mode, the highlighted next lane 1008 can be used to inform the user that the vehicle is changing lanes or is about to change lanes.

[0156] As described above, users can interact with icon 1010 to display menus or user interfaces associated with vehicle functions. When an icon is selected, the size of the combined view of vehicle 1002 and environment 1004 can be reduced.

[0157] Additionally, when in driving mode, users can zoom out of the combined view. For example, users can use the pinch-to-zoom technique to zoom out. The zoomed-out combined view can therefore reflect a larger map area. For example, additional streets, highways, etc., can be reflected in the user interface. Furthermore, the zoomed-out combined view can reduce the size of the vehicle's graphical representation. Vehicles and / or objects nearby can be included in the zoomed-out combined view. For example, if the user zooms out slightly, different vehicles that are not close enough can be rendered. Figure 10A It is not visible in the combined view. Furthermore, the zoomed-out view can highlight most of the route to that location.

[0158] Figure 10B Another example user interface 1000 for vehicle navigation is shown. In the example shown, the vehicle has moved to... Figure 10A The next lane shown is 1008. The user interface 1000 has since been updated to reflect that a vehicle will exit from this lane.

[0159] Figure 10C Another example user interface 1000 for vehicle navigation is shown. In the example shown, the vehicle is approaching the next turn (e.g., an exit). In some embodiments, icons indicating movement may be presented. As another example, lane 1012 is highlighted and indicates the winding path the vehicle should follow. The user interface 1000 can advantageously zoom out to show more of the path (e.g., the curved section 1014 in the upper right corner), allowing the driver to quickly follow the navigation. In some embodiments, upon exiting, the user interface 1000 may partially zoom out to show more of the top-down view. Optionally, the user interface 1000 can zoom out depending on the complexity of an upcoming series of turns. (The above is about...) Figure 2B The techniques used for scaling are described.

[0160] Figures 11A-11C A user interface 1100 for selecting navigation information is shown. For example... Figure 11A As shown, users can search for locations via the navigation user interface 1102. Therefore, contextual information reflects the user's desired navigation. A combined view of autonomous visualization 1104 and map information is also included in the user interface 1100.

[0161] Figure 11B Location selection via navigation user interface 1102 is shown. Summary information 1106 of the location can be presented in user interface 1100. Additionally, the combined view 1104 can be zoomed out to present a graphical depiction 1108 of the location. In this way, the user can view the location of his / her vehicle along with the selected location.

[0162] Figure 11CThe route 1110 to the location is shown. In some embodiments, a summary of the direction 1112 may be included in the user interface 1100. Optionally, this summary may be included when the vehicle is in parking mode or within a threshold time after location selection. Subsequently, the combined view 1104 may be zoomed in to depict a driving view, as described herein. For example, a seamless zoom-in animation may be included in the user interface. In this example, the animation may reflect the position of the rendered or virtual camera from the top perspective to a closer rear view (e.g., as shown in the image). Figure 10A (As shown).

[0163] Facial recognition and passenger tracking

[0164] As described above, in some embodiments, the system included in the vehicle can identify which vehicle passenger (e.g., driver, front passenger) will provide user input to the display. For example, this system could be a contextual user interface system 100. Based on this identification, a unified user interface presented on the display can be updated. For example, icons can be switched between the left and right sides of the display. As described below, the system can adjust other aspects of vehicle operation based on passenger tracking. As an example, the system can use facial recognition technology to learn about vehicle passengers. Passenger preferences and / or profile information can thus be learned by the system. As another example, the system can adjust the rearview mirror, air conditioning, steering wheel, etc., based on the detected passenger position.

[0165] Figure 12 This is a flowchart of an example process 1200 for associating passenger preference information with passengers. For convenience, process 1200 will be described as being executed by a system of one or more computers (e.g., context user interface system 100).

[0166] In box 1202, the system identifies passengers in the vehicle. The system can use cameras and / or sensors to identify when a passenger has entered the vehicle. For example, a sensor could represent an infrared emitter or projector. In this example, if the beam is interrupted, the system can identify that the passenger is seated. As another example, machine learning techniques can be used to identify individuals based on images. For example, convolutional neural networks can be utilized. As yet another example, the system can identify a passenger when the door is open and / or when a seat weight sensor indicates that the passenger is in their seat.

[0167] In box 1204, the system stores information that can be used to identify passenger faces. The system can use deep learning techniques. For example, the system can use one or more cameras located inside or outside the vehicle to acquire images of the passenger's face (e.g., the image can be acquired when the passenger enters the vehicle). The system can then use a deep learning model to encode the passenger's face in an example face space previously learned by the model. Therefore, only a limited number of facial images can be used to uniquely identify the passenger.

[0168] In box 1206, the system receives instructions regarding passenger preferences. Passengers can adjust features such as the reflector, seat position, air conditioning, and music preferences. These preferences, or portions thereof, can be stored by the system.

[0169] In box 1208, the system associates passenger preferences with passengers. The system can store preferences so that they can be used when a passenger enters the vehicle at a later time.

[0170] In all cases where facial recognition technology is used, it should be understood that these additions can be selective. Furthermore, the stored information can be stored locally in the vehicle and can be encrypted.

[0171] Figure 13A This is a flowchart of an example process 1300 for adjusting air conditioning based on passenger tracking. For convenience, process 1300 will be described as being executed by a system of one or more computers (e.g., context user interface system 100).

[0172] In box 1302, the system identifies the vehicle's passengers, such as... Figure 12 As described in section 1304, the system outputs air via a heating, ventilation, and air conditioning (HVAC) system. For example, the system may output air conditioning via an airflow from an HVAC system in a vehicle. The HVAC system may additionally include another flow (e.g., a vertical axis) that can be used to change the height associated with the airflow. Furthermore, the HVAC system may regulate airflow along a horizontal axis. Additional descriptions associated with HVAC systems capable of regulating horizontal and vertical air output are included in U.S. Patent Application No. 15,060,590, the entire contents of which are incorporated herein by reference.

[0173] In box 1306, the system determines the location of a portion of the passenger's body based on an image or video of the passenger. For example, this portion could represent the passenger's face. In some embodiments, the system may have stored facial recognition information for the passenger. In some embodiments, the system can determine the face without facial recognition technology. For example, the system may use a deep learning model or other machine learning model that can be used to segment the face from an image of the passenger.

[0174] In box 1308, the system adjusts air output based on tracking position. As the passenger moves around in his / her seat, such as squatting / sitting / turning his / her head / etc., the system can adjust the air to concentrate it at that position. In this way, air is continuously supplied to the passenger's face. In some embodiments, the system can direct air to the passenger's face if the internal temperature is greater than a threshold. In some embodiments, the system can direct air if the external temperature is high and the passenger has entered the vehicle within a previous threshold time period. In this way, the system can cool the passenger. In some embodiments, a thermal sensor or camera can be used to monitor the passenger's temperature. If the passenger's face is hot, such as due to running or in hot outdoor air, the system can ensure that air is supplied to the passenger's face.

[0175] Optionally, the system can output air to avoid the passenger's face. For example, after a threshold time has elapsed since the air hits the passenger's face, the system can cause air to be directed into the passenger's body. As another example, the system can identify when the internal temperature falls below a threshold and then cause air to be directed into the body.

[0176] Optionally, passengers can indicate whether they prefer air tracking of their face or body, or whether they prefer no tracking at all.

[0177] Figures 13B-13D An example user interface for air conditioning tracking passengers is shown. As shown in these user interfaces, airflow 1310 is being adjusted. User interfaces such as the unified user interface described herein can present this real-time adjustment of air. For example, the adjustment can be presented in an air conditioning control user interface positioned near a combined view of vehicle depiction and map information.

[0178] Figure 14 This is a flowchart of an example process 1400 for adjusting a reflector based on passenger tracking. For convenience, process 1400 will be described as being executed by a system of one or more computers (e.g., context user interface system 100).

[0179] In box 1402, the system identifies passengers in the vehicle. In box 1404, the system determines the position of a portion of the passenger's face based on an image or video of the passenger. For example, the system determines the position of the passenger's eyes. In some embodiments, the system may determine the eye height, the horizontal position of the eyes (e.g., X, Y, Z coordinates), the proximity of the eyes, etc. In some embodiments, the system may determine vectors extending from each eye to indicate the convergence point of the eyes.

[0180] In box 1408, the system adjusts the reflector based on tracking this part. The system can adjust side mirrors, rearview mirrors, etc., based on the passenger's eye position information. For example, the system can adjust the side mirrors to ensure the passenger has a substantially optimal view using them. Similarly, the system can adjust the rearview mirrors as the passenger moves his / her head left or right to ensure a consistent rear view is provided to the passenger.

[0181] In some embodiments, a user interface may be presented suggesting that the passenger adjust his / her position for a better view of the reflector. For example, the passenger seat may be in a position that is not conducive to viewing the reflector. As another example, the passenger may be too short or too tall to have a good view of the reflector from a certain seating position. In some embodiments, the system may automatically adjust the seat (e.g., raise the seat, lower the seat, etc.).

[0182] In addition to adjusting the air conditioning and mirrors, this system can also adjust the steering wheel. For example, the system can adjust the steering wheel based on the passenger's height. This height can be based on the height of the passenger's face or other parts of the passenger's body (e.g., arms, torso, etc.). Therefore, using images and / or videos of the passenger, the system can automatically adjust the steering wheel.

[0183] In some embodiments, the system can use images or videos from inside the vehicle to determine the presence of any animals. For example, when a human leaves the vehicle, the system can automatically trigger a dog mode if it detects the presence of a dog or other animal in the vehicle. In some embodiments, the system can use facial tracking or eye tracking to ensure that the driver is monitoring the road when using autonomous driving technology.

[0184] Other embodiments

[0185] Example embodiments may include methods, systems, and non-transitory computer storage media. One example method is implemented by a system of one or more processors, included in a vehicle, and the method includes: inducing the presentation of a unified user interface via a display of the vehicle, the unified user interface including a combined view aggregating autonomous visualization and map information, the autonomous visualization and map information being associated with a zoom level, wherein the unified user interface is generated based on first contextual information instructing actions of the vehicle; accessing second contextual information instructing subsequent actions of the vehicle; and updating the unified user interface based on the second contextual information, wherein the combined view is adjusted, and wherein adjusting the combined view includes adjusting the size of the combined view in the unified user interface or adjusting the zoom level.

[0186] The above embodiments may include one or more of the following: Autonomous visualization includes a graphical depiction of the vehicle. Autonomous visualization includes a graphical depiction of the vehicle and a graphical representation of the real-world environment in which the vehicle is located. The graphical representation of the real-world environment includes one or more graphical depictions of other vehicles approaching the vehicle. The graphical representation of the real-world environment includes graphical depictions of other vehicles and graphical representations of one or more objects approaching the vehicle. Objects include road markings, stop signs, traffic lights, pedestrians, trash cans, or road signs. A composite view includes a graphical representation of one or more lanes of the road on which the vehicle is located, a graphical depiction of a vehicle in a specific lane of the one or more lanes, and map information, including a graphical representation of a map associated with the road, wherein the graphical depiction of the vehicle is depicted as being on the map. The composite view represents a rendered view of a real-world environment in which a rendered or virtual camera is located at a threshold distance above and behind the vehicle. The composite view represents a driving view, and wherein the graphical depiction of the vehicle is animated as driving in a specific lane. The map indicates the names of the roads. The map indicates different names of different roads approaching the road. The embodiment also includes receiving user input associated with updating the zoom level associated with the combined view; identifying a portion of map information based on the updated zoom level; and updating the combined view, wherein the combined view includes a graphical representation of that portion of the map information. Updating the combined view includes adjusting the size of the autonomous visualization. The autonomous visualization includes a graphical depiction of the vehicle, and wherein the graphical depiction is reduced in size based on the updated zoom level. First contextual information indicates that the vehicle is parked, and wherein second contextual information is associated with control of vehicle functions via a unified user interface. Control of vehicle functions includes control of the heating, ventilation, and air conditioning (HVAC) system, or control of a music application, or control of a navigation user interface. The unified user interface includes multiple icons associated with the respective vehicle functions, wherein updating the unified user interface includes: updating the unified user interface to include a menu associated with a selected icon, wherein the menu is included in a first portion of the unified user interface; and readjusting the size of the combined view, wherein the combined view is included in a second portion not obscured by the menu. The first contextual information indicates that the vehicle is navigating along a route, wherein the combined view further aggregates navigation information, and wherein the second contextual information indicates that the vehicle has moved along the route. The route is associated with multiple driving events indicated in the navigation information, and updating the unified user interface includes: identifying a subset of driving events within a threshold distance of the vehicle's location or within a threshold driving time of the location; determining the zoom level to be adjusted based on the identified subset. The determination of the zoom level to be adjusted is based on one or more complexity metrics associated with the subset of driving events. The determination of the zoom level to be adjusted is also based on the number of identified subsets exceeding a threshold.The size of the autonomous visualization is adjusted based on the adjusted zoom level, and the area associated with the map information is increased based on the adjusted zoom level. The route is associated with multiple driving events indicated in the navigation information, wherein the combined view illustrates a first driving event among the multiple driving events, and the method further includes: adjusting the zoom level, wherein the combined view illustrates multiple second driving events among the multiple driving events. The route is associated with multiple driving events indicated in the navigation information, and updating the unified user interface includes: identifying at least one driving event within a threshold distance of the vehicle's location or within a threshold driving time at that location; determining the zoom level to be adjusted based on the at least one driving event. The at least one driving event indicates a transition from a highway to a surface road. The unified user interface responds to user input causing an adjustment of the zoom level or a panning of the map information in a specific direction. Operation also includes receiving user input causing a panning of the map information in a specific direction, the user input including a swipe gesture indicating the specific direction; accessing a portion of the map information based on the panning; and updating the combined view to present that portion of the map information. Updating the combined view includes: identifying the location of the vehicle not included in the portion of the map information; and updating the combined view to include the portion of the map information, wherein the autonomous visualization includes a graphical depiction of the vehicle, and wherein the graphical depiction of the vehicle is not included in the combined view. Updating the combined view also includes: identifying the location of the vehicle not included in the portion of the map information; adjusting the zoom level such that the updated portion of the map information is identified, wherein the updated portion of the map information includes the location; and updating the combined view based on the adjusted zoom level, wherein the autonomous visualization includes a graphical depiction of the vehicle, and wherein the size of the graphical depiction of the vehicle is reduced in the combined view.

[0187] Example embodiments may also include a system comprising one or more processors and a non-transitory storage medium storing instructions that cause one or more processors to render a user interface for presentation via a display included in a vehicle, wherein the user interface: presents a combined view that unifies autonomous visualization and map information, wherein the autonomous visualization includes a graphical depiction of the vehicle; in response to selection of an icon from a plurality of icons, the icons being associated with control of corresponding vehicle functions, wherein in response to selection, the user interface presents an icon user interface; and dynamically adjusts the combined view such that the icon user interface does not obscure the combined view.

[0188] The above embodiments may include one or more of the following: The display is touch-sensitive, and the user interface responds to touch input. The user interface responds to one or more verbal commands. The instruction causes one or more processors to access contextual information associated with the operation of the vehicle, wherein the contextual information indicates that the vehicle is in a driving mode, and wherein autonomous visualization includes a graphical representation of the real-world environment approaching the vehicle. The graphical representation of the real-world environment includes one or more graphical depictions of other vehicles approaching the vehicle. The autonomous visualization is updated by one or more processors at a threshold frequency. To update the autonomous visualization, the instruction causes one or more processors to: acquire information determined from image sensors located around the vehicle, reflecting positional information about other vehicles approaching the vehicle; access corresponding models associated with the other vehicles; and render the models to be included in a combined view. Control of vehicle functions includes control of heating, ventilation, and air conditioning systems, or control of music applications, or control of the navigation user interface. This instruction causes one or more processors to access contextual information associated with vehicle operation, wherein the contextual information indicates that the vehicle is in navigation mode, wherein the combined view further unifies the navigation information, wherein for the combined view, autonomous visualization includes: a graphical representation of the vehicle's real-world environment, wherein for the combined view, navigation information includes: one or more graphical representations of corresponding driving events at a threshold distance or travel time from the vehicle's position, and wherein for the combined view, map information includes a graphical representation of a portion of the map associated with the vehicle's position. Specific driving events include exiting a highway, and wherein the graphical representation of a specific driving event includes an indication of the number of lanes the vehicle will move over. The graphical representation of a specific driving event includes an animation of the graphical depiction of the vehicle moving across multiple lanes. The zoom level associated with the combined view is increased based on the specific driving event. The size of the graphical depiction of the vehicle is decreased based on the zoom level, and wherein the area of ​​that portion of the map is increased. A graphical representation of the route is included in the combined view, wherein the graphical representation of the route summarizes one or more upcoming driving events associated with that portion of the map. Autonomous visualization includes a graphical depiction of the vehicle in a specific lane among multiple lanes. Dynamically adjusting the combined view includes: reducing the size of the combined view in the user interface, and adjusting the combined view to fit within a first portion of the user interface. An icon-based user interface is located in a second portion of the user interface, wherein the second portion does not obscure the first portion. The user interface responds to user input directed to the icon-based user interface, where the user input causes adjustment of vehicle functions associated with the selected icon; and dynamically adjusts the combined view to substantially contain the user interface, wherein the icon-based user interface is removed from the user interface. The icon-based user interface is associated with attaching a trailer to the vehicle.The icon-based user interface displays an indication of the distance measurement between the trailer and the vehicle, and also shows images or videos of the vehicle acquired from an image sensor. Autonomous visualization includes a graphical depiction of the trailer. The user interface displays the icon on the side of the monitor closest to the user who selected it.

[0189] Example embodiments may include methods, systems, and nontransitory computer storage media. One exemplary nontransitory computer storage medium stores instructions for execution by a system of one or more processors, included in a vehicle, and the instructions cause the one or more processors to perform operations including: causing the rendering of a unified user interface via a display of the vehicle, the unified user interface including a combined view aggregating autonomous visualization, map information, and navigation information, wherein the combined view is associated with a zoom level indicating the area of ​​the real-world environment reflected in the combined view, and wherein the vehicle is in a navigation mode associated with a destination; accessing sensor information from multiple sensors, wherein the sensor information includes the vehicle's position and images from image sensors located around the vehicle; and determining, based on the sensor information, that the zoom level should be adjusted, wherein the combined view is updated to be associated with the adjusted zoom level.

[0190] The above embodiments may include one or more of the following. For the combined view, autonomous visualization includes: a graphical depiction of the vehicle and a graphical representation of the real-world environment surrounding the vehicle, the graphical representation being generated based on images from an image sensor; map information including: a graphical representation of a portion of a map associated with a region of the real-world environment; and navigation information including: one or more graphical representations of driving events associated with a region of the real-world environment. The graphical representation of the real-world environment depicts one or more lanes associated with the road on which the vehicle is traveling, and wherein the graphical depiction of the vehicle is depicted in a specific lane within the one or more lanes. The specific lane is identified based on images from an image sensor. Determining the zoom level to be adjusted based on sensor information includes: identifying a specific number of driving events within a threshold distance or driving time period at the vehicle's location based on navigation information, wherein the determination is based on the specific number of driving events. Determining the zoom level to be adjusted is based on one or more complexity measures associated with the specific number of driving events. Determining the zoom level to be adjusted is based on the specific number of driving events exceeding a threshold. Determining the zoom level to be adjusted based on sensor information includes: identifying upcoming driving events to be skipped based on sensor and navigation information, thus updating the route to the destination; and adjusting the zoom level, wherein the combined view includes a graphical representation of the updated route. Identifying upcoming driving events to be skipped is based on images from an image sensor, wherein the images are analyzed to identify hazards associated with the upcoming driving events. Determining the zoom level to be adjusted based on sensor information includes: identifying the upcoming driving event as a specific type associated with adjusting the zoom level based on sensor and navigation information; and adjusting the zoom level, wherein the combined view includes a graphical representation of one or more driving events associated with the adjusted zoom level. Specific types include exiting a highway onto a surface road. The combined view is updated to depict the route associated with the adjusted zoom level.

[0191] Example embodiments may include methods, systems, and non-transitory computer storage media. An example method implemented by a system of one or more processors, the system communicating with one or more sensors located inside a vehicle, the method comprising: identifying the presence of a passenger inside the vehicle; determining a portion of the passenger to be tracked based on sensor information from the sensors; and controlling the operation of vehicle functions, including air conditioning control, mirror control, and / or steering wheel positioning, based on the tracking of that portion of the passenger.

[0192] The above embodiments may include one or more of the following: Identifying the presence of a passenger based on one or more images acquired from image sensors facing the vehicle interior. Identifying the presence of a passenger based on pressure sensors included in the vehicle seats. Identifying the presence of a passenger based on detecting a user device used by the passenger, wherein the system wirelessly communicates with the user device. Identifying the presence of a passenger based on an interruption of an infrared transmitter pointed at the seat. Operation also includes storing information that can be used to identify the passenger, wherein the information is based on one or more images of the passenger's face. This information includes an encoded vector about a learned vector space, which is generated via a machine learning model. Operation also includes generating a profile associated with the passenger, wherein the profile is associated with information that can be used to identify the passenger, and wherein the profile indicates the passenger's preferences regarding the operation of vehicle functions. Determining a portion of the passenger to be tracked based on one or more images of the passenger from image sensors facing the vehicle interior. Operations controlling vehicle functions include: adjusting the heating, ventilation, and air conditioning (HVAC) system such that air output is maintained toward that portion of the passenger. Operations controlling vehicle functions include: adjusting one or more mirrors based on that portion of the passenger, wherein that portion includes the passenger's eyes. The operation also includes the method further including causing output to a user interface via a display in the vehicle, wherein the user interface presents recommendations instructing the passenger to adjust the position of the passenger's head.

[0193] It should be understood that not all objectives or advantages may be achieved according to any particular embodiment described herein. Therefore, for example, those skilled in the art will recognize that certain embodiments may be configured to operate in a manner that achieves or optimizes one or more advantages as taught herein, without necessarily achieving other objectives or advantages as taught or suggested herein.

[0194] All processes described herein can be embodied in software code modules executed by a computing system comprising one or more general-purpose computers or processors, and are fully automated via such software code modules. The code modules can be stored on any type of non-transitory computer-readable medium or other computer storage device. Some or all of these methods can alternatively be embodied in specialized computer hardware. Furthermore, the components referred to herein can be implemented in hardware, software, firmware, or a combination thereof.

[0195] In addition to the variations described herein, many other variations will be apparent from this disclosure. For example, according to embodiments, certain actions, events, or functions of any algorithm described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are necessary for algorithmic practice). Furthermore, in some embodiments, actions or events may be performed concurrently, for example, through multithreaded processing, interrupt handling, or multiple processors or processor cores or other parallel architectures, rather than sequentially. Moreover, different tasks or processes may be performed by different machines and / or computing systems that can run together.

[0196] The various illustrative logic blocks, modules, and algorithmic elements described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, and elements have been described above generally according to their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the system as a whole. The described functionality may be implemented differently for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

[0197] The various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein can be implemented or executed by a machine designed to perform the functions described herein, such as a processing unit or processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The processor may be a microprocessor, but alternatively, it may be a controller, a microcontroller, or a state machine, a combination thereof, etc. The processor may include a circuit system configured to process computer-executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Although described primarily with regard to digital technologies herein, the processor may also primarily comprise analog components. For example, some or all of the signal processing algorithms described herein may be implemented in an analog circuit system or a mixed analog and digital circuit system. Computing environments can include any type of computer system, including but not limited to microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within appliances, to name just a few.

[0198] Elements of the methods, processes, or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in software modules stored in one or more memory devices and executed by one or more processors, or in a combination of both. The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD ROM, or any other form of non-transitory computer-readable storage medium known in the art, or in physical computer storage. Example storage media may be coupled to a processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The storage medium may be volatile or non-volatile.

[0199] Unless otherwise expressly stated, conditional language such as “can,” “could,” “might,” or “may” is understood in the context to generally convey that certain embodiments include certain features, elements, and / or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, elements, and / or steps are included or will be performed in any particular embodiment, with or without user input or prompting.

[0200] Unless otherwise explicitly stated, disjunctive language such as the phrase “at least one of X, Y, or Z” is understood in the context to generally indicate that an item, term, etc., may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0201] Any process description, element, or block described herein and / or depicted in the accompanying drawings should be understood as potentially representing a code module, segment, or portion comprising one or more executable instructions for implementing a particular logical function or element within the process. Alternative implementations are included within the scope of the embodiments described herein, wherein elements or functions may be omitted, performed, or discussed in a different order than that shown (including substantially simultaneously or in reverse order), depending on the functionality involved, as will be understood by those skilled in the art.

[0202] Unless otherwise explicitly stated, articles such as “a” or “an” should generally be interpreted as including one or more of the described items. Therefore, phrases such as “device configured to…” are intended to include one or more of the listed devices. Such one or more listed devices may also be configured collectively to perform the enumeration. For example, “processors configured to perform enumerations A, B, and C” could include a first processor configured to perform enumeration A working in conjunction with a second processor configured to perform enumerations B and C.

[0203] It should be emphasized that many variations and modifications can be made to the above embodiments, and the elements therein should be understood as other acceptable examples. All such modifications and variations are intended to be included within the scope of this disclosure.

Claims

1. A method implemented by a system comprising one or more processors, said system being included in a vehicle, and said method comprising: This causes the presentation of a unified user interface via the vehicle's display, the unified user interface including a composite view that aggregates autonomous visualization and map information, the autonomous visualization and map information being associated with a zoom level, wherein the unified user interface is generated based on first contextual information instructing the operation of the vehicle; Access to second contextual information indicating subsequent actions of the vehicle, wherein the first contextual information indicates that the vehicle is navigating along a route, wherein the combined view further aggregates navigation information for the vehicle, and wherein the second contextual information indicates that the vehicle has moved along the route; as well as The unified user interface is updated based on the second context information, wherein the combined view is adjusted, and adjusting the combined view includes adjusting the zoom level; The system determines a complexity metric for the upcoming portion of the route within a threshold distance from the vehicle's location or within a threshold travel time from the vehicle's location. If the upcoming portion of the route has a high complexity metric, the system adjusts the zoom level to reduce the view to depict a greater number of navigation events. The complexity metric mentioned here is related to the likelihood of missing an upcoming navigation event. The complexity metric increases if the vehicle’s previous driver missed one or more of the upcoming navigation events.

2. The method of claim 1, wherein the autonomous visualization comprises a graphical depiction of the vehicle and a graphical representation of the real-world environment in which the vehicle is located.

3. The method of claim 2, wherein the graphical representation of the real-world environment includes graphical depictions of other vehicles and graphical representations of one or more objects approaching the vehicles.

4. The method of claim 1, wherein the combined view comprises: A graphical representation of one or more lanes of the road on which the vehicle is located. A graphical depiction of the vehicle located in a specific lane of the one or more lanes, and Map information, which includes a graphical representation of a map associated with the road, wherein the graphical depiction of the vehicle is shown on the map.

5. The method of claim 4, wherein the combined view represents a driving view, and wherein the graphical depiction of the vehicle is animated to indicate that it is driving in the specific lane.

6. The method according to claim 1, wherein the method further comprises: Receive and update user input associated with the zoom level of the combined view; A portion of the map information is identified based on the updated zoom level; as well as Update the combined view, wherein the combined view includes a graphical representation of the portion of the map information.

7. The method of claim 1, wherein the autonomous visualization comprises a graphical depiction of the vehicle, and wherein the graphical depiction is reduced in size based on the adjusted scaling level.

8. The method of claim 1, wherein the second context information is associated with control of vehicle functions via the unified user interface, wherein control of vehicle functions includes control of heating, ventilation and air conditioning systems, or control of music applications, or control of navigation user interfaces.

9. The method of claim 8, wherein the unified user interface includes a plurality of icons associated with corresponding vehicle functions, wherein updating the unified user interface includes: Update the unified user interface to include a menu associated with the selected icon, wherein the menu is included in a first part of the unified user interface; as well as Resize the combined view, which is included in a second part that is not obscured by the menu.

10. The method of claim 1, wherein the first context information indicates that the vehicle is navigating along a route, wherein the combined view further aggregates the navigation information, and wherein the second context information indicates that the vehicle has moved along the route.

11. The method of claim 10, wherein the route is associated with a plurality of driving events indicated in the navigation information, and wherein updating the unified user interface comprises: A subset of the driving events identified within a threshold distance of the vehicle's location or within a threshold driving time at that location; Based on the identified subset, it is determined that the scaling level should be adjusted.

12. The method of claim 11, wherein the size of the autonomous visualization is adjusted based on the adjusted zoom level, and wherein the area associated with the map information is increased based on the adjusted zoom level.

13. The method of claim 10, wherein the route is associated with a plurality of driving events indicated in the navigation information, wherein the combined view illustrates a first driving event among the plurality of driving events, and wherein the method further comprises: Adjust the zoom level, wherein the combined view illustrates multiple second driving events among the multiple driving events.

14. The method of claim 10, wherein the route is associated with a plurality of driving events indicated in the navigation information, and wherein updating the unified user interface comprises: Identify at least one driving event within a threshold distance of the vehicle's location or within a threshold driving time at the location; Based on the at least one driving event, it is determined that the zoom level needs to be adjusted.

15. The method of claim 1, wherein the unified user interface responds to user input causing adjustment of the zoom level or translation of the map information in a particular direction.

16. A system comprising one or more processors and a non-transitory storage medium storing instructions, the instructions causing the one or more processors to render a user interface for presentation via a display included in a vehicle, wherein the user interface: Presenting a combined view that unifies autonomous visualization and map information, wherein the autonomous visualization includes a graphical depiction of the vehicle, and wherein the autonomous visualization and map information are associated with a zoom level; In response to the selection of an icon from a plurality of icons, the icon is associated with the control of the corresponding vehicle function, wherein the user interface presents an icon-based user interface in response to the selection; as well as The combined view is dynamically adjusted so that the icon user interface does not obscure the combined view. The system determines a complexity metric for the upcoming portion of the route within a threshold distance from the vehicle's position or within a threshold travel time at the vehicle's position, along which the vehicle is navigating. If the upcoming portion of the route has a high complexity metric, the system adjusts the zoom level to reduce the view to depict a greater number of navigation events. The complexity metric mentioned here is related to the likelihood of missing an upcoming navigation event. The complexity metric increases if the vehicle’s previous driver misses one or more of the upcoming navigation events.

17. The system of claim 16, wherein the instructions cause the one or more processors to access context information associated with the operation of the vehicle, wherein the context information indicates that the vehicle is in a driving mode, and wherein the autonomous visualization includes a graphical representation of a real-world environment approximating the vehicle.

18. The system of claim 17, wherein the graphical representation of the real-world environment includes one or more graphical depictions of other vehicles approaching the vehicle.

19. The system of claim 17, wherein the autonomous visualization is updated by the one or more processors at a threshold frequency.

20. The system of claim 19, wherein, for updating the autonomous visualization, the instruction causes the one or more processors to: Acquire information determined from image sensors positioned around the vehicle, the information reflecting the location information of other vehicles approaching the vehicle; Access the corresponding model associated with the other vehicles; and The model is rendered for inclusion in the combined view.

21. The system of claim 16, wherein control of vehicle functions includes control of heating, ventilation and air conditioning systems, or control of music applications, or control of navigation user interfaces.

22. The system of claim 16, wherein the instruction causes the one or more processors to access context information associated with the operation of the vehicle, wherein the context information indicates that the vehicle is in a navigation mode, and wherein the combined view further unifies the navigation information. Wherein, for the combined view, the autonomous visualization includes: A graphical representation of the real-world environment of the vehicle. For the combined view, the navigation information includes: One or more graphical representations of corresponding driving events within a threshold distance of the vehicle's location or during a driving time. Furthermore, for the combined view, the map information includes a graphical representation of a portion of a map associated with the location of the vehicle.

23. The system of claim 22, wherein a particular driving event includes leaving a highway, and wherein the graphical representation of the particular driving event includes an indication of the number of lanes the vehicle will move over.

24. The system of claim 23, wherein the zoom level associated with the combined view is increased based on the specific driving event, and wherein the size of the graphical depiction of the vehicle is reduced based on the zoom level, and wherein the area of ​​the portion of the map is increased.

25. A non-transitory computer storage medium storing instructions for execution by a system comprising one or more processors, the system being included in a vehicle, and the instructions causing the one or more processors to perform operations, the operations comprising: This causes the presentation of a unified user interface via the vehicle's display, the unified user interface including a combined view that aggregates autonomous visualization, map information and navigation information, wherein the combined view is associated with a zoom level so that the zoom level indicates the area of ​​the real-world environment reflected in the combined view, and wherein the vehicle is in a navigation mode associated with a destination. Access sensor information from multiple sensors, wherein the sensor information includes the vehicle’s position and images from image sensors located around the vehicle; as well as The system determines a complexity metric for the upcoming portion of the route within a threshold distance from the vehicle's position or within a threshold travel time at the vehicle's position, along which the vehicle is navigating. If the upcoming portion of the route has a high complexity metric, the system adjusts the zoom level to reduce the view to depict a greater number of navigation events. The complexity metric mentioned here is related to the likelihood of missing an upcoming navigation event. The complexity metric increases if the vehicle’s previous driver misses one or more of the upcoming navigation events.

26. The computer storage medium of claim 25, wherein, for the combined view, The autonomous visualization includes: The graphic depiction of the vehicle, and A graphical representation of the vehicle's real-world environment, generated based on images from the image sensor. The map information includes: A graphical representation of a portion of a map associated with the area of ​​the real-world environment, and The navigation information includes: One or more graphical representations of driving events associated with the area of ​​the real-world environment.

27. The computer storage medium of claim 26, wherein the graphical representation of the real-world environment depicts one or more lanes associated with a road on which the vehicle is traveling, and wherein the graphical representation of the vehicle is depicted in a particular lane of the one or more lanes.

28. The computer storage medium of claim 25, wherein determining that the scaling level is to be adjusted based on the sensor information comprises: Based on the navigation information, a specific number of driving events are identified within a threshold distance or during a driving time at the vehicle's location. The determination is based on a specific number of the driving events.

Images