Improved situation-sensing user interface for vehicle operation

Abstract

A system and method for a context-sensitive user interface for improved vehicle operation. An exemplary method is implemented by a system of one or more processors, the system being included in a vehicle. The method includes causing presentation, via a display of the vehicle, of an integrated user interface including a combined view aggregating autonomous visualization and map information, the autonomous visualization and map information being associated with a magnification level, the integrated user interface being generated based on first context information indicative of operation of the vehicle. Second context information indicative of subsequent operation of the vehicle is accessed. The integrated user interface is updated based on the second context information, and the combined view is adjusted, where adjusting the combined view includes adjusting a size of the combined view in the integrated user interface or adjusting a magnification level.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 62 / 938,842, filed on November 21, 2019, entitled "CONTEXT SENSITIVE USER INTERFACE FOR ENHANCED VEHICLE OPERATION", and U.S. Provisional Patent Application No. 62 / 938,769, filed on November 21, 2019, entitled "ENHANCED VEHICLE FUNCTIONALITY VIA PASSENGER RECOGNITION", the disclosures of which are hereby incorporated by reference in their entireties.

[0002] This disclosure relates to user interfaces, and more particularly, to user interfaces for vehicle operation.

Background Art

[0003] Displays used to present digital user interfaces have come to be included in vehicles for vehicle control and / or operation. For example, a display may be included in a vehicle's dashboard. In this example, the display may present a user interface including the vehicle's current speed, the total distance the vehicle has traveled, the temperature outside the vehicle, etc. As another example, a display may be included in the central part of the vehicle. This display may be used for presenting navigation information, controlling air conditioning, etc. Thus, such digital user interfaces can be used in place of mechanical displays, buttons, knobs, etc.

[0004] However, as digital user interfaces (CUITS) expand to encompass many vehicle controls and / or operations, they are becoming increasingly complex. For certain CUITS, this complexity reduces the ease of use associated with the interface, potentially degrading the user experience related to operating the vehicle. For example, certain functions may be hidden between various levels of menus. Another example is that users may need to memorize complex user input sequences to traverse the CUITS. Therefore, providing an intuitive CUITS can enhance the user's enjoyment when operating the vehicle. [Brief explanation of the drawing]

[0005] Many of the aforementioned embodiments and associated advantages will be readily apparent, as they will be best understood by referring to the following detailed description in conjunction with the attached drawings.

[0006] [Figure 1A] This is a block diagram of an exemplary user interface system that presents a user interface.

[0007] [Figure 1B] This is a block diagram of an exemplary situational user interface system that presents an updated user interface based on received user input.

[0008] [Figure 2A] This is a flowchart illustrating an example process for updating an integrated user interface.

[0009] [Figure 2B] This is a flowchart illustrating an exemplary process for updating the integrated user interface according to the vehicle's configuration.

[0010] [Figure 2C] It is a flowchart diagram of an exemplary process for updating an integrated user interface based on the selection of vehicle functions.

[0011] [Figure 3A] It is a flowchart diagram of an exemplary process for using multitask control of an integrated user interface.

[0012] [Figure 3B] It is a diagram of a user interface that presents a depiction of a vehicle along with a music user interface.

[0013] [Figure 3C] It is a diagram of a user interface that presents a depiction of a vehicle along with a quick control user interface.

[0014] [Figure 3D] It is a diagram of a user interface having a presented climate user interface.

[0015] [Figure 4] It is a flowchart diagram of an exemplary process for adjusting an icon on the driver side or the passenger side.

[0016] [Figure 5A] It is a diagram of an exemplary user interface of a parked vehicle.

[0017] [Figure 5B] It is a diagram of an example of a user interface showing a parked vehicle while the vehicle is being charged.

[0018] [Figure 6A] It is a diagram of an exemplary user interface of a driving view.

[0019] [Figure 6B]FIG. is an illustration of an exemplary user interface of a driving view having icons adjusted for a passenger view.

[0020] [Figure 6C] FIG. is an illustration of an exemplary user interface of a driving view having a current user interface.

[0021] [Figure 6D] FIG. is an illustration of an exemplary user interface of a driving view having an air conditioning user interface.

[0022] [Figure 7A] FIG. is an illustration of an exemplary user interface of a driving view while the vehicle is being driven in an "off-road" environment.

[0023] [Figure 7B] FIG. is an illustration of an exemplary user interface of a driving view while driving the vehicle in an "off-road" environment and presenting driveline information.

[0024] [Figure 8A] FIG. is an illustration of an exemplary user interface of a vehicle towing a tractor-trailer.

[0025] [Figure 8B] FIG. is an illustration of an exemplary user interface of a driving view of a vehicle when towing a trailer.

[0026] [Figure 9] FIG. is an illustration of an exemplary user interface of a vehicle while in camper mode.

[0027] [Figure 10A] FIG. is an illustration of an exemplary user interface of a driving view for vehicle navigation.

[0028] [Figure 10B]This is a diagram of another example user interface for the driving view.

[0029] [Figure 10C] This is a diagram of another example user interface for the driving view.

[0030] [Figure 11A] This is a diagram of a user interface for selecting navigation information. [Figure 11B] This is a diagram of a user interface for selecting navigation information. [Figure 11C] This is a diagram of a user interface for selecting navigation information.

[0031] [Figure 12] This is a flowchart illustrating an exemplary process for associating passenger preference information with specific passengers.

[0032] [Figure 13A] This is a flowchart illustrating an exemplary process of adjusting the air conditioning based on passenger tracking.

[0033] [Figure 13B] This is a diagram illustrating an exemplary user interface for an air conditioning system that tracks passengers. [Figure 13C] This is a diagram illustrating an exemplary user interface for an air conditioning system that tracks passengers. [Figure 13D] This is a diagram illustrating an exemplary user interface for an air conditioning system that tracks passengers.

[0034] [Figure 14] This is an illustrative flowchart of the process of adjusting mirrors based on the passenger's eye-tracking. [Modes for carrying out the invention]

[0035] This specification describes improved user interfaces that can be used to control and / or operate a vehicle, with respect to several embodiments. The user interfaces described herein are examples of digital user interfaces rendered by a processor or system and presented via a display located within the vehicle. In some embodiments, the display may be located in the central part of the vehicle. For example, the display may be larger than a certain diagonal size (e.g., 10 inches, 13 inches, 15 inches, 17 inches, 21 inches, etc.). In some embodiments, since the display may be centrally located and larger than a certain diagonal size, the display may provide a centralized interface for controlling and / or operating the vehicle. Thus, making the centralized interface easy to understand and use can improve the usability of the vehicle. As will be discussed later, the improved user interfaces dynamically integrate (e.g., aggregate) autonomous visualizations (e.g., visualizations generated from sensors around the vehicle), map information, navigation information, and / or vehicle function controls.

[0036] One embodiment of the present invention is a vehicle having a user interface (hereinafter referred to as an integrated user interface) that can be dynamically updated based on contextual information relating to the operation of the vehicle. For example, the contextual information may indicate that the vehicle is parked or being driven. Another example is that the contextual information may indicate that the vehicle is traveling toward a destination and / or is being controlled autonomously or semi-autonomously. Another example is that the contextual information may indicate that the vehicle is towing a trailer or driving off-road. Another example is that the contextual information may relate to the control of vehicle functions. The integrated user interface includes an easy understanding of interactive elements that respond to the current contextual information, as will be described later.

[0037] With respect to a parked vehicle, the integrated user interface may present an autonomous visualization, including a graphic depiction of the vehicle, which can be selected using a touchscreen interface. Vehicle controls may be included in the graphic depiction, such as allowing the end user to open the trunk, open and close the windows, or detach or connect a trailer. In some embodiments, the autonomous visualization may include a graphic representation of the real-world environment in which the vehicle is located. With respect to an autonomously or semi-autonomously controlled vehicle, the integrated user interface may present an autonomous visualization overlaid on map information. For example, the autonomous visualization may include a representation of the surface on which the vehicle is being driven. In this example, the graphic depiction of the vehicle may be shown on a specific lane of a road, along with graphic representations of other vehicles or objects in close proximity to the vehicle. Examples may include representations of lanes, highway exits or entrances, pedestrians, hazards, signs, or other objects. The map information may include a graphic representation of a portion of the map on which the vehicle is being driven. For example, the map information may indicate the number of lanes on the current road, other roads adjacent to the current road, or other map-based information. For example, a road may be represented along with a road identifier.

[0038] For a vehicle traveling toward a destination, the integrated user interface may include autonomous visualizations, map information, and navigation information. As described later, the integrated user interface may graphically depict navigation events identified by the navigation information (e.g., the next turn, highway exit, highway entrance, etc.). For example, the integrated user interface may graphically depict a vehicle moving from its current lane to a different lane. Furthermore, the integrated user interface may enlarge or reduce the presented information depending on the context. For example, a magnification level may be set to adjust the size of autonomous visualizations, map information, and navigation information.

[0039] For example, when a vehicle exits a highway, the information presented may be reduced in size. In this example, as a portion of the map information representing a large real-world geographic area (e.g., many roads may be shown) becomes relatively large, the autonomous visualization (e.g., the graphic depiction of the vehicle) may shrink. Similarly, upcoming navigation events, such as navigation routes, may be reduced in size so that a large portion of the route is shown. In this way, the end user may be presented with the next turn on a ground road. After the user has exited the highway and is driving on a ground road, the presented information may be optionally enlarged.

[0040] Additional contextual information may relate to the control of vehicle functions, such as music playback, climate control adjustment, video streaming, or overlaying video acquired by one or more cameras. As is understood, the control of such vehicle functions may consume various parts of the user interface. For example, to adjust climate settings, the user interface may present climate control. As will be discussed later, the integrated user interface may ensure that autonomous visualizations, map information, and / or navigation information do not obstruct the view. For example, an end user may select a user interface element to control climate settings while driving toward a destination. In this example, the integrated user interface may coordinate the presentation of autonomous visualizations, map information, and / or navigation information and include climate control, or an overlay of climate control on the navigation display. Exemplary techniques for coordinating the presentation of information to passengers may include moving autonomous visualizations, map information, and / or navigation information to different parts of the integrated user interface (e.g., further away from the end user). Another exemplary technique is to overlay specific controls on top of map or navigation information.

[0041] Therefore, situational information may relate to the operation of the vehicle. While exemplary situational information is described herein, other information may be used and should be understood to be included within the scope of this disclosure. For example, situational information may indicate that the vehicle is being navigated to an arbitrary location. In this example, the situational information may reflect the vehicle's position relative to the arbitrary location. As an example, the arbitrary location may indicate a future or previous driving event (e.g., a turn, a lane change). Therefore, when the vehicle is being navigated to an arbitrary location, the situational information may be updated based on the arbitrary location. In this way, the user interface may be updated based on the arbitrary location (e.g., the presented map may be zoomed out), as will be described later.

[0042] The above description focuses on the user interface presented on the display, but in some embodiments, multiple displays may be used. For example, the first display may be positioned close to or in front of the driver of the vehicle. In this example, the second display is positioned in the central part of the vehicle or further away from the driver. As an example, the vehicle represents a semi-truck with two or more displays.

[0043] First, an exemplary user interface is described. This exemplary user interface may provide centralized control of the vehicle. However, this exemplary user interface may obstruct important information, such as maps and / or navigation information indicating the vehicle's location, during the everyday use of the user interface. As will be discussed later, an integrated user interface may improve upon this exemplary user interface. It should be understood that the selection of controls or other information on the user interface may also be done via a touch interface or other well-known user interface mechanisms.

[0044] • Exemplary user interface An exemplary user interface for centralized vehicle control may be separated into a first and a second part. The first part may include, for example, a graphic representation of the vehicle, which may be updated substantially in real time based on the operation of the vehicle. For example, if the vehicle is braking, the graphic representation may be updated to depict brake lights. As another example, if the lights on the front of the vehicle, such as hazard lights, are on, the graphic representation may be updated accordingly. The second part may include a graphic representation of a map. For example, the map may include icons such as arrows indicating the vehicle's position on the map. In this example, when the vehicle is being driven, the icons may be updated to reflect the vehicle's substantially real-time positioning.

[0045] Therefore, the first part can provide a concise view of real-time conditions related to vehicle operation. The user interface may, for example, display lane markings in this first part. Thus, the user can quickly see whether or not they are properly following the lane markings. Furthermore, autonomous operation by the vehicle is possible. Therefore, when in autonomous mode, the user can look at the first part to ensure that the vehicle is properly recognizing the lane markings.

[0046] The second section may require the presentation of local conditions relevant to vehicle operation. For example, the second section may show traffic conditions for roads included on the map. In this example, the user can quickly check whether there are accidents on highways or main roads, whether traffic is heavy, etc. The second section may further present navigation information. For example, the route to a specific destination may be shown on the map along with instructions for the next turn. In this way, the user can quickly identify how to reach that destination.

[0047] This exemplary user interface may further include the display of numerous icons or other graphic representations associated with different vehicle functions controllable through the user interface. Exemplary vehicle functions may include, for example, climate control, music, monitoring of electric vehicle energy usage, driving control, and autonomous driving control. To adjust a vehicle function, such as adjusting climate control settings, the user may select an icon. As an example, the user interface may be presented via a touchscreen display. In this example, the user may touch a portion of the display depicting the icons. As another example, the user interface may respond to verbal commands. In this example, the user may provide a verbal command indicating the type of vehicle function (e.g., "climate control settings").

[0048] When a user selects an icon, menu, or other user interface element, it may be presented with elements associated with the type of vehicle function. Since the user interface is divided into two parts, the presented menu may be overlaid on at least one of the parts. For example, if a user selects an icon related to music, the user interface may be updated to a music selection menu. It will be understood that the menu may obscure at least one of the two parts. As an example, the menu may be overlaid on a map. Therefore, at least a portion of the map may be obscured by this music menu. The user may have to remove the menu to see the entire map. For example, the user may swipe the menu with a downward motion.

[0049] Therefore, during the routine operation of the exemplary user interface, information contained in the first and / or second parts may be masked by controlling vehicle functions. This may be disadvantageous while the vehicle is in motion. For example, the second part may show a route to be followed on a presented map. If the user prefers to select updated climate control settings, adjust the music, view energy tracking information, etc., at least a portion of the shown route may be hidden.

[0050] • Integrated user interface As will be discussed later, a system or processor rendering an integrated user interface may combine autonomous visualizations (e.g., graphic representations of the vehicle) with map information. This combination may be referred to herein as a combined view and may be presented in such a way that the user has a consistent view of at least the autonomous visualizations and map information. This view is advantageously not obstructed by menus or user interfaces associated with types of vehicle functions. As will be discussed later, menus for vehicle functions may optionally be presented dynamically, separate from the combined menus of autonomous visualizations and map information.

[0051] Furthermore, the system or processor may dynamically update the combined view according to subsequent operating events. For example, the system may adjust the magnification level associated with the combined view. In this example, the magnification level may indicate the rendering camera or virtual camera associated with the combined view. As an example, a high magnification level may represent a rendering camera or virtual camera that encompasses a wide area of ​​map information. Therefore, the size of the autonomous visualization may be reduced accordingly.

[0052] • Integrated user interface - combined autonomous visualization and map information In some embodiments, the system or processor may update the integrated user interface according to current situation information related to the operation of the vehicle. For example, if the vehicle is in a parking lot, the integrated user interface may display a large graphic representation of the vehicle. Thus, the graphic representation may substantially fill the integrated user interface. This graphic representation may include interactive choices associated with the control of the vehicle. For example, the vehicle may be an electric truck, and its choices may include adjusting the suspension, opening and closing the charging port, opening and closing the tonneau cover, opening and closing the tailgate, etc. If the user selects to adjust the suspension, the integrated user interface may be updated to reflect the various suspension levels that can be selected.

[0053] The large graphic representation of the vehicle described above may be included in a magnified view of the map. For example, the integrated user interface may optionally reflect environmental information close to the vehicle. The graphic representation may optionally reflect a third-person view of the vehicle, such as reflecting a virtual or rendering camera positioned above and behind the vehicle. In some embodiments, the graphic representation may be presented in a street view of the map. In some embodiments, the graphic representation may optionally be presented on a road representation along with the names of roads and / or nearby roads. The user of the integrated user interface may provide user input for zooming out the map. For example, the user may use pinch / zoom techniques to zoom out the map. Since the graphic representation of the vehicle is placed within the map, the graphic representation may shrink in size as the map is zoomed out. In this way, the user can see a large map showing a large area around the vehicle. Furthermore, the user may trigger a transition of map information via user input. Exemplary user input may include swiping along a particular direction, thereby adjusting the map to reflect the drawn area. Furthermore, vehicle graphics may be removed from the combined view if they are not included in the adjusted area. Optionally, the map may be scaled down so that the combined view includes the area containing the vehicle graphics (e.g., based on its position) and the area encompassed by the shifted area (e.g., the adjusted area).

[0054] Situational information may be updated by the user based on a map. For example, the user may select any location on the map. In this example, the integrated user interface may present a path that the vehicle could take to reach the chosen location. Thus, the integrated user interface may present the path as extending from the vehicle's graphic representation to the chosen location.

[0055] When a user is driving a vehicle, the system or processor may update the integrated user interface to present a larger graphical view of the vehicle. Thus, situational information may be updated to reflect that the user is driving according to navigation information. For example, as described below, the integrated user interface may present an autonomous visualization including a driving view of the vehicle through the real-world environment. The driving view may reflect a rearward-rising view of the vehicle driving towards any given location through the environment. For example, lane markings, other vehicles, etc., may be rendered in the integrated user interface. The integrated user interface may further present directions for each corner. As an example, a left-turn arrow may be included before prompting the user to turn left onto a street.

[0056] In this way, in contrast to the exemplary user interface described above, the vehicle's graphic representation can be combined with map information. Therefore, the integrated user interface can dynamically adjust the presentation of the graphic representation and / or map information in virtually real time. For example, the map can be dynamically zoomed in and out.

[0057] The vehicle may optionally include cameras or sensors that can be used to determine the positions of other vehicles, lane positions, highway exits, highway entrances, stop signs, traffic lights, etc., in substantially real time. Therefore, the integrated user interface may indicate that while driving, the user needs to move across a certain number of lanes in order to turn or exit a highway.

[0058] In one embodiment, the integrated user interface may present an autonomous visualization of obstacles at highway exits. To ensure the user notices the obstacles, the obstacles may optionally be highlighted or otherwise invoked. In some embodiments, the integrated user interface may be updated to present an updated navigation route that avoids the highway exit.

[0059] While the above description focuses on navigation, it should be understood that such obstacles may also be presented in the integrated user interface during normal vehicle operation. For example, the integrated user interface may present a large graphic representation of the vehicle when parked. When the user is driving the vehicle, the situation information may be updated to reflect that the vehicle is being driven. Thus, the integrated user interface may be updated to present an autonomous visualization that reflects the driving view of the vehicle. For example, the driving view may include a rear-elevated view of the vehicle traversing a street or off-road area. The driving view may further show lane markings, other vehicles, stop signs, streetlights, pedestrians, potholes, rocks, obstacles or hazards, etc. Furthermore, the integrated user interface may include map information such as street names, the location of stop signs or traffic lights, and the names of nearby shops or stores. The user may also zoom out the map via touch input or voice commands, etc., and have the integrated user interface present a zoomed-out view.

[0060] • Integrated User Interface - Vehicle Function User Interface The integrated user interface may further include icons associated with types of vehicle functions. For example, the icons may allow selection of functions such as air conditioning and music, as described herein. In contrast to the exemplary user interface described above, the integrated user interface may ensure that the combined autonomous visualization and map information are not obstructed. For example, if the end user selects the air conditioning icon, a menu may be presented. Advantageously, the integrated user interface may reduce the size associated with the combined autonomous visualization and map information. Furthermore, the integrated user interface may present the menu in the remaining portion of the integrated user interface.

[0061] In some embodiments, the dynamic resizing described above may be based on current contextual information. For example, if the user is navigating a complex series of streets, the integrated user interface may enlarge the size associated with the combined depiction of the autonomous visualization and map information. In this way, the user can clearly see the next turn. Similarly, if there is an approaching hazard, the integrated user interface may enlarge the size of the map and the depicted hazard to ensure that the user can easily see the hazard.

[0062] • Integrated user interface - Driver / passenger side adjustments In certain vehicles, such as trucks, the centralized display may be positioned beyond a threshold distance from the driver's and passenger's seats. Therefore, the driver may have easy access to a portion of the display closer to them (e.g., the left side). Conversely, the driver may have difficulty accessing the distal portion of the display (e.g., the right side). Similarly, passengers may have easier access to the right side of the display and less access to the left side (in vehicles manufactured for left-hand drive). In some embodiments, the integrated user interface may be updated based on whether the driver or passenger is interacting with the integrated user interface.

[0063] For example, icons associated with the type of function may be displayed on the driver's side (for example, along the left side of the display from top to bottom). Therefore, the driver can easily access these icons. When an icon is selected, the menu or user interface associated with that icon may be displayed to the driver. In contrast, the integrated user interface may display icons on the passenger's side. When an icon is selected, the menu or user interface associated with that icon may be displayed to the passenger.

[0064] As described later, in some embodiments, the integrated user interface may be updated based on whether the driver or passenger is interacting with or attempting to interact with the integrated user interface. For example, the system or processor may identify whether the driver or passenger is interacting with or attempting to interact with the integrated user interface. The system may optionally acquire information from an infrared emitter or projector. These may indicate whether the end user's hand is reaching the integrated user interface from a particular side of the display. Thus, the integrated user interface may be updated based on a particular side after detection of the user's hand moving toward the display. The system may optionally acquire information based on one or more cameras positioned in the vehicle. For example, the system may analyze video or images to determine whether the driver or passenger is using or attempting to use the integrated user interface.

[0065] The above-described embodiment of the integrated user interface, as well as other embodiments, are described in detail below.

[0066] • Exemplary block diagram Figure 1A shows a block diagram of an exemplary situational user interface system 100 that presents a user interface 102 (e.g., an integrated user interface as described herein). The situational user interface system 100 may be a system of one or more processors, application-specific integrated circuits, etc. The situational user interface system 100 may be included in a vehicle such as an electric vehicle, and the situational user interface system 100 may cause a display to present the user interface 102. The display may be, for example, a touch-sensitive display included in the front of the vehicle. For example, the display may be included in the central front of the vehicle.

[0067] The situational user interface system 100 can receive user input 104 provided by the user of user interface 102. Exemplary user input 104 may include touch-based user input, verbal commands, etc. In this way, the user of user interface 102 can interact with user interface 102. For example, the user may provide user input 104 for operating different aspects of the vehicle. In this example, the user may select from icons 108A to 108H. Each icon can control any type of vehicle function. For example, icon 108A can control a specific driving function (e.g., steering sensitivity, acceleration characteristics, etc.). As another example, icon 108H can control a music streaming application.

[0068] Therefore, the situation user interface system 100 can output vehicle operation information 106. For example, the information 106 may be provided to a system, module, application, etc. that adjusts the operation of the vehicle. As an example, the user may adjust the steering sensitivity. Therefore, this vehicle operation information 106 may reflect the adjustment so that the system, module, software, or application related to steering control can be updated accordingly.

[0069] As shown in the figure, the user interface 102 includes an autonomous visualization (e.g., a graphic representation of the vehicle 110A) along with map information 110B in a combined view. As described above, the graphic representation 110A may reflect real-time operational information related to the vehicle. For example, if the vehicle's lights are on, the graphic representation 110A may be updated to indicate that the lights are on. The map information 110B may represent a map close to the vehicle's location. As will be described later, this map information may be updated according to the current situation related to the vehicle's operation. For example, when the vehicle is being driven, the map may optionally be zoomed in to present a driving view. As another example, while driving, the map may optionally show the path the vehicle should follow.

[0070] In some embodiments, map information 110B may be used at least partially to render an autonomous visualization. For example, the autonomous visualization may include a graphic representation of the external environment around the vehicle (e.g., the real-world environment). In this example, sensor information (e.g., images from an image sensor) may be analyzed by system 100, or another processor or system in the vehicle, to render the graphic representation. Map information may be used to determine physical features or characteristics of the external environment. For example, the number of lanes may be identified based on the map information. Another example is the identification of a roundabout, the next lane change or transition, the next highway interchange, etc. In some embodiments, these physical features or characteristics may be used to signal the generation of an autonomous visualization. For example, the autonomous visualization may include the exact route of a portion of the road that is approaching but may not yet be visible or may be obstructed. Another example is that the combined view may be zoomed out or shifted, and the map information may signal the generation of an autonomous visualization. For example, if the user adjusts the combined view to shift it one mile forward or zoom it out upward, these portions of the external environment may not yet be visible. Advantageously, map information can be used to render their appearances. For example, representations of buildings may be included in the combined view based on map information.

[0071] In Figure 1A, user interface 102 presents navigation user interface 112. This interface may be available to the user to indicate the location where the user should drive. Advantageously, the navigation user interface 112 is positioned so as not to obstruct the combined views 110A-110B described above. Furthermore, the navigation user interface 112 may be positioned on the driver's side so that the driver can easily interact with it.

[0072] Figure 1B shows a block diagram of an exemplary situational user interface system 100 that presents an updated user interface 102 based on received user input 104. In the illustrated embodiment, the user of the user interface 102 enters user input 104 that allows travel to any location. Thus, the navigation user interface 112 is updated to specify a direction toward the arbitrary location. Furthermore, a route 114A is presented within combined views 110A-110B. The route may be drawn on map 110B as a direction toward location 114B.

[0073] In some embodiments, the navigation user interface 112 may be optionally removed by the user interface 102 when the vehicle is being driven, as described below. For example, the size of the combined views 110A to 110B may be dynamically expanded. In this example, the combined views 110A to 110B may include detailed autonomous visualizations (e.g., driving views).

[0074] The driving view may, for example, represent a rear-upper viewpoint of the vehicle's graphic representation 110A. For example, the driving view may be similar to a camera or drone positioned at a threshold distance behind and above the vehicle. The system or processor may use sensor data, such as images from image sensors, in combination with one or more machine learning models (e.g., convolutional neural networks or other computer vision techniques) to generate information to be included in the driving view. For example, the machine learning model may analyze input images from image sensors positioned around the vehicle. In some implementations, the image sensors may provide a 360-degree field of view around the vehicle. These input images may be analyzed to classify vehicles (e.g., sedans, trucks, motorcycles), objects (hazards, potholes, speed bumps, pedestrians, brake lights, traffic lights), etc., depicted in the images. Optionally, the input images may be stitched together to provide a consistent view (e.g., 360-degree) of the real-world environment in which the vehicle is located (e.g., via a machine learning model such as a neural network). Images or other sensor data may be received at a specific frequency (e.g., 30 frames / second, 60 frames / second, 120 frames / second, etc.). These images can be analyzed to update the information contained in the driving view.

[0075] In some embodiments, the user interface system 100 may access models related to vehicles, objects, etc. The user interface system 100 may then render the models according to the received sensor data in order to generate a driving view. For example, the user interface system 100 may receive information indicating the current position related to a nearby vehicle of a particular type (e.g., a truck). In this example, the user interface system 100 may render a model of the truck in the driving view that matches the current position. As can be understood, the placement of the rendered model in the driving view may be determined using camera parameters from one or more image sensors that have acquired images of the truck. Thus, the placement of the truck in the real world can be translated into a placement in the driving view. Furthermore, in embodiments in which the images are stitched together, the stitched result may represent a map showing the placement, size (e.g., bounding box), etc., of vehicles and / or objects. To identify the placement in the driving view for rendering the truck, this map may be used, for example, by system 100, or by a different system or processor. While the use of the model has been described above, in some embodiments, the physical properties of a vehicle or object may be extracted from one or more images. For example, the system or processor may generate an appearance of the vehicle or object for rendering within the user interface 102.

[0076] Therefore, the driving view may reflect a graphical representation of the real-world environment in which the vehicle is located. Graphic representations of other vehicles may be presented as moving around the vehicle, corresponding to their actual placement in the real-world environment. Furthermore, road markings, signs, etc., may be presented in the driving view. To render vehicles and / or objects included in the real-world environment, the user interface system 100 may update the user interface 102 in real time (e.g., substantially real time). For example, images may be received, analyzed, and used to update the user interface 102.

[0077] As will be described in detail below, with respect to Figure 2B, the user interface system 100 may further update the user interface 102 to reflect navigation information. Thus, the combined view described herein may include autonomous visualization, map information, and navigation information. For example, when the vehicle is being driven along a route, the user interface 102 may be updated to indicate navigation events. In this example, the user interface 102 may indicate that the user should move across one or more lanes, turn, etc.

[0078] In some implementations, the vehicle may operate in autonomous or semi-autonomous mode. Therefore, the driving view may provide insight into the view of the real-world environment determined by the system or processor included in the vehicle. The user can view the driving view to ensure that the real-world environment is being interpreted appropriately. Furthermore, the use of navigation information can proactively project the actions the vehicle should perform onto the user.

[0079] Therefore, the user interface 102 can be dynamically updated according to circumstances related to the operation of the vehicle. For example, the navigation user interface 112 is presented in Figure 1A based on user input 104 indicating the selection of a navigation function. Accordingly, the size of the combined views 110A-110B is dynamically reduced to accommodate the navigation user interface 112. In Figure 1B, the navigation user interface 112 presents directions toward an arbitrary location. As described above, the navigation user interface 112 may be dynamically removed when the user starts driving. In this way, the user can focus on the combined views 110A-110B, which can be updated to include navigation information (e.g., the next navigation event).

[0080] • Example flowchart Figure 2A is a flowchart of an exemplary process 200 for updating an integrated user interface. For convenience, process 200 is described as being performed by a system of one or more processors (e.g., a situational user interface system 100).

[0081] In block 202, the system presents an integrated user interface that depicts the vehicle. As shown in Figures 1A and 1B, the integrated user interface may include autonomous visualization (e.g., a graphic depiction of the vehicle). Furthermore, the integrated user interface may include map information. The extent to which the map information is zoomed in or out may be based on the current situation related to the operation of the vehicle. For example, if the vehicle is parked, the integrated user interface may present a zoomed-in view of the map. In this example, the graphic depiction of the vehicle may substantially fill the display presenting the integrated user interface.

[0082] As another example, if a vehicle is being driven, the map information may be further reduced in size. As an example, the map information may be rendered in the integrated user interface as a street view. In this example, the integrated user interface may include a graphic depiction of the driving vehicle in a rendered version of the street. As another example, the view of the map information may substantially represent a top view. For example, while driving, the integrated user interface may reduce the map to indicate the next turn. In this example, the reduction may be based on a set of approaching next turns that have been identified. Thus, the integrated user interface may favorably depict a set of turns on the map information. As yet another example, the reduction may be based on the type of upcoming driving event. For example, it may be reduced by exiting a highway onto a ground road. As yet another example, the reduction may be based on the detection of hazards. For example, hazards may be detected using images from the vehicle's image sensors. In this example, the route is updated to avoid hazards, and the map information may be reduced to depict at least a portion of the updated route.

[0083] The integrated user interface may further include menus or user interfaces associated with the types of vehicle functions. As described herein, these menus or user interfaces may be located next to the map information. Thus, users of the integrated user interface can interact with the menus while maintaining a view of the map information.

[0084] In block 204, the system receives information to update the status information. The user of the integrated user interface may provide user input to update the status related to the operation of the vehicle. For example, the user may instruct the integrated user interface to present navigation information to a specific location. As another example, the user may provide user input to select an icon associated with a type of vehicle function.

[0085] Furthermore, the received information does not necessarily have to be based on user input. For example, cameras or sensors placed around the vehicle may be used to identify hazardous materials, traffic lights, stop signs, environmental conditions, etc. This information may be used to update the integrated user interface. For example, if the vehicle is being driven in an off-road environment, the environmental conditions may reflect fast-moving water ahead or inclines greater than a threshold angle. Similarly, the received information may be further related to the vehicle's position or location. As shown in Figure 2B, the system may update the integrated user interface depending on the vehicle's position relative to the navigation path. For example, the combined views described herein may be zoomed in or out depending on the location.

[0086] In block 206, the system updates the integrated user interface based on the information. With respect to user input associated with an icon, the integrated user interface may be updated to include the menu or user interface associated with the icon. Similarly, when the user enters navigation mode, the integrated user interface may be updated to include the direction of travel on a combined view of the vehicle and map.

[0087] With respect to information determined from image sensors or other sensors (e.g., radar, ultrasound), the integrated user interface may be updated to reflect this information. For example, if a stop sign is approaching, the integrated user interface may highlight the detected stop sign. As another example, the vehicle may automatically stop upon the presence of a stop sign and the information within the integrated user interface included to reflect this stop. If a hazardous object such as a pothole is detected, the integrated user interface may be updated to depict the hazardous object in the combined view. With respect to the driving view described above, the integrated user interface may indicate the location of any potholes in the street the vehicle is approaching. Optionally, the integrated user interface may present information indicating that the user should adjust their lane.

[0088] Advantageously, when updating the integrated user interface, the system can optionally ensure that the combined view of the map with autonomous visualizations (e.g., depictions of the vehicle, and optionally nearby vehicles and / or objects) is not obstructed. For example, the system may overlay navigation information onto the combined view. As another example, the system may include information determined using cameras and / or sensors in the combined view. As yet another example, the system may present a menu or user interface associated with a vehicle function separate from the menu or user interface of the combined view. In this example, the system may dynamically resize and / or adjust the position of the combined view. For example, the combined view may be reduced in size and positioned on the passenger side (e.g., to the right) of the user interface when the driver is operating the user interface.

[0089] The integrated user interface may optionally present images or videos from cameras mounted on the vehicle. As detailed below, the integrated user interface may present images or videos in close proximity to the combined view. These may be triggered for presentation, for example, when the vehicle is towing something. They may also be triggered when the vehicle is in off-road mode. Examples of presenting images from imaging devices or videos from cameras are included below.

[0090] Figure 2B is a flowchart of an exemplary process 210 for updating the integrated user interface according to the vehicle's configuration. For convenience, process 210 is described as being carried out by a system of one or more processors (e.g., the Situation User Interface System 100).

[0091] In block 212, the system presents an integrated user interface via a display. In block 214, the system receives user input to select navigation control. As described above, with respect to Figures 1A and 1B, the integrated user interface can enable navigation functionality. In block 216, the system receives a destination selection. The user may provide user input indicating any location to drive to. For example, the user may specify a name associated with any location. The user may search for any location using the search user interface.

[0092] In block 218, the system updates the integrated user interface to display a combined view of autonomous visualizations (e.g., a graphic depiction of a vehicle) included in the graphic representation of the map. As shown in Figure 1B, the user interface may be updated to reflect route information related to a route to any location. Thus, the combined view may reflect a substantially top view of the map on which the vehicle is depicted. For example, the combined view may represent a rendered view of the real-world environment with a rendering camera or virtual camera positioned at a threshold distance above the vehicle.

[0093] In block 220, the system updates the integrated user interface based on the vehicle's movement. As described above, as the vehicle moves, the system dynamically adjusts the combined view. For example, the integrated user interface may depict a driving view showing the rear of the vehicle traveling along a path. Thus, the driving view may represent a rendered view of the real-world environment with a rendering camera or virtual camera positioned at the rear of the vehicle.

[0094] The system may update the integrated user interface based on updated contextual information, such as updating the vehicle's position on the route. For example, the system may maintain the rear driving view of the vehicle while there are fewer than a threshold of upcoming driving events. For instance, when driving a vehicle along a highway with no upcoming turns or lane changes, the system may maintain the driving view so that it occupies a significant portion of the display. As another example, if the route indicates a threshold number of upcoming driving events, the system may update the combined view to shrink. For example, the autonomous visualization and map information may be shrunk. In some embodiments, when shrunk, the autonomous visualization may remove certain visualizations (e.g., visualizations of other vehicles, pedestrians, signs, road markings, etc.) while maintaining the graphical representation of the vehicle. Upcoming driving events may be identified on the map information. For example, route instructions may be graphically included in these driving events.

[0095] In some embodiments, the system may scale up or down depending on the vehicle placement and one or more complexity measures related to the route. For example, the system may determine a complexity measure for the upcoming portion of the route that is within a threshold distance of the vehicle placement or within a threshold driving time of the vehicle placement. Based on the complexity measure, the system may update the combined view. For example, if the upcoming portion of the route has a high complexity measure, the system may scale down to show more navigation events.

[0096] The complexity scale may relate to the number of upcoming turns, the number of upcoming lane changes, the number of upcoming driving maneuvers (e.g., U-turns), etc. In some embodiments, the above information may be associated with various weights. For example, the number of upcoming turns may be associated with a higher complexity scale when compared to the number of lane changes. Furthermore, the complexity scale may relate to the likelihood associated with missing upcoming driving events. For example, the likelihood may be associated with historical information. In this example, the complexity scale may be increased if the vehicle's previous driver had missed one or more of the upcoming driving events.

[0097] Advantageously, when a user selects a vehicle function, such as by selecting an icon, a menu associated with the function may be presented next to the driving view. In this way, the user can control the function without the driving view being obstructed.

[0098] Figure 2C is a flowchart of an exemplary process 230 for updating the integrated user interface based on the selection of vehicle functions. For convenience, process 230 is described as being carried out by a system of one or more processors (e.g., Situation User Interface System 100).

[0099] In block 232, the system presents an integrated user interface via a display. In block 234, the system receives user input indicating a specific vehicle function. For example, the integrated user interface may include a number of icons associated with various vehicle controls (e.g., air conditioning, music, etc., as described herein). The user may select a specific icon to adjust the corresponding vehicle control. For example, the user may provide touch input to the display. As another example, the user may provide verbal commands for explanation by a system (e.g., one included in or communicating wirelessly with the vehicle) or another process or system. As described herein, context information may be updated to reflect the selection of a specific vehicle function.

[0100] In block 236, the system adjusts autonomous visualizations (e.g., graphic representations of the vehicle) in the integrated user interface. Certain vehicle functions may adjust the graphic representation of the vehicle. For example, the user may adjust the vehicle's lighting (e.g., fog lights, high beams of the headlights, etc.). In this example, the system may update the graphic representation accordingly. In another example, the user may adjust the truck's suspension. In this example, the graphic representation may show the truck rising or falling. In yet another example, the user may activate the wipers, and the representation may show the wiper's movement. In yet another example, the user may open or close the tonneau cover, and the representation may show this opening and closing.

[0101] In block 238, the system updates the integrated user interface to include menus or user interfaces associated with functions. For example, it may include user interfaces for control types. As described herein, the menus or user interfaces may be presented in a manner that does not obstruct the depiction of vehicles and maps. For example, the size of the depiction of vehicles and maps may be changed (e.g., reduced in size). Thus, the user can interact with the menus or user interfaces while maintaining a view of the map.

[0102] Figure 3A is a flowchart of an exemplary process 300 for using multitasking control of an integrated user interface. For convenience, process 300 is described as being performed by a system of one or more processors (e.g., Situation User Interface System 100).

[0103] In block 302, the system presents an integrated user interface. In block 304, the system receives user input indicating a selection of a first control type. For example, the user of the user interface may select an icon associated with a specific vehicle function. In block 306, the system receives a selection of a second control type. In response, the system may remove the menu or user interface associated with the first control type. Instead, it may include a menu or user interface associated with the second control type.

[0104] In block 308, the system updates the integrated user interface to include multitasking control. This control allows for quick switching between recently selected vehicle functions. For example, the user may switch between a first type of control and a second type of control by manipulating the multitasking control. In some embodiments, the multitasking control may represent one or more lines included along the same axis (e.g., horizontal). The user may use swipe gestures to navigate between the various menus or user interfaces associated with the first and second types of control.

[0105] In block 310, the system receives user input for multitasking control. User input may include, for example, a selection of multitasking control on a touchscreen display. For example, the user may swipe along one or more axes (e.g., the horizontal axis). User input may further include verbal commands associated with the multitasking control. As described above, the user can quickly switch between the first type of control and the second type of control.

[0106] Next, we refer to Figures 3B to 3D, which show user interfaces associated with multitasking control. Figure 3B shows a user interface 320 presenting a depiction of the vehicle 322, along with a music user interface 324. For example, the vehicle may be in parking mode. The following description may further apply when the vehicle is being driven or in motion. For example, a combined view of autonomous visualization and map information may be presented to the user interface 320.

[0107] Figure 3C shows a user interface 320 presenting a depiction of the vehicle 322, along with a quick control user interface 326. For example, the user may interact with the music user interface 324 and close it, or select an icon associated with the user interface 326.

[0108] As shown in Figure 3C, a multitasking control unit 328 is included. Each line of the multitasking control line can represent a different user interface. Therefore, selecting a line (for example, via input to the line or via swiping) can result in the presentation of various user interfaces.

[0109] For example, the leftmost line may correspond to the quick control user interface 326. In another example, the rightmost line may correspond to the music user interface 324. In some embodiments, the most recently presented user interface may be included in the user interface 320 closest to the user. In this example, the driver may be operating user interface 320. Therefore, the last used user interface may be the leftmost line. If a passenger was operating user interface 320, the last used user interface may be the rightmost line. As shown in the figure, the leftmost line is highlighted to indicate that the relevant user interface (e.g., user interface 326) is presented.

[0110] Figure 3D shows a user interface 320 having the presented climate user interface 330. As shown, the multitasking control unit 328 includes three lines. In some embodiments, the user interface 320 may limit the number of lines so that the user can select within a threshold number of recently used vehicle functions. In some embodiments, on the lines, the user interface 320 may include a live view or representation of the relevant vehicle function. For example, on the center line, a live view of the user interface for quick control 326 may be included. In some embodiments, the live view may be presented based on a system that identifies that the user is about to interact with the user interface 320. For example, an infrared emitter, camera, or other sensor may be used to determine that the user is about to interact with or is interacting with the user interface 320.

[0111] Figure 4 is a flowchart of an exemplary process 400 for adjusting icons on the driver or passenger side. For convenience, process 400 is described as being performed by a system of one or more processors (e.g., a situation user interface system 100).

[0112] In block 402, the system presents an integrated user interface via a display. Icons associated with vehicle functions may be presented on the driver side of the display presenting the integrated user interface. In some embodiments, icons may be presented on the driver side when the display is illuminated. For example, the display may illuminate when the driver enters the vehicle. In some embodiments, the system may determine whether the driver or a passenger is inside the vehicle when the display is illuminated. For example, if only a passenger is in the vehicle, the icon may be placed on the passenger side. In this example, the system may use a sensor in the seat to determine that a passenger is seated. The system may also identify that the passenger-side door has been used. The system may also use an on-board camera or other sensor inside the vehicle to identify that a passenger is inside the vehicle. Similarly, the system may use an on-board camera outside the vehicle to identify that a passenger has entered or is about to enter the vehicle.

[0113] In block 404, the system identifies whether user input has been received or is about to be received from the driver or a passenger. The system may use an infrared emitter or projector to determine if a hand is moving toward the display. The system may also use a camera to determine hand movements. In some embodiments, a machine learning model may be used to distinguish between movements toward the display and movements toward the center of the vehicle (e.g., to access the center console or a drink).

[0114] In block 406, the system updates the icon to represent the passenger. If the system identifies that the passenger is providing user input, the system may update the icon to represent the passenger. In contrast, in block 408, if the user input is provided by the driver, the system may maintain the icon for the driver.

[0115] As described above, the relevant vehicle control user interface can be presented by selecting an icon. This user interface can be presented in a way that does not obstruct the combined view of the autonomous visualization and map information. For example, the size of the combined view can be reduced, and the combined view can be placed far away from the passenger or driver. As an example, if the system determines that a passenger is providing user input, the system may present the vehicle control user interface to the passenger. Similarly, the system may present the combined view closer to the driver.

[0116] • Exemplary user interface Exemplary embodiments of integrated user interfaces are described below. Each user interface is described in one or more embodiments, and it will be understood that specific aspects or features of these user interfaces may be adapted and fall within the scope of the disclosure herein. These user interfaces may be presented via a display included in the vehicle. In some embodiments, the display may be a touch-sensitive display. The user interfaces may be rendered by a system or processor included in the vehicle, such as the user interface system 100 described above.

[0117] Figure 5A shows an exemplary user interface 500 of a combined view showing a parked vehicle. As shown, the user interface includes a graphic representation 502 of the vehicle. Furthermore, it includes a representation 504 of the environment. For example, an autonomous visualization including the graphic representation 502 and the representation 504 may be rendered. In addition, the user interface 500 may optionally include map information. For example, it may include a display of the street name where the vehicle is parked. As another example, shops, restaurants, etc., may be depicted in the representation 504. These may be obtained, for example, from street view information. Thus, the situation information may reflect the parked vehicle in some embodiments.

[0118] Optionally, the combined view may include topographic information. For example, the vehicle may be parked while off-road. In this example, the autonomous visualization may include a graphic representation of the terrain surrounding the vehicle. For example, sensor information (e.g., images) of the terrain may be acquired. Map information may be used to inform the user of terrain information. For example, a topographic map may be accessed for the vehicle's location. Thus, the user interface 500 may include a representation of the terrain based on a topographic map. Optionally, sensor information and a topographic map may be combined. As an example, a topographic map may be used to determine the shape related to the vehicle's location. The sensor information may then inform the user of the shape or other detailed information. For example, the sensor information may indicate that part of the terrain is steeper than shown on the topographic map. This may be useful for a user who is off-road in the vehicle. Thus, the representation 504 may be based on the terrain information of the surrounding area.

[0119] As described above, representation 504 may include information determined using cameras or sensors placed around the vehicle. For example, pedestrians, other vehicles, cones, animals, lighting, stop signs, etc., may be determined by systems included in the vehicle. These can be rendered on the user interface 500. Similarly, representation 504 may include rocks, depressions, slopes, etc., in the off-road area surrounding the vehicle. In some embodiments, a drone may be stored in the vehicle. The drone may fly from the vehicle and perform mapping operations of the surrounding terrain. For example, photogrammetry techniques may be used. This information may be provided to the vehicle and used to render representation 504. In some embodiments, the vehicle may render representation 504 at least in part using simultaneous localization and mapping (SLAM) techniques.

[0120] The vehicle depiction 502 includes selectable options 506A to 506E. Each selectable option can be used to operate the vehicle. For example, option 506A may open the hood. Option 506E may open the tonneau cover. Similarly, icons 508A to 508H may be used to present menus or user interfaces associated with different vehicle functions. For example, icon 508H may present a menu or user interface associated with music streaming. As described above, if this icon 508H is selected, the user interface 500 may be updated to reduce the size of the vehicle depiction 502 and the associated environment 504. In this way, the user of the user interface can interact with the music streaming application.

[0121] Figure 5B shows the user interface 510 for a parked vehicle being charged. As shown, the vehicle depiction 502 reflects the vehicle being charged. The user interface includes, for example, charging information 510. Therefore, in some embodiments, the status information may reflect the vehicle being charged.

[0122] Figure 6A shows an exemplary user interface 600 for a driving view. In this example, autonomous visualization is included in the user interface 600. The visualization may include a graphic depiction 602 of the vehicle along with a representation of an exemplary environment 604 in which the vehicle is being driven. As mentioned above, this exemplary environment 604 may be rendered based on map information (e.g., satellite information, topographic maps). The environment 604 may also be rendered using cameras and / or sensors placed on the vehicle. Thus, the user of the user interface 600 can see approaching hazards, changes in slope, and so on.

[0123] In the illustrated example, the user interface 600 includes lines 605A–605B that the user can follow. In off-road situations, it may be advantageous for the user to quickly see exemplary lines to follow. In this way, if a hazard appears, the lines may indicate that the user should avoid this hazard. The lines may be determined using sensors and / or cameras on the vehicle. For example, lines 605A–605B may be determined based on existing road markings (e.g., tire tracks). In another example, lines 605A–605B may be determined based on the vehicle's width and terrain information. In this example, lines 605A–605B may be positioned so that the vehicle remains beyond a threshold distance of a raised or walled portion of the terrain. Thus, lines 605A–605B may indicate substantially flat or drivable portions of the terrain being approached.

[0124] The user interface 600 further includes a quick control user interface 606. This quick control user interface 606 may be presented based on the user's selection of an associated icon 608. When the quick control user interface 606 is presented, the user interface 606 may be updated to resize the combination view described herein. For example, as shown in the figure, the combination view is positioned on the right portion of the user interface 606 and its size is reduced (e.g., its width is reduced). For example, when selecting information in the user interface 606, the user may dismiss the quick control user interface 606. The combination view may then dynamically increase in size to encompass many user interfaces 600.

[0125] While the example in Figure 6A shows an off-road environment, environment 604 can also represent an on-road environment. For example, it may depict a city road or highway on which a vehicle is being driven.

[0126] Figure 6B shows an exemplary user interface 600 with icons 610 adjusted for a passenger view. The placement of the icons 610 may be updated based on whether a passenger or driver is accessing the user interface 600, as described at least in Figure 4. In the illustrated embodiment, the quick control user interface 606, along with the icons 610, is moved to the passenger (e.g., right) side of the user interface 600. Similarly, the autonomous visualizations (e.g., vehicle 602 and environment 604) are moved to the driver side.

[0127] Figure 6C shows an exemplary user interface 600 having the user interface 620 of today. The user interface 620 of today may display weather information, calendar information, etc., as shown in the figure.

[0128] Figure 6D shows an exemplary user interface 600 having an air conditioning user interface 630.

[0129] Figure 7A shows an exemplary user interface 700 for an off-road driving view. This user interface 700 displays autonomous visualizations (e.g., a depiction of the vehicle 702A and the environment 704 in which the vehicle is being driven). Furthermore, the user interface 700 may include map information that indicates the topology of the environment 704. In some embodiments, the integrated user interface described herein may automatically present off-road information based on detected road conditions. For example, a system or processor included in the vehicle may determine when an off-road condition is detected and cause the user interface 700 to present off-road information. In this example, the off-road condition may be detected based on the detected driving conditions. Thus, in some embodiments, the condition information may reflect the off-road condition.

[0130] For example, driving conditions may be based on measures related to vibration, angle changes related to the vehicle, tire slip, etc. Furthermore, sensor information (e.g., image sensors) may be used to determine the type of surface the vehicle is driving on. Thus, if the surface type is rocky, sandy, etc., off-road conditions can be detected. As another example, the system may use the vehicle's location to notify the presentation of off-road information. For example, a map may indicate that the vehicle is outside a known road or on an unpaved road known to be off-road.

[0131] Exemplary off-road information may include a scale of roll related to the vehicle. For example, depiction 702A may be shown as driving relative to the environment 704, as described above with respect to autonomous visualization. However, for off-road driving, depiction 702A may further indicate a scale of roll (e.g., "8 degrees"). This roll may be depicted graphically, as shown in Figure 7A. Furthermore, a scale of pitch 702B may be included. In the example in Figure 7A, the pitch 702B is 14 degrees. Sensors within the vehicle may be used to report these measured angles.

[0132] The driver side of the user interface 700 includes video or images from the image sensor 706. The user of the user interface 700 may select an icon that triggered the presentation of an image or video. Advantageously, the image or video may be presented without obstructing the driving view. As shown in the figure, information related to climbing rocky terrain and / or the drive system may be presented.

[0133] Figure 7B shows an exemplary user interface 700 that presents drivetrain information 708. This information may include, for example, tire pressure for each wheel, torque, and suspension-related information.

[0134] Figure 8A shows an exemplary user interface 800 for a truck trailer. The vehicle described herein may be a truck. Advantageously, the integrated user interface may present information related to the towing of the truck trailer. For example, information may be presented based on the user selecting a towing mode via the user interface 800. Information may also be presented based on cameras or sensors in the truck that detect the proximity of the truck trailer (e.g., behind the truck). Information may also be presented based on the truck trailer being equipped with wireless communication to the vehicle. Thus, situational information may indicate that the truck is attempting to tow or otherwise connect to the trailer.

[0135] The illustrated example includes autonomous visualization. For example, a graphic representation of the vehicle 802 is included. Depending on the current situation related to the trailer's mounting, the representation 802 is presented as a top view of the vehicle. A representation of the trailer 804 is further included. This representation of the trailer 804 may be generated using cameras and / or sensors. For example, a model may be generated based on the trailer's field of view. Optionally, the user interface 800 may provide a pre-rendered version of the trailer. For example, a system included in the vehicle (e.g., system 100) may have a stored model of the trailer. Thus, the user interface 800 may reflect the placement of the trailer 804.

[0136] The user interface 800 further includes mounting points 806A–806B for the vehicle and trailer. In some embodiments, the user of the user interface 800 may have these mounting points 806A–806B positioned aft and aligned. In some embodiments, the vehicle may be autonomous and may automatically reverse. As shown, the trailer may be automatically detected (e.g., “Trailer Detected”), and the distance between the vehicle and the trailer may also be indicated (e.g., “4 Feet Remaining”). The distance may be determined via radar, ultrasonic, and / or image sensors. In some embodiments, the user of the user interface may select a “Mount” selectable object to initiate mounting or otherwise have it mounted.

[0137] Images or videos from camera 808 are also included in the user interface 800. These may be presented based on user input of icons included in the user interface 800. These may also be presented while the vehicle is towing or attempting to tow a trailer.

[0138] Figure 8B shows an exemplary user interface 800 including a driving view for towing a trailer. Situational information may indicate that the trailer is attached to the vehicle and the vehicle is being driven, as described herein. The illustrated example includes an autonomous visualization. The visualization includes a graphic depiction 802 of the vehicle along with the trailer 804. This depiction can be adjusted in real time. For example, cameras and / or sensors in the vehicle may monitor the trailer. Thus, the user interface 800 may present any rolling, jostling, sliding, etc. In some embodiments, the trailer may include available sensors and / or cameras to monitor its condition. This may be received and used to indicate renderings of the trailer 804 and / or the environment. For example, a rear image sensor of the trailer may be used to identify vehicles and / or objects behind the trailer.

[0139] In this example, a highway environment 810 is rendered. For example, it includes highway lanes along with other vehicles. In some embodiments, the size of the trailer may also be indicated so that the driver can trust the user interface 800 to see its proximity to other vehicles and / or objects.

[0140] Camera information from the image sensor 814 is further included in the user interface 800 for easier viewing. As described herein, this camera information may be offset from the driving view of the vehicle 802 and the environment 810. Towing information 812, such as the weight of the trailer, is further included.

[0141] Figure 9 shows an exemplary user interface 900 for camper mode. In some embodiments, the vehicle may be set to camper mode. Therefore, the situation information may reflect camper mode. The integrated user interface described herein may be updated to reflect a graphic depiction of the vehicle 902, along with a graphic depiction of the camper 904. Furthermore, accessories to the vehicle may be graphically represented. In some embodiments, these accessories may be automatically detected by a system included in the vehicle. For example, accessories may be detected using a camera on the vehicle. Accessories may also provide wireless information that identifies their function or use.

[0142] In the example shown in Figure 9, the exemplary accessory is a stovetop 906. The user can control the operation of the stovetop 906 via the user interface 900. For example, the user may instruct the stovetop to begin cooking food. Alternatively, the user may specify a temperature or measurement related to heat (e.g., high temperature, medium temperature). Control information may be transmitted to the stovetop 906 via wired or wireless communication.

[0143] Figure 10A shows an exemplary user interface 1000 for vehicle operation. In this example, situational information may indicate that the vehicle is traveling toward an arbitrary location. The user may use the exemplary user interface 1000 to follow driving instructions to the arbitrary location. This user interface 1000 may also be presented while the vehicle is in autonomous mode or semi-autonomous mode (e.g., autonomous driving mode).

[0144] Autonomous visualizations (e.g., graphic representations of the vehicle 1002 and the highway environment 1004) are included in the user interface 1000. As shown in the figure, the environment 1004 shows carpool lanes, double lines, other vehicles, etc. The user interface also includes map information. For example, the map information may indicate street names (e.g., "Crenshaw Street" 1006), lane directions, etc. The combined view of autonomous visualizations and map information can represent the driving mode as described above.

[0145] The user interface 1000 includes information 1006 indicating the vehicle's next turn. During driving, as shown at least in Figures 1A and 1B, the integrated user interface may optionally present a combined view showing driving events. Thus, the combined view may indicate the vehicle's next turn. In the case of complex turns, the environment 1004 may optionally be reduced to show multiple top views, as shown in Figure 2B.

[0146] Since navigation technology relies on sensors and / or cameras, systems included in a vehicle may have precise knowledge of the vehicle's position. For example, the system may identify the exact lane the vehicle is driving in. Therefore, the user interface 1000 may indicate that the vehicle should move to the next lane 1008. As shown in the figure, the user interface 1000 displays the vehicle moving to the next lane 1008 (e.g., by animation). Similarly, the user interface 1000 may indicate the presence of other vehicles. Therefore, if there is a vehicle in the next lane 1008, the user interface 1000 may indicate that the vehicle should wait until it moves safely. In autonomous driving mode, the highlighted next lane 1008 may be used to notify the user that the vehicle is changing lanes or is about to change lanes.

[0147] As described above, the user can interact with icon 1010 to present a menu or user interface associated with vehicle functions. When an icon is selected, the size of the combined view of vehicle 1002 and environment 1004 may be reduced.

[0148] Furthermore, when in driving mode, the user can reduce the combined view. For example, the user may reduce the combined view using pinch / zoom techniques. Thus, the reduced combined view may reflect a larger map area. For example, additional streets, highways, etc., may be reflected in the user interface. Furthermore, the reduced combined view may reduce the size of the vehicle graphic representation 1002. Vehicles and / or objects close to vehicles may be included in the reduced combined view. For example, if the user reduces the view slightly, another vehicle that is not close enough to be seen in the combined view of Figure 10A may be rendered. Furthermore, the reduced view may highlight large portions of the route to any given location.

[0149] Figure 10B shows another exemplary user interface 1000 for vehicle movement. In the illustrated example, the vehicle is moving to the next lane 1008 shown in Figure 10A. The user interface 1000 then updates to reflect the vehicle exiting this lane.

[0150] Figure 10C shows another exemplary user interface 1000 for vehicle navigation. In the illustrated example, the vehicle is approaching the next turn (e.g., an exit). In some embodiments, icons indicating movement may be presented. In another example, lane 1012 is highlighted to indicate the curved path the vehicle should follow. The user interface 1000 may be advantageously reduced to show many paths (e.g., the upper right curve 1014) so ​​that the driver can quickly follow the navigation. In some embodiments, when exiting, the user interface 1000 may be partially reduced to show many top views. Optionally, the user interface 1000 may be reduced depending on the complexity of the upcoming series of turns. The technique of reduction is described above with respect to Figure 2B.

[0151] Figures 11A to 11C show the user interface 1100 for selecting navigation information. As shown in Figure 11A, the user can search for a location via the navigation user interface 1102. Therefore, the situational information can reflect the user's intention to drive. A combined view of autonomous visualization 1104 and map information is further included in the user interface 1100.

[0152] Figure 11B shows the selection of an arbitrary location via the navigation user interface 1102. Summary information 1106 of the arbitrary location may be presented on the user interface 1100. Furthermore, the combined view 1104 may be reduced to present a graphic representation 1108 of the arbitrary location. In this way, the user can see the location of their vehicle along with the selected location.

[0153] Figure 11C shows a path 1110 to an arbitrary position. In some embodiments, an outline of direction 1112 may be included in the user interface 1100. Optionally, this outline may be included while the vehicle is in parking mode or for a threshold time after the selection of any position. As described herein, the combined view 1104 can then be zoomed in to depict the driving view. For example, a seamless animation of the zoom may be included in the user interface. In this example, the animation may reflect a rendering camera or virtual camera adjusting its position from an overhead oblique view to a close rear view (as shown, for example, in Figure 10A).

[0154] • Facial recognition and passenger tracking As described above, in some embodiments, a system included in the vehicle may identify which vehicle occupant (e.g., driver, passenger in the front seat) is attempting to provide user input to the display. For example, the system may be a situational user interface system 100. Based on this identification, the integrated user interface presented on the display may be updated. For example, icons may be switched between the left and right sides of the display. As described below, the system may adjust other aspects of vehicle operation based on occupant tracking. As an example, the system may use facial recognition technology to learn the vehicle's occupants. Thus, the system may learn preferred settings and / or profile information for the occupants. As another example, the system may adjust the mirrors, air conditioning, steering wheel, etc., based on the detected occupant's position.

[0155] Figure 12 is a flowchart of an exemplary process 1200 for associating passenger preference information with a passenger. For convenience, process 1200 is described as being carried out by one or more computer systems (e.g., situational user interface system 100).

[0156] In block 1202, the system identifies occupants in the vehicle. The system may use cameras and / or sensors to determine that an occupant is in the vehicle. For example, the sensors may be infrared emitters or projectors. In this example, the system may determine that an occupant is seated if the beam is blocked. In another example, machine learning techniques may be used to identify people based on images. For example, a convolutional neural network may be used. In yet another example, the system may identify an occupant when the vehicle door is opened and / or when a seat weight sensor indicates that an occupant is in a seat.

[0157] In block 1204, the system stores information that can be used to recognize the faces of passengers. The system may use deep learning techniques. For example, the system may use one or more cameras located inside or outside the vehicle to acquire images of the passengers' faces (for example, the images may be taken when the passenger gets into the vehicle). The system may then use a deep learning model to encode the passengers' faces in an exemplary face space previously learned by the model. Thus, a passenger can be uniquely identified using only a limited number of images of faces.

[0158] In block 1206, the system receives instructions regarding the passenger's preferences. The passenger may adjust the mirrors, seat arrangement, air conditioning, music preferences, etc. The system may store these preferences, or some of them.

[0159] In block 1208, the system associates the passenger's preferences with the passenger. The system can remember the preferences so that the passenger can use them when they get into the vehicle later.

[0160] It should be understood that in all situations where facial recognition technology is used, these may be opt-in. Furthermore, the stored information may be stored locally in the vehicle and may be encrypted.

[0161] Figure 13A is a flowchart of an exemplary process 1300 for adjusting the air conditioning based on passenger tracking. For convenience, process 1300 is described as being carried out by one or more computer systems (e.g., situation user interface system 100).

[0162] In block 1302, the system identifies the occupants of the vehicle, as shown in Figure 12. In block 1304, the system delivers air through a heating, ventilation, and air conditioning (HVAC) system. For example, the system may deliver conditioned air through an airflow from an HVAC system within the vehicle. The HVAC system may further include another flow that can be used to change the height (e.g., vertical axis) of the airflow. Furthermore, the HVAC system may adjust the airflow along the horizontal axis. An additional description of an HVAC system capable of adjusting the horizontal and vertical delivery of air is contained in U.S. Patent Application No. 15,060,590, which is incorporated herein by reference in its entirety.

[0163] In block 1306, the system determines the location of any part of the passenger's body based on an image or video of the passenger. For example, that part may be the passenger's face. In some embodiments, the system may have stored facial recognition information about the passenger. In some embodiments, the system may determine faces without facial recognition technology. For example, the system may use a deep learning model or other machine learning model that can be used to segment faces from images of the passenger.

[0164] In block 1308, the system adjusts the airflow based on tracking of an arbitrary position. When a passenger moves within their seat, for example, leaning forward / sitting / turning their head, the system may adjust the airflow to focus on an arbitrary position. In this way, the air is continuously directed at the passenger's face. In some embodiments, if the internal temperature inside is above a threshold, the system may direct the airflow at the passenger's face. In some embodiments, if the external temperature is high and the passenger entered the vehicle within a pre-threshold time, the system may direct the airflow. In this way, the system can cool the passenger. In some embodiments, a thermal sensor or camera may be used to monitor the passenger's temperature. For example, if the passenger's face is hot, such as after driving or being in hot outside air, the system may ensure that air is directed at the passenger's face.

[0165] Optionally, the system may direct airflow to avoid the passenger's face. For example, after a threshold time during which air is directed at the passenger's face, the system may direct the airflow towards the passenger's body. Alternatively, the system may identify when the internal temperature falls below a threshold and then direct the airflow towards the body.

[0166] Passengers may optionally indicate whether they prefer air to track their face and body, or whether they prefer not to have tracking performed.

[0167] Figures 13B–13D show exemplary user interfaces for an air conditioning system that tracks passengers. As shown in these user interfaces, the airflow 1310 is being adjusted. User interfaces, such as the integrated user interfaces described herein, may present real-time adjustment of the airflow. For example, the adjustment may be presented in an air conditioning control user interface positioned in close proximity to a combined view of the vehicle and map information.

[0168] Figure 14 is a flowchart of an exemplary process 1400 for adjusting the mirrors based on passenger tracking. For convenience, process 1400 is described as being performed by one or more computer systems (e.g., situation user interface system 100).

[0169] In block 1402, the system identifies an occupant in the vehicle. In block 1404, the system determines the position of any part of the occupant's face based on an image or video of the occupant. For example, the system determines the position of the occupant's eyes. In some embodiments, the system may determine the height of the eyes, the horizontal position of the eyes (e.g., X, Y, Z coordinates), the distance between the eyes, etc. In some embodiments, the system may determine a vector extending from each eye to indicate the position the eyes are looking at.

[0170] In block 1408, the system adjusts the mirrors based on any partial tracking. The system may adjust the side mirrors, rearview mirrors, etc., based on the positioning information of the occupant's eyes. For example, the system may adjust the side mirrors to ensure that the occupant's eyes using the side mirrors have a substantially optimal field of view. Similarly, the system may adjust the rearview mirrors as the occupant moves their head to ensure that the occupant has a consistent rearward view image.

[0171] In some embodiments, a user interface may be presented that recommends adjusting the passenger's seating position to favorably view the mirror. For example, the passenger's seat may be in an unfavorable position for viewing the mirror. Another example is that the passenger may be too low or too high in a particular seating position to favorably view the mirror. In some embodiments, the system may automatically adjust the seat (e.g., raise the seat, lower the seat, etc.).

[0172] In addition to adjusting the air conditioning and mirrors, the system may also adjust the steering wheel. For example, the system may adjust the steering wheel based on the passenger's height. Height may be determined based on the height of the passenger's face or another part of the passenger's body (e.g., arms, torso, etc.). Thus, using the passenger's image and / or video, the system may automatically adjust the steering wheel.

[0173] In some embodiments, the system may use images or videos of the vehicle's interior to determine whether any animals are present. For example, the system may automatically trigger dog mode if it detects that a dog or other animal is present inside the vehicle when a person leaves the vehicle. In some embodiments, the system may use face tracking or eye tracking to ensure the driver monitors the road when autonomous driving technology is involved.

[0174] Other embodiments Exemplary embodiments may include methods, systems, and non-temporary computer storage media. An exemplary method is implemented by a system of one or more processors, the system being included in a vehicle, and the method is to cause a vehicle display to present an integrated user interface having a combined view aggregating autonomous visualizations and map information, wherein the autonomous visualizations and map information are associated with magnification levels, and the integrated user interface is generated based on first context information indicating vehicle operation; accessing second context information indicating subsequent vehicle operation; updating the integrated user interface based on the second context information, wherein the combined view is adjusted, and adjusting the combined view includes adjusting the size of the combined view in the integrated user interface or adjusting the magnification level.

[0175] The embodiments described above may include one or more of the following: Autonomous visualization includes a graphic depiction of a vehicle. Autonomous visualization includes a graphic depiction of a vehicle and a graphic representation of the real-world environment in which the vehicle is located. The graphic representation of the real-world environment includes one or a graphic depiction of other vehicles in close proximity to the vehicle. The graphic representation of the real-world environment includes a graphic depiction of other vehicles and a graphic representation of one or more objects in close proximity to the vehicle. The objects include road markings, stop signs, traffic lights, pedestrians, trash cans, or road signs. A combined view includes map information comprising a graphic representation of one or more lanes of the road in which the vehicle is located, a graphic depiction of a vehicle positioned in a specific lane of the one or more lanes, and a graphic representation of a map related to the road, wherein the graphic depiction of the vehicle is depicted as if it were on a map. The combined view represents a rendered view of the real-world environment, with a rendering camera or virtual camera positioned at a threshold distance above the rear of the vehicle. The combined view represents a driving view, and the graphic depiction of the vehicle is animated as driving in a specific lane. The map shows the names of roads. The map shows other names of other roads adjacent to the road. The embodiment further includes receiving user input relating to updating the magnification level associated with the combined view, identifying a portion of the map information based on the updated magnification level, and updating the combined view, the combined view including a graphic representation of a portion of the map information. Updating the combined view includes resizing the autonomous visualization. The autonomous visualization comprises a graphic depiction of the vehicle, and the graphic depiction is resized based on the updated magnification level. First situation information indicates that the vehicle is in a parking lot, and second situation information is associated with the control of vehicle functions via an integrated user interface. The control of vehicle functions includes the control of heating, ventilation, and air conditioning systems, or the control of a music application, or the control of a navigation user interface.The integrated user interface includes multiple icons associated with each vehicle function, and updating the integrated user interface includes updating the integrated user interface to include a menu associated with the selected icon, the menu being included in the first part of the integrated user interface, updating and resizing the combined view, the combined view being included in the second part not obstructed by the menu. The first status information indicates that the vehicle is traveling along a route, the combined view further aggregates the navigation information, and the second status information indicates that the vehicle has moved along the route. The route is associated with multiple driving events shown in the navigation information, and updating the integrated user interface includes identifying a subset of driving events that are within a threshold distance of the vehicle's position or within a threshold driving time of that position, and determining that the magnification level should be adjusted based on the identified subset. The determination that the magnification level should be adjusted is based on one or more complexity measures associated with the subset of driving events. The determination that the magnification level should be adjusted is based on the number of identified subsets that exceed the threshold. The size of the autonomous visualization is adjusted based on the adjusted magnification level, and the area related to the map information is expanded based on the adjusted magnification level. The route is associated with multiple driving events shown in the navigation information, and the combined view shows the first driving event among the multiple driving events, and the method further includes adjusting the magnification level, and the combined view shows multiple second driving events among the multiple driving events. The route is associated with multiple driving events shown in the navigation information, and updating the integrated user interface includes identifying at least one driving event that is within a threshold distance of the vehicle's position or within a threshold driving time of that position, and determining that the magnification level should be adjusted based on at least one driving event. At least one driving event indicates a transition from a highway to a ground road.The integrated user interface responds to user input, such as adjusting the magnification level or moving map information along a specific direction. The operation further includes receiving user input to move map information along a specific direction, the user input including a swipe operation to indicate a specific direction, accessing a portion of the map information based on the map information movement, and updating the combined view to present the portion of the map information. Updating the combined view includes identifying that the vehicle's location is not included in the portion of the map information, and updating the combined view to include the portion of the map information, the autonomous visualization comprising a graphic depiction of the vehicle, the vehicle's graphic depiction not included in the combined view. Updating the combined view includes identifying that any location of the vehicle is not included in the portion of the map information, adjusting the magnification level so that the updated portion of the map information is identified, the updated portion of the map information includes any location, and updating the combined view based on the adjusted magnification level, the autonomous visualization comprising a graphic depiction of the vehicle, the size of the vehicle's graphic depiction being reduced in the combined view.

[0176] An exemplary embodiment may further include a system comprising one or more processors and a non-temporary storage medium storing instructions for one or more processors to render a user interface for presentation via a display included in the vehicle, wherein the user interface presents a combined view integrating autonomous visualization and map information, the autonomous visualization comprising a graphic representation of the vehicle, presenting, responding to the selection of one of a plurality of icons, the icons associated with the control of each vehicle function, the user interface presenting an icon user interface in response to the selection, and dynamically adjusting the combined view so that the icon user interface does not obstruct the combined view.

[0177] The above embodiments may include one or more of the following: The display is touch-sensitive, and the user interface responds to touch input. The user interface responds to one or more verbal commands. Instructions cause one or more processors to access situational information associated with the operation of the vehicle, the situational information indicating that the vehicle is in driving mode, and the autonomous visualization includes a graphic representation of the real-world environment in close proximity to the vehicle. The graphic representation of the real-world environment includes one or more graphic depictions of other vehicles in close proximity to the vehicle. The autonomous visualization is updated by one or more processors at an arbitrary threshold frequency. To update the autonomous visualization, instructions cause one or more processors to perform the following actions: acquire information determined from image sensors positioned around the vehicle, the information reflecting positional information about other vehicles in close proximity to the vehicle; access each model associated with the other vehicles; and render the models for inclusion in a combined view. Vehicle function control includes control of heating, ventilation, and air conditioning systems, or control of a music application, or control of a navigation user interface. The command causes one or more processors to access situational information associated with the vehicle's operation, the situational information indicating that the vehicle is in navigation mode, the combined view further integrates the navigation information, for the combined view, the autonomous visualization includes a graphic representation of the real-world environment in close proximity to the vehicle, for the combined view, the navigation information includes a graphic representation of one or more driving events within a threshold distance or driving time of the vehicle's position, and for the combined view, the map information includes a graphic representation of a portion of the map related to the vehicle's position. A specific driving event includes exiting a highway, and the graphic representation of a specific driving event includes an indication of the number of lanes the vehicle is moving through. The graphic representation of a specific driving event includes an animation of a graphic depiction of the vehicle moving across several lanes. The magnification level associated with the combined view is increased based on the specific driving event.The size of the vehicle's graphic representation is reduced based on a magnification level, expanding a portion of the map. A graphic representation of the route is included in the combined view, and the route graphic representation summarizes one or more upcoming driving events associated with a portion of the map. Autonomous visualizations include a graphic representation of a vehicle in a specific lane of multiple lanes. Dynamically adjusting the combined view includes reducing the size of the combined view within the user interface and adjusting the combined view to become the first part of the user interface. The icon user interface is the second part of the user interface, and the second part does not obstruct the first part. The user interface responds to user input directed to the icon user interface, which causes adjustments to the vehicle functions associated with the selected icon, dynamically adjusting the combined view to substantially encompass the user interface, and removing the icon user interface from its inclusion in the user interface. The icon user interface is associated with connecting a trailer to a vehicle. The icon user interface displays a scale of distance between the trailer and the vehicle, and presents images or videos acquired from image sensors around the vehicle. Autonomous visualization includes a graphic depiction of the trailer. The user interface presents icons on the side of the display close to the user who selects an icon.

[0178] Exemplary embodiments may include methods, systems, and non-temporary computer storage media. Exemplary non-temporary computer storage media store instructions to be executed by a system of one or more processors, the system being installed in a vehicle, and the instructions causing the vehicle's display to display an integrated user interface comprising a combined view aggregating autonomous visualization, map information, and navigation information, wherein the combined view is associated with magnification levels indicating areas of the real-world environment reflected in the combined view, and the vehicle is in a navigation mode associated with a destination; accessing sensor information from a plurality of sensors, the sensor information comprising the vehicle's position and images from image sensors positioned around the vehicle; and determining that the magnification levels should be adjusted based on the sensor information, and the combined view is updated to be associated with the adjusted magnification levels.

[0179] The embodiments described above may include one or more of the following: For a combined view, the autonomous visualization comprises a graphic depiction of the vehicle and a graphic representation of the real-world environment adjacent to the vehicle, the graphic representation being generated based on images from an image sensor; map information comprising a graphic representation of a portion of a map associated with a region of the real-world environment; and navigation information comprising one or more graphic representations of driving events associated with a region of the real-world environment. The graphic representation of the real-world environment depicts one or more lanes associated with the road on which the vehicle is driving, and the graphic depiction of the vehicle is depicted in a specific lane of the one or more lanes. The specific lane is identified based on images from an image sensor. Determining that the magnification level should be adjusted based on sensor information includes identifying a certain number of driving events within a threshold distance or driving time of the vehicle's position, based on navigation information, and the decision is based on a certain number of driving events. Determining that the level at which the magnification should be adjusted is based on one or more measures of complexity associated with a certain number of driving events. Determining that the level at which the magnification should be adjusted is based on a certain number of driving events exceeding a threshold. Determining that the magnification level should be adjusted based on sensor information includes identifying, based on sensor and navigation information, that upcoming driving events should be skipped so that the route to the destination is updated, and adjusting the magnification level, with the combined view including a graphic representation of the updated route. Identifying that upcoming driving events should be skipped is based on images from image sensors, which are analyzed to identify hazards associated with the upcoming driving events. Determining that the magnification level should be adjusted based on sensor information includes identifying, based on sensor and navigation information, that upcoming driving events are of a specific type associated with the adjustment of the magnification level, and adjusting the magnification level, with the combined view including a graphic representation of one or more driving events associated with the adjusted magnification level. Specific types include exiting a highway onto a ground road.The combination view is updated to show the paths associated with the adjusted magnification levels.

[0180] Exemplary embodiments may include methods, systems, and non-temporary computer storage media. An exemplary method, implemented by a system of one or more processors, the system communicates with one or more sensors located inside a vehicle, and the method includes identifying the presence of occupants inside the vehicle, determining a portion of the occupants to be tracked based on sensor information from the sensors, and controlling the operation of vehicle functions based on the tracking of a portion of the occupants, wherein the vehicle functions include air conditioning control, mirror control, and / or steering wheel arrangement.

[0181] The embodiments described above may include one or more of the following: Identifying the presence of an occupant is based on one or more images acquired from opposing image sensors within the vehicle. Identifying the presence of an occupant is based on pressure sensors included in the seats of the vehicle. Identifying the presence of an occupant is based on the detection of a user device used by the occupant, and the system communicates wirelessly with the user device. Identifying the presence of an occupant is based on the interruption of an infrared emitter directed towards the seat. The operation further includes storing information available for recognizing an occupant, the information being based on one or more images of the occupant's face. The information comprises coded vectors into a trained vector space, the coded vectors being generated via a machine learning model. The operation further includes generating a profile associated with the occupant, the profile being associated with the information available for recognizing the occupant, and the profile indicating the occupant's preferences regarding the operation of vehicle functions. Determining which occupants are being tracked is based on one or more images of the occupants from opposing image sensors within the vehicle. Controlling the operation of vehicle functions includes adjusting the heating, ventilation, and air conditioning (HVAC) systems to maintain the direction of airflow towards which occupants are being reassigned. Controlling the operation of vehicle functions includes adjusting one or more mirrors based on a portion of the occupant, the portion of which is the occupant's eyes. The operation further includes outputting a user interface via the vehicle's display, the user interface presenting recommendations indicating that the occupant should adjust the position of the occupant's head.

[0182] It should be understood that not all objectives or benefits may necessarily be achieved by following any particular embodiment described herein. Therefore, for example, a particular embodiment may be configured to operate in a manner that achieves or optimizes one benefit or group of benefits taught herein, without needing to achieve any other objectives or benefits that may be taught or suggested herein.

[0183] All processes described herein are embodied in software code modules executed by a computing system including one or more general-purpose computers or processors, and can be fully automated by the software code modules. The code modules may be stored in any kind of non-temporary computer-readable medium or other computer storage device. Alternatively, some or all of the methods may be embodied in dedicated computer hardware. Furthermore, the components referred to herein may be implemented in hardware, software, firmware, or a combination thereof.

[0184] Many other variations not described herein will be apparent from this disclosure. For example, depending on the embodiment, any particular operation, event, or function of any of the algorithms described herein may be performed in a different order, and may be added, merged, or excluded entirely (for example, not all described operations or events are necessary for the implementation of the algorithm). Furthermore, in certain embodiments, operations or events may be performed not sequentially, but simultaneously, for example, through multithreading, interrupt handling, or via multiple processors or processor cores, or on other parallel architectures. Moreover, various tasks or processes may be performed by different machines and / or computing systems that can work together.

[0185] The various exemplary logic blocks, modules, and algorithmic elements described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, and elements are described above in general terms with respect to their function. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The described functions may be implemented in various ways for each specific application, but such implementation decisions should not be construed as resulting in a departure from the scope of this disclosure.

[0186] Various exemplary logic blocks and modules stored in relation to the embodiments disclosed herein may be implemented or carried out by machines such as processing units or processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. The processor may be a microprocessor, but in alternative examples, the processor may be a controller, microcontroller, or state machine, or a combination thereof. The processor may include electrical circuits configured to process computer executable instructions. In another embodiment, the processor includes an FPGA or other programmable device that performs logical operations without processing computer executable instructions. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Although this specification primarily describes digital technologies, the processor may also include primarily analog components. For example, some or all of the signal processing algorithms described herein may be implemented in analog circuits or mixed analog and digital circuits. The computing environment may include any type of computer system, including, but not limited to, a microprocessor, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computer system based on an in-device computing engine.

[0187] Elements of methods, processes, or algorithms described in relation to embodiments disclosed herein may be directly embodied in hardware, software modules stored in one or more memory devices and executed by one or more processors, or a combination of the two. Software modules may reside in one or more non-temporary computer-readable storage media or physical computer storage devices, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form known in the art. Exemplary storage media may be coupled to a processor so that the processor can read information from and write information to the storage media. Alternatively, the storage media may be integrated with the processor. The storage media may be volatile or non-volatile.

[0188] In particular, conditional language such as “can,” “could,” “might,” or “may” is generally understood to be used in contexts where it is commonly used to convey that a particular embodiment includes certain features, elements, and / or steps, but other embodiments do not, unless otherwise specified. Accordingly, such conditional language is generally not intended to mean that features, elements, and / or steps are required in any way in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or should be implemented in any particular embodiment, with or without user input or prompting.

[0189] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally understood in commonly used contexts to indicate that an item, term, etc., can be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise specified. Therefore, such disjunctive language is not, and should not be intended to, mean that a particular embodiment requires at least one of X, at least one of Y, or at least one of Z, respectively, to exist.

[0190] Any process description, element, or block in the flowcharts described herein and / or shown in the accompanying drawings should be understood as potentially representing a module, segment, or portion of code containing one or more executable instructions for implementing a particular logical function or element in the process. As will be understood by those skilled in the art, depending on the function included, alternative implementations are included within the scope of the embodiments described herein, in which elements or functions may be deleted or executed in the illustrated or described order, including substantially simultaneously or in reverse order.

[0191] Unless otherwise specified, articles such as "a" or "an" should generally be interpreted as including one or more of the listed items. Therefore, phrases such as "devices configured to do ~" are intended to include one or more of the enumerated devices. Such one or more enumerated devices may also be collectively configured to perform the stated enumeration. For example, "processors configured to perform enumerations A, B, and C" may include a first processor configured to perform enumeration A, which works in conjunction with a second processor configured to perform enumerations B and C.

[0192] It should be emphasized that many variations and modifications can be made to the embodiments described above, and that elements thereof are understood to be found in other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure.

Claims

1. A method carried out by a system of one or more processors, wherein the system is included in a vehicle. The method described above is The steps include providing a combined view via the vehicle's display that aggregates visualizations and map information regarding the autonomously or semi-autonomously controlled vehicle, and presenting an integrated user interface generated based on first situation information indicating a first operation of the vehicle; A step of accessing second status information indicating a second operation following the first operation of the vehicle, The steps include updating the integrated user interface based on the second status information, Includes, In the step of presenting the integrated user interface, the size of the visualization and the map information is adjusted based on the magnification level, the first action indicates that the vehicle is traveling along the route, and the map information includes a graphic representation of a portion of the map on which the vehicle is driving. In the step of accessing the second situation information, the combined view further aggregates the navigation information, and the second operation indicates that the vehicle has moved along the route. In the step of updating the integrated user interface, the combined view is adjusted by adjusting the size of the combined view within the integrated user interface or by adjusting the magnification level. The step of updating the integrated user interface includes determining a measure of complexity for the remaining portion of the route that is within a threshold distance of the vehicle's position or within a threshold driving time of the position, The aforementioned route is associated with a number of driving events, including the next turn, highway exit, and highway entrance, as indicated in the navigation information. A method wherein the magnification level is adjusted according to the next driving event and the complexity scale, and while driving, the integrated user interface causes the map to zoom out to indicate the next turn based on a series of identified approaching turns, the number of upcoming lane changes, and the complexity scale, which includes the previous driver missing one or more of the driving events.

2. The method according to claim 1, wherein the visualization includes a graphic depiction of the vehicle and a graphic representation of the real-world environment in which the vehicle is located.

3. The method according to claim 2, wherein the graphic representation of the real-world environment comprises a graphic depiction of other vehicles and a graphic representation of one or more objects adjacent to the vehicle.

4. The aforementioned combination view is A graphic representation of one or more lanes of the road on which the vehicle is located, A graphic representation of the vehicle positioned in a specific lane among the one or more lanes, Map information including a graphic representation of a map related to the aforementioned road, The method according to claim 1, wherein the graphic representation of the vehicle is depicted as it appears on the map.

5. The method according to claim 4, wherein the combined view represents a driving view, and the graphic representation of the vehicle is animated as driving in the specific lane.

6. The method described above is The steps include receiving user input related to updating the magnification level associated with the combination view, The steps further include updating the combined view to include a portion of the map information based on the updated magnification level, The method according to claim 1, wherein the combined view includes a graphic representation of a portion of the map information.

7. The method according to claim 1, wherein the visualization comprises a graphic representation of the vehicle, and the graphic representation is reduced in size based on the adjusted magnification level.

8. The aforementioned integrated user interface enables control of vehicle functions, The method according to claim 1, wherein the control of the vehicle functions includes control of heating, ventilation, and air conditioning systems, or control of a music application, or control of a navigation user interface.

9. The integrated user interface includes multiple icons associated with each vehicle function, The step of updating the integrated user interface is: The integrated user interface is updated to display the menu associated with the selected icon, The method of claim 8, comprising resizing the combination view, wherein the menu is included in a first portion of the integrated user interface and the combination view is included in a second portion not obstructed by the menu.

10. The size of the visualization is adjusted based on the adjusted magnification level. The method according to claim 1, wherein the region related to the map information is adjusted based on the adjusted magnification level.

11. The method according to claim 1, further comprising the step of adjusting the magnification level.

12. The step of updating the integrated user interface is: Identifying at least one driving event that is within a threshold distance of the vehicle's position or within a threshold driving time of the position, The method according to claim 1, comprising determining that the magnification level should be adjusted based on at least one of the driving events.

13. The method according to claim 1, wherein the integrated user interface responds to user input that causes the magnification level to be adjusted or the map information to move along a specific direction.

Images