Vehicle and Mobile Device Interfaces for Vehicle Occupant Assistance

Abstract

A system, method, and non-transitory medium are provided for a vehicle and mobile device interface for vehicle occupant assistance. An example method can include determining an attitude of the mobile device relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle, determining a state of an occupant of the vehicle, and transmitting data to the vehicle indicative of the state of the occupant and the attitude of the mobile device relative to the coordinate system of the vehicle.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to extended reality for vehicle occupant assistance. For example, aspects of the present disclosure relate to a vehicle-to-device interface for extended reality in a vehicle. [Background technology]

[0002] Many devices and systems enable the capture of a scene by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a device with a camera can capture a sequence of frames of a scene (e.g., a video of the scene). In some cases, the sequence of frames can be processed for one or more functions, output for display, output for processing and / or consumption by other devices, among other uses. A vehicle is an example of a device that can include one or more cameras. For example, a vehicle can include a camera that can capture frames of an area inside the vehicle and / or outside the vehicle. The frames can be processed for various purposes, such as determining or recognizing road conditions, identifying other vehicles, objects, pedestrians, and / or obstacles in proximity to the vehicle, among other purposes.

[0003] An extended reality (XR) device is another example of a device that may include one or more cameras. XR devices may include augmented reality (AR) devices, virtual reality (VR) devices, mixed reality (MR) devices, and the like. For example, examples of AR devices include smart glasses and head-mounted displays (HMDs). In general, AR devices may implement cameras and various sensors to track the position of the AR device and other objects in a physical environment. The AR device may use the tracking information to provide a realistic AR experience to a user of the AR device. For example, the AR device may enable a user to experience or interact with an immersive virtual environment or content. To provide a realistic AR experience, AR technology generally aims to integrate virtual content with the physical world. In some examples, the AR technology may match the relative pose and movement of an object and a device. For example, the AR device may use the tracking information to calculate the relative pose of a map of the device, object, and / or real-world environment, and match the relative position and movement of the device, object, and / or real-world environment. Using the pose and motion of one or more devices, objects, and / or the real-world environment, the AR device can anchor the content in a convincing manner to the real-world environment. The relative pose information can be used to match the virtual content with the user's perceived motion and the spatiotemporal state of the devices, objects, and real-world environment. Summary of the Invention

[0004] Systems and techniques for integrating mobile devices, such as augmented reality (AR) devices, with vehicle operation are described herein. According to at least one example, a method for augmented reality for vehicle occupant assistance is provided. The method can include determining an attitude of the mobile device relative to a vehicle coordinate system based on one or more images of an interior of the vehicle, determining a state of an occupant of the vehicle, and transmitting data to the vehicle indicative of the state of the occupant and the attitude of the mobile device relative to the vehicle coordinate system.

[0005] In accordance with at least one example, a non-transitory computer-readable medium for augmented reality for vehicle occupant assistance is provided. The non-transitory computer-readable medium can include stored instructions that, when executed by one or more processors, cause the one or more processors to determine an attitude of a mobile device (e.g., a device) relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle, determine a state of an occupant of the vehicle, and transmit data to the vehicle indicative of the state of the occupant and the attitude of the mobile device relative to the coordinate system of the vehicle.

[0006] According to at least one example, an apparatus for augmented reality for vehicle occupant assistance is provided. The apparatus may include a memory and one or more processors coupled to the memory, where the one or more processors are configured to determine, based on one or more images of an interior of the vehicle, an attitude of a mobile device (e.g., the device) relative to a coordinate system of the vehicle, determine a state of an occupant of the vehicle, and transmit data to the vehicle indicative of the state of the occupant and the attitude of the mobile device relative to the coordinate system of the vehicle.

[0007] In accordance with at least one example, another apparatus for augmented reality for vehicle occupant assistance is provided that can include means for determining an attitude of a mobile device relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle, means for determining a state of an occupant of the vehicle, and means for transmitting data to the vehicle indicative of the state of the occupant and the attitude of the mobile device relative to the coordinate system of the vehicle.

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may determine a context for the vehicle based on data associated with one or more sensors. In some examples, the context may include an event associated with the vehicle.

[0009] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can determine the line of sight of an occupant of a vehicle. In some examples, the state of the occupant includes the line of sight of the occupant, and the occupant is associated with a mobile device (e.g., the occupant is wearing the mobile device).

[0010] In some examples, the occupant's condition can include an occupant's impairment with respect to operating the vehicle. In some cases, the impairment can include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an impairment location to control the vehicle, and an impaired visibility with respect to at least one of the operation of the vehicle and an event associated with the vehicle.

[0011] In some cases, determining the state of the occupant may include receiving, from the vehicle, data associated with one or more sensors of the vehicle and determining the state of the occupant based on the data associated with the one or more sensors of the vehicle and an attitude of the mobile device. In some examples, the data associated with the one or more sensors of the vehicle indicates at least one of a state of the vehicle and an event associated with the vehicle.

[0012] In some examples, the events associated with the vehicle may include at least one of: a presence of an object in the path of the vehicle or a threshold proximity to the path of the vehicle, a traffic regulation associated with the path of the vehicle, and the vehicle failing to stay within at least one of a speed limit and a lane marking. In some cases, the object in the path of the vehicle or the threshold proximity to the path of the vehicle may include at least one of a pedestrian, an animal, and another vehicle.

[0013] In some cases, determining the occupant's status may include receiving one or more health measurements associated with the occupant from one or more sensors associated with at least one of the mobile device and a wearable device worn by the occupant, and determining the occupant's status based on the one or more health measurements. In some examples, the one or more health measurements may include at least one of a heart rate, blood pressure, body temperature, a galvanic skin response, a measurement of an electrical signal from the occupant's heart, a measurement of electrical activity of the occupant's brain, an amount of redness in the eyes, and a pupil size.

[0014] In some aspects, determining the occupant's condition may include determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time, and determining an impairment condition of the occupant based on a determination that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time and that the period of time exceeds a threshold period of time.

[0015] In some cases, determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time may include determining that the occupant's gaze is focused on virtual content rendered by the mobile device for at least a portion of the period of time.

[0016] In some examples, determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time may include determining that the occupant's gaze is focused in the direction of an obstacle in the vehicle's path or in a direction other than a threshold proximity to the vehicle's path.

[0017] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may transmit an indication of the occupant's status to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0018] In some cases, determining the occupant state may include determining a line of sight of an occupant wearing the mobile device, and determining the occupant state based on the attitude of the mobile device and the occupant's line of sight.

[0019] In some examples, determining the pose of the mobile device may include receiving, from the vehicle, a vehicle template including one or more markers associated with the vehicle, and determining a pose of the mobile device relative to a coordinate system of the vehicle based on the one or more images and the vehicle template.

[0020] In some examples, the one or more markers may include at least one of a visual pattern of an area within the interior of the vehicle and / or an object attached to the interior of the vehicle, an element within the interior of the vehicle, a surface within the interior of the vehicle, and an illuminated object inside the vehicle.

[0021] In some examples, the one or more images depict one or more markers, and determining the pose of the mobile device may include detecting the one or more markers in the one or more images, and determining a pose of the mobile device relative to a coordinate system of the vehicle based on the detected one or more markers and the vehicle template.

[0022] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may use one or more image sensors of a mobile device to acquire a set of images of an interior of a vehicle, the set of images depicting one or more visual landmarks associated with the vehicle, and generate a vehicle template based on the set of images. In some examples, the vehicle template may include the one or more visual landmarks.

[0023] In some cases, determining the attitude of the mobile device may include obtaining inertial sensor data associated with the mobile device and determining the attitude of the mobile device based on one or more of the images and the inertial sensor data.

[0024] In some aspects, each of the above-mentioned devices may be, be part of, or include a mobile device, a wearable device, a camera system, a personal computing device, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some examples, the device may include, be part of, and / or interface with a vehicle, a mobile device (e.g., a mobile phone or a so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotics device or system, an aviation system, or other device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the pose of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or other state), and / or for other purposes. As used herein, the term pose refers to the position and orientation of a device, sensor, or other real-world device or structure relative to a coordinate system.

[0025] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0026] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0027] Illustrative examples of the present application are described in detail below with reference to the following figures: [Brief description of the drawings]

[0028] [Figure 1] FIG. 2 illustrates an example of a computing system of a mobile device, according to some examples of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a vehicle computing system, according to some examples of the present disclosure. [Diagram 3] FIG. 1 illustrates an example system process for in-vehicle localization, according to some examples of the present disclosure. [Figure 4A] FIG. 1 illustrates an example system process for augmented reality for vehicle occupant assistance, according to some examples of the present disclosure. [Figure 4B] FIG. 1 illustrates an example system process for augmented reality for vehicle occupant assistance, according to some examples of the present disclosure. [Diagram 5] FIG. 1 illustrates an example use case for adjusting virtual content rendered for vehicle occupants, according to some examples of the present disclosure. [Figure 6A] FIG. 1 illustrates an example use case for adjusting virtual content, according to some examples of the present disclosure. [Figure 6B] FIG. 1 illustrates an example use case for adjusting virtual content, according to some examples of the present disclosure. [Figure 6C] FIG. 1 illustrates an example use case for adjusting virtual content, according to some examples of the present disclosure. [Figure 6D] FIG. 1 illustrates an example use case for adjusting virtual content, according to some examples of the present disclosure. [Figure 6E] FIG. 1 illustrates an example use case for adjusting virtual content, according to some examples of the present disclosure. [Figure 7A] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7B] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7C] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7D] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7E] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7F] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7G] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7H] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 7I] 1A-1C are diagrams illustrating different exemplary states of an occupant operating a vehicle, according to some examples of the present disclosure. [Figure 8] FIG. 1 illustrates an example of vehicle-to-everything communication including an occupant monitoring event, according to some examples of the present disclosure. [Figure 9] FIG. 1 illustrates an example vehicle mitigation event based on occupant state determined by a mobile device worn by an occupant driving a vehicle, according to some examples of the present disclosure. [Figure 10] FIG. 1 illustrates an example vehicle mitigation event based on occupant state determined by a mobile device worn by an occupant driving a vehicle, according to some examples of the present disclosure. [Figure 11] 1 is a flowchart illustrating an example process for controlling the presentation of virtual content during operation of a vehicle, according to some examples of the present disclosure. [Figure 12] 1 is a flowchart illustrating an example process for monitoring occupants of a vehicle, according to some examples of the present disclosure. [Figure 13] 1 is a flowchart illustrating an example process for controlling operation of a vehicle, according to some examples of the present disclosure. [Figure 14] 1 is a flowchart illustrating an example process for interfacing a vehicle with a mobile device associated with an occupant of the vehicle, according to some examples of the present disclosure. [Figure 15] FIG. 2 illustrates an example of a computing device architecture, according to some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Some aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are described to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0030] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the present application as set forth in the appended claims.

[0031] An extended reality (XR) system or device can provide virtual content to a user and / or combine a real-world or physical environment with a virtual environment (consisting of virtual content) to provide an XR experience to the user. The real-world environment can include real-world objects (also called physical objects), such as books, people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. The XR system or device can facilitate interaction with different types of XR environments (e.g., a user can interact with an XR environment using an XR system or device). An XR system can include a VR system that facilitates interaction with a virtual reality (VR) environment, an AR system that facilitates interaction with an augmented reality (AR) environment, an MR system that facilitates interaction with a mixed reality (MR) environment, and / or other XR systems. As used herein, the terms XR system and XR device are used interchangeably. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, among others. In some cases, the XR system may track parts of the user (e.g., the user's hands and / or fingertips) to enable the user to interact with items of virtual content.

[0032] AR is a technology that provides virtual or computer-generated content (referred to as AR content) over a user's view of a physical, real-world scene or environment. AR content can include virtual content such as video, images, graphic content, location data (e.g., Global Positioning System (GPS) data or other location data), sound, any combination thereof, and / or other augmented content. AR systems or devices are designed to enhance (or augment), rather than replace, a real person's current perception. For example, a user can see an actual stationary or moving physical object through an AR device display, but the user's visual perception of the physical object may be augmented or enhanced by a virtual image of that object (e.g., a real-world car replaced by a virtual image of a DeLorean), by AR content added to the physical object (e.g., virtual wings added to a live animal), by AR content displayed relative to the physical object (e.g., informational virtual content displayed near a sign on a building, a virtual coffee cup virtually anchored (e.g., placed on) a real-world table in one or more images, etc.), and / or by displaying other types of AR content. Various types of AR systems can be used for games, entertainment, and / or other applications.

[0033] In some cases, two types of AR systems that can be used to provide AR content include video see-through (also called video pass-through) displays and optical see-through displays. Video see-through and optical see-through displays can be used to enhance a user's visual perception of real-world or physical objects. In a video see-through system, a live video of a real-world scenario is displayed (e.g., including one or more objects augmented or enhanced on the live video). A video see-through system can be implemented using a mobile device (e.g., video on a mobile phone display), an HMD, or other suitable device that can display video and computer-generated objects on top of the video.

[0034] An optical see-through system with AR features can display AR content directly on a view of a real-world scene (e.g., without displaying video content of the real-world scene). For example, a user may view a physical object in the real-world scene through a display (e.g., glasses or lenses), and the AR system can display (e.g., project or otherwise display) AR content on the display to provide the user with an enhanced visual perception of one or more real-world objects. An example of an optical see-through AR system or device is an AR pair of glasses, an HMD, another AR headset, or other similar device that can include a lens or glass in front of each eye (or a single lens or glass on both eyes) to allow a user to directly view a real-world scene with a physical object, and an augmented image of the object or additional AR content can be projected onto the display to enhance the user's visual perception of the real-world scene.

[0035] VR provides a fully immersive experience in a three-dimensional computer-generated VR environment or video that depicts a virtual version of a real-world environment. The VR environment can be interacted with in a seemingly realistic or physical manner. As a user experiencing the VR environment moves in the real world, the images rendered in the virtual environment also change, giving the user the perception that they are moving within the VR environment. For example, the user can turn left or right, look up or down, and / or move forward or backward, thus changing the user's perspective of the VR environment. The VR content presented to the user can change accordingly, so that the user's experience is as seamless as the real world. VR content can, in some cases, include VR video, which can be captured and rendered with extremely high quality, potentially providing a truly immersive virtual reality experience. Example applications of virtual reality can include gaming, training, education, sports videos, online shopping, among others. VR content can be rendered and displayed using a VR system or device, such as a VR HMD or other VR headset that completely covers the user's eyes during the VR experience.

[0036] MR technology can combine aspects of VR and AR to provide an immersive experience for the user. For example, in an MR environment, real-world and computer-generated objects can interact (e.g., a real person can interact with a virtual person as if the virtual person were a real person).

[0037] Described herein are systems, apparatus, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and technologies") for integrating data, functions, actions, and / or device capabilities from vehicles and mobile (portable and / or wearable) devices, such as XR devices (e.g., augmented reality (AR) or mixed reality (MR) devices, virtual reality (VR) devices, etc.), wearable devices with networking / communication capabilities, smartphones, personal computing devices, etc. In some examples, the systems and technologies described herein can provide assistance to vehicle occupants using data and / or functionality from XR devices (e.g., AR devices, MR devices, etc.) and / or vehicles. For example, in some cases, the systems and technologies described herein can provide vehicle occupants with a vehicle-to-device (e.g., vehicle-to-mobile computing device) interface for augmented reality assistance.

[0038] In some examples, a vehicle may include a camera that can capture frames of the interior of the vehicle and / or an area external to the vehicle (e.g., the vehicle surroundings). The frames may be processed for various purposes, such as determining or recognizing road conditions, recognizing the identity of a person(s) in the vehicle, and identifying other vehicles, objects, pedestrians, and / or obstacles in proximity to the vehicle, among others. The frames may be processed for various purposes, such as determining and / or recognizing activities and / or events in an environment associated with the vehicle (e.g., an environment external to the vehicle, an environment inside the vehicle, etc.). The vehicle may also include and / or implement other types of sensor systems for measuring and / or determining various conditions. For example, in some cases, a vehicle may include and / or implement one or more radio detection and ranging (RADAR) systems, inertial measurement units (IMUs), light detection and ranging (LIDAR) systems, ultrasonic sensors, radio frequency (RF) sensors, acoustic navigation and ranging (SONAR) systems, electromagnetic detection and ranging (EmDAR) systems, acoustic detection and ranging (SODAR) systems, global navigation satellite system (GNSS) receiver systems (e.g., one or more global positioning system (GPS) receiver systems), accelerometers, gyroscopes, speed sensors, infrared sensor systems, laser ranging systems, ultrasonic sensor systems, extremely low frequency sensor systems, microphones, any combination thereof, and / or other sensor systems.

[0039] Mobile devices such as XR devices (e.g., head-mounted display AR devices, AR smart glasses, or other XR devices) may also include one or more cameras that can capture images and / or videos. For example, an AR device may implement a camera and various sensors to determine and / or track the position of the AR device and other objects in a physical environment. The AR device may use the tracking information to provide a realistic AR experience to a user of the AR device. For example, the AR device may enable a user to experience or interact with an immersive virtual environment or content. As described above, to provide a realistic AR experience, AR technology generally aims to integrate virtual content with the physical world. In some examples, the AR device may use the tracking information from one or more sensors to calculate a relative pose of a map of the device, object, and / or real-world environment and match the relative position and movement of the device, object, and / or real-world environment. Using the pose and movement of the one or more devices, objects, and / or real-world environment, the AR device may anchor the content in a convincing manner to the real-world environment. The relative pose information can be used to match virtual content with the user's perceived motion and the spatiotemporal states of devices, objects, and the real-world environment.

[0040] A user of an XR device (e.g., an AR device) may wear the XR device while the user is in a vehicle (e.g., while the user is driving the vehicle or as a passenger in the vehicle). For example, in some cases, the user may wear the AR device while operating (e.g., driving) the vehicle. In some cases, a user (e.g., a driver) wearing an AR device and operating a vehicle may be impaired by one or more conditions, such as, for example, distraction. In other words, the user's ability to operate the vehicle may be impaired by such conditions. Distraction may be caused by a variety of things, such as sources external to the user. For example, the user may be distracted by virtual content rendered by the AR device (e.g., email notifications, world-locked virtual billboards or signs, application content, video and / or image content, interface content, etc.), objects and / or activities occurring within an environment outside the vehicle, objects and / or events occurring inside the vehicle, other occupants of the vehicle, thoughts and / or inattention (e.g., daydreaming, etc.), sounds or noises (e.g., inside and / or outside the vehicle), a user device, etc.

[0041] Even without such distractions, vehicle occupants (e.g., drivers) may generally suffer from other impairments. Other impairments may include and / or result from certain physical and / or personal limitations and / or conditions, such as, for example, perceptual limitations, attention limitations, real-world events and conditions, sleep disturbances, intoxications, health conditions, etc. For example, a vehicle occupant (e.g., driver) wearing an AR device may have difficulty seeing certain areas / views of the vehicle's surroundings from the occupant's perspective or field of view (e.g., areas behind the vehicle, areas below the vehicle, areas close to the exterior of the vehicle, areas obstructed by one or more objects and / or conditions (e.g., poor lighting, etc.), areas above the vehicle, areas a certain distance away from the vehicle, etc.). Additionally, certain road conditions may be invisible or difficult to see to the vehicle occupant, such as standing water, ice, obstacles beyond the reach of one or more lights of the vehicle (e.g., headlights, fog lights, off-road lights, emergency lights, signal lights, daytime running lights, reverse lights, tail lights, brake lights, etc.). Vehicle occupants may also be distracted by certain events and objects, such as passing vehicles, pedestrians, surrounding activities / events, navigation alerts, etc. In some cases, such distractions may interfere with the driver's ability to safely operate the vehicle or respond to driving conditions.

[0042] In some aspects, the systems and techniques described herein can enable the vehicle and the XR device (e.g., AR device) to interface with each other, such as to share / integrate data, functions, actions, and / or device capabilities from the vehicle and the XR device (e.g., AR device). By interfacing with each other and sharing / integrating data, the vehicle and the XR device (e.g., AR device) can provide assistance to vehicle occupants, such as, for example, the vehicle driver / operator, the vehicle passengers, etc. In some examples, the XR device (e.g., AR device) can locate itself (determine its position and / or orientation, generally its pose) within the vehicle (e.g., within the interior of the vehicle, such as the vehicle cabin) using data from one or more sensors on the vehicle and / or the XR device. In some cases, the XR device may implement a localization process that may perform image-based (e.g., visual-based) and / or audio-based (e.g., via audio beamforming) localization of in-vehicle landmarks / markers (e.g., Quick Response (QR) codes in the vehicle, lights in the vehicle, objects in the vehicle (e.g., doors, windows, seats, headrests, dashboard components, vehicle control systems (e.g., steering wheel, horn, signaling systems, etc.), patterns in the vehicle, shapes in the vehicle, etc.). The localization process may use such in-vehicle landmarks / markers to locate the XR device in the vehicle. For example, the localization process may use the in-vehicle landmarks / markers to determine the pose of the XR device relative to the coordinate system of the XR device and / or the vehicle.

[0043] In some cases, the localization process may detect in-vehicle landmarks based on data from one or more devices (e.g., sensors, emitters, transceivers, imaging devices, etc.) on the XR device (e.g., AR device), such as one or more cameras, wireless interfaces, ultrasonic sensors, radar, etc. For example, auxiliary sensor data from one or more IMUs may be used to track the pose of the XR device. In some cases, the localization process may use a vehicle template to determine the location of the XR device relative to one or more in-vehicle landmarks specified in the vehicle template and detected as described above. For example, the localization process may use the vehicle template to identify one or more in-vehicle landmarks that can be used for localization. In some cases, the vehicle template may specify coordinates, such as position and / or orientation information, of one or more in-vehicle landmarks, which the localization process may use to localize itself relative to the coordinate system of the XR device (e.g., AR device) and / or the vehicle. For example, the localization process may use data from one or more devices (e.g., sensors, emitters, transceivers, etc.) as described above to detect one or more in-vehicle landmarks identified in the vehicle template. The vehicle template can specify coordinates of one or more in-vehicle landmarks, which can define the position and / or orientation of the one or more in-vehicle landmarks relative to the vehicle's coordinate system. The vehicle template can be vehicle specific or specific to a vehicle model, specification, series, class, or combinations thereof. The localization process can use the coordinates specified in the vehicle template to determine the position and / or orientation of one or more in-vehicle landmarks detected by the XR device (e.g., the AR device) relative to the vehicle's coordinate system.The localization process may translate, translate, and / or correlate the location and / or orientation of one or more in-vehicle landmarks relative to the vehicle's coordinate system to a location and / or orientation relative to the XR device's coordinate system. The XR device (e.g., AR device) may use the location and / or orientation information associated with one or more in-vehicle landmarks to understand, determine, and / or track the pose of the XR device within the vehicle and relative to the XR device's coordinate system and / or the vehicle's coordinate system.

[0044] In some aspects, the XR device can implement an occupant monitoring process, such as a driver monitoring process that can monitor the user for obstacles based on vehicle conditions, occupant positions / orientations, virtual content rendered by the XR device, eye tracking using one or more cameras on the XR device, inertial sensor data, audio sensor data, radar data, radio signals, etc. In some examples, the XR device can include a virtual content filtering process that can filter or block virtual content from being presented to an occupant (e.g., driver or passenger) based on conditions from the vehicle and occupant monitoring processes. A vehicle user interface process can render user interface elements, for example, based on conditions from the vehicle and occupant monitoring processes.

[0045] In some cases, the vehicle may implement a vehicle monitoring process that may recognize and monitor conditions and / or events related to the vehicle, the vehicle's occupants, and / or the vehicle's surroundings. The vehicle monitoring process may transmit data regarding such conditions / events (e.g., via a wireless radio link) to an XR device (e.g., an AR device) that is in wireless communication with the vehicle, e.g., paired with the vehicle. The XR device may use such data for occupant monitoring processes, virtual content filtering processes, vehicle user interface processes, etc. In some examples, the vehicle may implement an event mitigation process that may modify vehicle operation and / or autonomous driving policies based on occupant (e.g., driver) monitoring events generated by the XR device, such as occupant conditions and / or activities detected by the XR device.

[0046] As previously mentioned, in some cases, an in-vehicle localization process of an XR device (e.g., an AR device) can enable the XR device to understand the pose of the XR device in the vehicle. The XR device can use the localization process to determine its pose relative to and / or within the vehicle's coordinate system. For example, to render data and / or events from vehicle-based sensors and / or instructions, the XR device can use the localization process to gain a common understanding of the XR device's pose relative to the vehicle's coordinate system. In some cases, the localization process can perform a transformation from the vehicle's coordinate system to the XR's coordinate system. In some cases, the vehicle can provide a vehicle template to the XR device to help the XR device locate itself within the vehicle and / or map the interior of the vehicle into three-dimensional (3D) space. The vehicle template can include, for example, but not limited to, visual, IR, and / or RF descriptors of one or more in-vehicle landmarks. Non-limiting examples of intra-vehicle landmarks can include, among others, visual patterns (e.g., QR codes, calibration patterns such as checkerboard patterns, engravings / carvings, patterned materials, labels, etc.) attached to objects or areas within the interior (e.g., cabin) of the vehicle (e.g., objects / areas within the cabin, areas on the windshield, objects / areas on the dashboard, objects / areas on the doors, objects / areas on the seats, etc.), active lighting (e.g., Light Emitting Diodes (LEDs), etc.) attached to objects and / or areas within the interior of the vehicle, immovable elements within the vehicle (e.g., instrument cluster, corner(s) of the dashboard, corner(s) of the window(s) and / or windshield, vehicle roof, vehicle center console, etc.), movable elements within the vehicle that have a known position / offset from the body of the vehicle (e.g., passenger seat(s), steering wheel, etc.). In some examples, the vehicle template can include / specify the location and / or orientation of intra-vehicle landmarks relative to the vehicle's coordinate system.

[0047] The localization process of the XR device (e.g., AR device) can use the in-vehicle landmarks / markers, visual descriptors, IR descriptors, and / or RF descriptors, as well as location and / or orientation information to locate itself relative to the in-vehicle landmarks and the vehicle's coordinate system. For example, using one or more cameras of the XR device, the XR device can perform a localization process to (continuously) search for the in-vehicle landmarks specified in the vehicle template and locate itself relative to the in-vehicle landmarks and (by extension) the vehicle. In some cases, the XR device can also use other sensors / devices to assist with localization, such as, for example, WiFi devices, Bluetooth devices, ultrasonic devices, IMUs, etc.

[0048] The vehicle monitoring process may monitor conditions such as the XR device (e.g., AR device), the vehicle, and / or one or more surroundings. The vehicle monitoring process may recognize events related to the vehicle and its surroundings, driving conditions, road conditions, vehicle conditions, etc. The vehicle monitoring process may transmit data regarding these events to the XR device, which may use this data for occupant monitoring processes, virtual content filtering processes, vehicle user interface processes, etc. Non-limiting examples of data provided by the vehicle monitoring process to the XR device may include instrumentation readings, sensor data (e.g., camera data, RADAR data, SONAR data, SODAR data, EmDAR data, GNSS data, LIDAR data, IMU data, GPS data, etc.), Internet data indicative of conditions in the vehicle's path (e.g., weather conditions, traffic conditions, road conditions, etc.), navigation information, etc. In some examples, the sensor data may be pre-processed by the vehicle before being provided to the XR device, for example, by applying image processing, sensor fusion, object / subject detection, etc. In some examples, at least some sensor data may be provided to the XR device raw.

[0049] In some examples, the sensor data may indicate a state or condition of the vehicle, such as an operating status of the vehicle, a motion state of the vehicle, an autonomous driving policy, a vehicle-related event, e.g., a vehicle part malfunction or a sensor-triggered alert, and combinations thereof, a presence of an object or obstacle in or approaching the path of the vehicle (e.g., as determined in some cases via a machine learning classifier trained on positive and negative examples), a presence of a pedestrian in or approaching the path of the vehicle (e.g., as determined in some cases via a machine learning classifier trained on positive and negative examples), a presence of something in the path of the vehicle (e.g., a pothole, an accident, an animal, a bumpy portion of the road, a lane marking, etc.) (e.g., as determined in some cases via a machine learning classifier trained on positive and negative examples), a presence of an object (e.g., an emergency vehicle, etc.) behind or near the path of the vehicle (e.g., as determined in some cases via a machine learning classifier trained on positive and negative examples). In some cases, the sensor data may indicate traffic regulations (e.g., stop signs, stop lights, etc.) along the vehicle's path (e.g., determined in some cases via a machine learning classifier trained on positive and negative examples), such as that the vehicle is violating lane markings (e.g., determined in some cases via a machine learning classifier trained on positive and negative examples), that the vehicle is exceeding a speed limit (e.g., determined in some cases based on the vehicle's speedometer and a database of speed limit zones), etc.

[0050] In some cases, the vehicle monitoring process may provide the XR device (e.g., the AR device) Vehicle-to-Everything (V2X) (e.g., Vehicle-to-Vehicle (V2V), Vehicle-to-Infrastructure (V2I), etc.) data and / or events. Non-limiting examples of V2X data that the vehicle monitoring process may provide to the XR device include V2X data indicating the presence of an obstacle in the path of the vehicle, V2X data indicating the presence of a pedestrian in the path of the vehicle or approaching the vehicle, V2X data indicating the presence of something (e.g., a pothole, an animal, an object, etc.) in the path of the vehicle, V2X data indicating the presence of an emergency vehicle behind or near the path of the vehicle, V2X data indicating the presence of a traffic regulation (e.g., a stop sign, a stop light, etc.) along the path of the vehicle, V2X data indicating the vehicle is violating a lane marking, V2X data indicating the vehicle is exceeding a speed limit, V2X data indicating the presence of a particular condition (e.g., weather conditions, road conditions, traffic conditions, an accident, etc.) in the path of the vehicle, etc.

[0051] In some examples, the occupant monitoring process can monitor users / occupants for impairments based on event data and / or situational information from the vehicle, virtual content rendered by the XR device (e.g., AR device), occupant position / orientation, user activity (e.g., gestures, movements, etc.), eye tracking, inertial sensor data, audio data (e.g., speech data, acoustic data, sound waves, etc.), user head pose, etc. In some cases, an occupant (e.g., driver, passenger, etc.) impairment may be detected independent of the virtual content rendered by the XR device and / or the vehicle situation. For example, in some cases, an occupant impairment can be detected using sensor data and / or sensor data processing, such as, for example, eye tracking, inertial sensing, heart rate information, temperature data, etc. In some cases, eye tracking, inertial sensing, heart rate information, temperature data, etc. may indicate drowsiness, intoxication, health emergency (e.g., seizure, heart attack, loss of consciousness, etc.), stress or heightened emotional state, general inattention / distraction, etc. (e.g., in some cases determined via machine learning classifiers trained on positive and negative examples). In some cases, impairments may be detected using a combination of virtual content, vehicle situation information, and / or sensor data. For example, in some cases, occupant impairments may be detected based on eye tracking data indicating an occupant's gaze is focused on virtual content for a certain (configurable) period of time, eye tracking data indicating an occupant's gaze is focused elsewhere when a vehicle event monitor detects an impending event or object (e.g., obstacle, pedestrian, pothole, animal, etc.), ray tracing of the occupant's gaze, vehicle templates, location information at reported vehicle events, sensor data indicating the occupant's position and / or movement, etc.

[0052] In some cases, the XR device may adjust / modify (e.g., via a virtual content filtering process and / or a user interface process) the virtual content presented by the XR device to the occupant. For example, the XR device may adjust the virtual content in response to a determination by the occupant monitoring process that the occupant is impaired, a report from the vehicle monitoring process that a vehicle event is imminent or occurring, a determination that virtual content requested by the occupant may distract the occupant from a particular driving and / or related event, etc. In some examples, the XR device may adjust the virtual content by disabling, dimming, or increasing the transparency of all virtual content except a subset of virtual content marked as necessary (e.g., head-up display (HUD) content, vehicle instrumentation content, etc.), any portion of the virtual content on which the occupant is focused (e.g., as the case may be determined based on ray tracing of the occupant's line of sight and / or image data capturing the occupant's line of sight), any virtual content that is obscuring the vehicle event (e.g., as the case may be determined based on ray tracing from the XR device to the location of the vehicle event, the vehicle template, and / or information in the reported vehicle event), etc.

[0053] In some cases, in addition to disabling distracting virtual content, the AR device can render other virtual content to wake impaired occupants and / or highlight the impending or occurring vehicle event. In some examples, the XR device can pulse the edges of the occupant's field of view (FOV), circle or highlight the location of the vehicle event in the displayed image / video, render head-locked arrows, pulse the periphery of the occupant's field of view and use other directional indicators to direct the occupant's head and / or gaze in the direction of the vehicle event, stream vehicle camera feeds (e.g., backup cameras, side-view cameras, etc.) near the location of the vehicle event, stream vehicle conditions (e.g., instrumentation, etc.) as head-locked HUD or world-locked user interface (UI) elements, render distorted and / or perspective-corrected exterior views locked to areas of the vehicle (e.g., vehicle walls, etc.) so that the vehicle appears transparent or translucent and the vehicle event is visible, and render virtual content to replace distracting real-world content (e.g., billboards, accidents, etc.). In some cases, such virtual content may be rendered using deep neural networks trained to generate synthetic, or "deepfaked," virtual content from a database of images, and rendered in a style observed by augmented reality devices and / or vehicles.

[0054] The vehicle monitoring process can send vehicle events to the XR device (e.g., AR device) as described above. Similarly, the occupant monitoring process of the XR device can send occupant monitoring events (e.g., faults, etc.) to the vehicle. In some examples, the vehicle can have some autonomous or semi-autonomous driving capabilities that can benefit from knowledge of occupant monitoring events generated by the XR device. The vehicle can use such events as part of an autonomous driving policy or capability. For example, upon receiving an occupant monitoring event indicating an occupant (e.g., driver) impairment, the vehicle can activate an autonomous function to alert the occupant and / or prevent certain events or risks associated with impaired driving, and increase the level or confidence of an autonomous function already engaged. In some examples, upon receiving an occupant monitoring event indicating a health emergency, the vehicle can engage an autonomous function to safely stop the vehicle or proceed to a specific location (e.g., hospital, clinic, etc.) for assistance. In some cases, the vehicle can use online or reinforcement learning algorithms to learn over time which vehicle and / or world events have a higher correlation with certain driver states (e.g., certain obstacles, etc.) and use that information to handle and / or avoid such situations. For example, if occupants are often distracted by billboards, the vehicle may reduce speed or enable more aggressive emergency braking in environments with billboards.

[0055] In some cases, the XR device can use data from one or more wearable devices of the occupant to assist in the occupant monitoring process. The wearable device can have additional and / or redundant sensor modalities that can assist in occupant monitoring. The wearable device can transmit such data (or a digest of relevant events) to the XR device to assist in its occupant monitoring process. In some examples, the wearable device can transmit sensor data to the XR device, such as, for example, inertial sensor data, heart rate measurements, blood pressure measurements, galvanic skin response measurements, ECG / EKG / EEG data, temperature data, oxygen levels, movement information, sleep tracking information, etc. In some cases, data from the wearable device can indicate disorders such as drowsiness, intoxication, health emergency, stress, heightened emotional state, loss of consciousness, etc., and / or can be used to determine disorders (e.g., via machine learning classifiers trained on positive and negative examples).

[0056] In some cases, the XR device may obtain occupant monitoring information from the vehicle. For example, in some cases, the vehicle may incorporate its own occupant monitoring capabilities / functionality. In some cases, the XR device may transmit raw occupant monitoring data from the XR device's occupant monitoring process to the vehicle for fusion processing with the vehicle's occupant monitoring system (e.g., in addition to or instead of transmitting vehicle processed occupant monitoring events).

[0057] In some cases, the vehicle can send occupant monitoring events to the infrastructure and / or other vehicles to enable mitigation of driver events. For example, the vehicle can report to other vehicles that its driver is impaired so that the other vehicles can take preventative action, change autonomous driving policies (e.g., slow down, give ample distance to the impaired driver, etc.), notify the driver's XR device (e.g., in some cases, the virtual content can highlight or indicate that a nearby vehicle has an impaired driver), etc. In some cases, the driver's impairment can be reported to pedestrians crossing the infrastructure (e.g., so that crossing signals can prevent pedestrians from crossing in front of a vehicle with an impaired driver), to law enforcement (e.g., for assistance and / or protection), to the pedestrian's XR device (e.g., so that the pedestrian's XR device can render virtual content signaling the impaired driver, etc.). In some cases, the vehicle can render any of the virtual content described herein with respect to the XR device. For example, the vehicle can render the same content and / or types of content as described herein with respect to the XR and / or mobile devices. The vehicle may render such content in addition to, or instead of, any content rendered by the XR device.

[0058] Various aspects of the application are described with respect to the figures.

[0059] 1 is a diagram illustrating an example of a computing system 100 of a mobile device 150. The mobile device 150 is an example of a computing device that can be used by an end user. For example, the mobile device 150 can include a portable device, a mobile phone, an XR device (e.g., HMD, smart glasses, etc.), a tablet computer, a laptop computer, a wearable device (e.g., smart watch, etc.), a connected device or Internet of Things (IoT) device, and / or any other device used by a user to communicate over a wireless communication network. The computing system 100 includes software and hardware components that may be electrically or communicatively coupled (or otherwise in communication as appropriate) via a communication system 134, such as a bus.

[0060] The computing system 100 may include one or more sensor systems 102, computational components 110, one or more input devices 120 (e.g., a mouse, keyboard, touch screen, touchpad, keypad, microphone, controller, etc.), one or more output devices 122 (e.g., one or more displays, speakers, light emitting devices, printers, projectors, etc.), one or more modems 126, one or more wireless transceivers 128, one or more antennas 130, and / or one or more memory devices 132. In some cases, the computing system 100 may optionally include one or more SIMs 124. The computing system 100 may include a communication system 134 (e.g., a bus) that may transfer data between components of the computing system 100. In some examples, the one or more output devices 122 may include a left display and a right display of a mobile device 150, such as an HMD, smart glasses, etc. In some examples, the one or more output devices 122 may include a back display and / or a front display of a mobile device 150, such as a smartphone, smart watch, etc. In some cases, the output device(s) 122 may include one or more optical devices, such as one or more projectors.

[0061] In some examples, the communication system 134 can interconnect one or more sensor systems 102, the computational component 110, one or more input devices 120, one or more output devices 122, one or more modems 126, one or more wireless transceivers 128, one or more antennas 130, and / or one or more memory devices 132. For example, in some cases, the computational component 110 can use the communication system 134 to communicate between processing processors / cores and / or with any devices / components of the computing system 100, such as, for example, one or more sensor systems 102, one or more input devices 120, one or more output devices 122, one or more SIMs 124, one or more modems 126, one or more wireless transceivers 128, one or more antennas 130, and / or one or more memory devices 132.

[0062] The computational components 110 may include, for example, one or more central processing units (CPUs) 112, graphics processing units (GPUs) 114, digital signal processors (DSPs) 116, and / or image signal processors (ISPs) 118. In some cases, the computational components 110 may additionally or alternatively include one or more other processing components not shown in FIG. 1, such as, for example, but not limited to, one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), application processors (APs), vision processing units (VPUs), neural network signal processors (NSPs), microcontrollers, computer vision (CV) processors, special purpose hardware, any combination thereof, and / or other processing devices or systems. In some cases, the computational components 110 may include other electronic circuits or hardware, computer software, firmware, or any combination thereof for performing any of the various operations described herein. In some examples, the computational components 110 may include more or fewer computational components than those shown in FIG. 1. Furthermore, CPU 112, GPU 114, DSP 116, and ISP 118 are merely illustrative examples of computing components provided for illustrative purposes.

[0063] The computing system 100 may use one or more computing components 110 to perform various computing operations such as, for example, extended reality operations (e.g., tracking, localization, object detection, classification, pose estimation, mapping, content anchoring, content rendering, etc.), device control operations, image / video processing, graphics rendering, event mapping, machine learning, data processing, modeling, computation, computer vision, event monitoring, any operations described herein, and / or any other operations. For example, in some cases, the one or more computing components 110 may use data from the sensor system 102, one or more input devices 120, one or more SIMs 124, one or more modems 126, one or more wireless transceivers 128, one or more antennas 130, one or more memory devices 132, the vehicle computing system 210 shown in FIG. 2, and / or any other devices to perform image / video processing, event mapping, XR processing, device management / control, and / or other operations described herein.

[0064] To illustrate, in some examples, one or more of the computing components 110 may perform monitoring (e.g., device monitoring, user monitoring, vehicle monitoring, event monitoring, activity monitoring, object monitoring, etc.), device control / management, tracking (e.g., device tracking, object tracking, hand tracking, gaze tracking, etc.), localization, object detection and / or recognition, object classification, pose estimation, shape estimation, scene mapping, scene detection and / or recognition, face detection and / or recognition, emotion detection and / or recognition, content anchoring, content rendering, content filtering, image processing, modeling, content generation, gesture detection and / or recognition, user interface generation, power management, event detection and / or recognition, and / or other operations based on data from one or more of the components of the user computing system 100, the vehicle computing system 210, and / or any other systems or components.

[0065] In some examples, the one or more computational components 110 may implement one or more software engines and / or algorithms, such as, for example, feature extractors (e.g., Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Oriented FAST and Rotational BRIEF (ORB), etc.), machine learning models, computer vision algorithms, neural networks, tracking algorithms, localization algorithms, object detection algorithms, recognition algorithms, mapping algorithms, applications (e.g., XR applications, messaging applications, social media network applications, web browser applications, productivity applications, gaming applications, entertainment applications, multimedia applications, authentication applications, photo applications, scanning applications, media playback applications, security applications, e-commerce applications, content management applications, interface and / or windowing applications, assistant applications, automation applications, email applications, voice applications, camera applications, navigation applications, vehicle applications, etc.), computer vision algorithms, image processing algorithms, content filtering algorithms, and / or any other algorithms and / or components.

[0066] Image sensor 104A and image sensor 104N may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device such as a television or computer, a camera, etc. In some cases, image sensor 104A and image sensor 104N may be part of a camera or computing device, such as a digital camera, a video camera, an IP camera, a smartphone, a smart television, a gaming system, etc. Further, in some cases, image sensor 104A and image sensor 104N may include and / or implement a dual (or multiple) image sensor system or setup, such as a rear and forward sensor device. In some examples, image sensor 104A and image sensor 104N may be part of a dual camera or other multi-camera assembly (e.g., including two cameras, three cameras, four cameras, or other number of cameras). In some cases, image sensor 104A and / or image sensor 104N may include an RF sensor, such as a radar sensor, a LIDAR sensor, and / or an IR sensor configured to perform RF and / or IR imaging of the environment.

[0067] In some examples, each of the image sensors 104A and 104N can capture image data, generate frames based on the image data, and / or provide image data or frames to one or more computing components 110 for processing. A frame can include a video frame of a video sequence or a still image. A frame can include a pixel array representing a scene. For example, a frame can be a Red-Green-Blue (RGB) frame with red, green, and blue color components per pixel, a Luminance, Red Difference, Blue Difference (YCbCr) frame with one Luminance component and two Chrominance (Color) components (Red Chrominance and Blue Chrominance) per pixel, or any other suitable type of color or monochrome image.

[0068] The one or more wireless transceivers 128 can receive wireless signals (e.g., signals 140) via one or more antennas 130 from one or more other devices and / or networks, such as other user devices, vehicles (e.g., vehicle 202 shown in FIG. 2), network devices (e.g., base stations such as eNBs and / or gNBs, WiFi devices (e.g., routers, access points, etc.), servers, switches, routers, gateways, firewalls, etc.), cloud networks, private networks, public networks, data centers, the Internet, satellites, connected or IoT devices, infrastructure devices / components, etc. In some examples, the computing system 100 can include multiple antennas. The wireless signals 140 may be transmitted over a wireless network. The wireless network may be any wireless network, such as a cellular or telecommunications network (e.g., 3G, 4G, 5G, etc.), a wireless local area network (e.g., a WiFi network), a Bluetooth™ network, and / or any other wireless network. In some examples, the one or more wireless transceivers 128 may include a radio frequency (RF) front end. The RF front end may include one or more components such as one or more amplifiers, mixers (also called signal multipliers) for signal downconversion, frequency synthesizers (also called oscillators) that provide signals to the one or more mixers, baseband filters, analog-to-digital converters (ADCs), power amplifiers, RF ports (e.g., transmit (Tx) and / or receive (Rx) ports), phase shifters, IQ gain and phase compensators, upsamplers, gain control devices, digital-to-analog converters (DACs), beamformers, low-pass filters, time delay filters, etc., among other components. In some examples, the RF front end may generally handle the selection and conversion of the wireless signal 140 to a baseband or intermediate frequency and may convert the RF signal to the digital domain.

[0069] In some cases, computing system 100 may include a coding-decoding device (or codec) configured to encode and / or decode data transmitted and / or received using one or more wireless transceivers 128. In some cases, computing system 100 may include an encryption-decryption device or component configured to encrypt and / or decrypt data transmitted and / or received by one or more wireless transceivers 128 (e.g., according to the Advanced Encryption Standard (AES), the Data Encryption Standard (DES), and / or any other standard).

[0070] A SIM is a device (e.g., an integrated circuit) that can securely store a particular subscriber or user's International Mobile Subscriber Identity (IMSI) number and associated keys (e.g., encryption-decryption keys). The IMSI and keys may be used to identify and authenticate a subscriber with a particular UE. In FIG. 1, one or more SIMs 124 may each securely store an IMSI number and associated keys assigned to a user of a mobile device 150. The IMSI and keys may be used to identify and authenticate a subscriber when accessing a network provided by a network service provider or operator associated with one or more SIMs 124.

[0071] A modem is a device that modulates one or more carrier signals to encode digital information for transmission and demodulates the signals to decode the transmitted information. The one or more modems 126 can modulate one or more signals to encode information for transmission using one or more wireless transceivers 128. The one or more modems 126 can also demodulate signals received by the one or more wireless transceivers 128 to decode the transmitted information. In some examples, the one or more modems 126 can include 4G (or LTE) modems, 5G (or New Radio (NR)) modems, modems configured for vehicle-to-everything (V2X) communications, and / or other types of modems. The one or more modems 126 and the one or more wireless transceivers 128 can be used to communicate data for one or more SIMs 124.

[0072] As described above, the computing system 100 may include one or more sensor systems 102. In some examples, the one or more sensor systems 102 may include one or more image sensors, such as image sensors 104A and 104N (hereinafter collectively "image sensors 104"), a location sensor 106 (e.g., an ultrasonic sensor, an infra-red sensor, a SONAR sensor, an RF-based sensor system (e.g., WiFi, Bluetooth, etc.), a microphone, etc.), an inertial measurement unit (IMU) 108, and / or one or more other sensors. In some cases, the computing system 100 may optionally include one or more other / additional sensors or sensor systems, such as, for example, but not limited to, a RADAR sensor system, a LIDAR sensor system, an electromagnetic detection and ranging (EmDAR) sensor system, an infrared sensor system, a laser ranging system, a acoustic detection and ranging (SODAR) system, a touch sensor, a pressure sensor (e.g., an air pressure sensor and / or any other pressure sensor), a gyroscope, an accelerometer, a magnetometer, and / or any other sensor. In some examples, computing system 100 may include additional components, such as, for example, a light emitting diode (LED) device, a cache, a wireless network interface, etc. An example architecture and example hardware components that may be implemented by computing system 100 are further described below with respect to FIG.

[0073] Computing system 100 may also include (and / or be in communication with) one or more non-transitory machine-readable storage media or storage devices (e.g., one or more memory devices 132), which may include, for example, but are not limited to, local and / or network-accessible storage (which may be, e.g., programmable and / or flash-updateable, etc.), disk drives, drive arrays, optical storage devices, solid-state storage devices such as RAM and / or ROM, etc. Such storage devices may be configured to implement any suitable data storage system, including, but not limited to, various file systems, database structures, etc.

[0074] In some examples, the functions may be stored as one or more computer program products (e.g., instructions or code) in memory device(s) 132 and executed by computational components 110. Computing system 100 may also include software elements (e.g., located in one or more memory devices 132) including, for example, an operating system, device drivers, executable libraries, and / or other code, such as one or more application programs, which may comprise computer programs that perform the functions provided by various embodiments and / or may be designed to implement the methods described herein and / or configure the systems described herein.

[0075] In some implementations, a user equipment (UE) may be configured for Dual SIM Dual Active (DSDA) functionality. For example, the mobile device 150, the vehicle 202 shown in FIG. 2, and / or other UEs may be equipped with DSDA functionality. A DSDA-capable UE may be equipped with at least two SIMs. In one illustrative example, a DSDA-capable vehicle (e.g., the vehicle 202 shown in FIG. 2) and a user device (e.g., the mobile device 150) may enable the vehicle and vehicle occupants (e.g., driver, passengers, etc.) and user device to select independent network operator (or provider) contracts, with each operator contract being associated with a particular SIM. For example, the vehicle may use a first operator for wireless communication access and the user device may use a second operator for wireless communication access.

[0076] In some cases, the DSDA function may support at least two valid SIMs for a vehicle, including an OEM SIM and a user SIM, such as those described below with respect to the vehicle computing system 210 of FIG. 2. As described below, the OEM SIM and / or the user SIM may be used with one or more modems (e.g., the modem 228 shown in FIG. 2 and / or other modems of the communication system 222 shown in FIG. 2). In some implementations, the OEM SIM, the user SIM, and the vehicle's modem(s) may be part of the vehicle's telematics control unit (TCU) or may be part of a network access device (NAD) (sometimes also referred to as a network control unit or NCU) of the TCU (e.g., as part of the communication system 222 of FIG. 2). As described below, the OEM SIM may store information that provides access to perform wireless communications for vehicle-based operations (e.g., for eCall functionality, to communicate with the vehicle manufacturer for software updates, etc., among other operations). The OEM SIM supports various services for the vehicle, including eCall for making emergency calls. The user SIM may be used to implement wireless network access for the user's UE to support user data connectivity, such as supporting calling, messaging, infotainment-related services, among others.

[0077] DSDA can enable a user SIM and a vehicle modem to be used for wireless network access (e.g., for cellular connectivity) instead of the SIM and / or UE modem. For example, upon coming within communication range of the vehicle, a user device (e.g., a mobile device) can connect with the vehicle via an interface (e.g., via Bluetooth™, WiFi™, USB port, lightning port, and / or other wireless or wired interface). Once connected, the communication unit of the user device can transition wireless network access functionality from the user device to the communication unit of the vehicle. The communication unit of the vehicle can initiate an interaction with a base station to perform one or more wireless communication operations, such as facilitating a call, transmitting and / or receiving data (e.g., messaging, video, audio, etc.), among other operations. As noted above, a "communication unit" of a device (e.g., a vehicle, a user device, other UE, a roadside unit (RSU), etc.) may be a TCU, a NAD, a modem, a SIM, a transceiver (or individual receivers and / or transmitters), any combination thereof, and / or other system, device, or component configured to perform wireless communication operations. In one illustrative example, a user SIM (e.g., information stored in the SIM and / or the actual SIM card) of a user device (e.g., a mobile device) may be migrated to a vehicle's TCU NAD, and the vehicle's modem may then communicate with a wireless network operator on behalf of the user using the user SIM information.

[0078] 2 is a block diagram illustrating an example vehicle computing system 210 of a vehicle 202. The vehicle 202 is an example of a UE that can communicate with a network (e.g., eNBs, gNBs, positioning beacons, location measurement units, and / or other network entities) and / or with other UEs (e.g., mobile devices 150) and can communicate using V2X communications (e.g., over a PC5 interface or other device-to-device direct interface). As shown, the vehicle computing system 210 can include at least a power management system 212, a control system 214, an infotainment system 216, an intelligent transportation system (ITS) 218, one or more sensor systems 220, and a communication system 222. In some cases, vehicle computing system 210 may include or be implemented using any type of processing device or system, such as one or more CPUs, DSPs, GPUs, ISPs, ASICs, FPGAs, application processors (APs), vision processing units (VPUs), neural processing units (NPUs), controlled voltage processors (CVPs), microcontrollers, special purpose hardware, any combination of these, and / or other processing devices or systems.

[0079] The control system 214 may be configured to control one or more operations of the vehicle 202, the power management system 212, the computing system 210, the infotainment system 216, the ITS 218, and / or one or more other systems of the vehicle 202 (e.g., braking system, steering system, safety systems other than the ITS 218, cabin systems, and / or other systems). In some examples, the control system 214 may include one or more electronic control units (ECUs). The ECUs may control one or more of the electronic systems or subsystems in the vehicle. Examples of specific ECUs that may be included as part of the control system 214 include an engine control module (ECM), a powertrain control module (PCM), a transmission control module (TCM), a brake control module (BCM), a central control module (CCM), a central timing module (CTM), among others. In some cases, the control system 214 may receive sensor signals from one or more sensor systems 220 and may communicate with other systems of the vehicle computing system 210 to operate the vehicle 202.

[0080] The vehicle computing system 210 also includes a power management system 212. In some implementations, the power management system 212 can include a power management integrated circuit (PMIC), a standby battery, and / or other components. In some cases, other systems of the vehicle computing system 210 can include one or more PMICs, batteries, and / or other components. The power management system 212 can perform power management functions for the vehicle 202, such as managing power sources for the computing system 210 and / or other parts of the vehicle. For example, the power management system 212 can provide a stable power source taking into account power fluctuations, such as based on starting the vehicle's engine. In another example, the power management system 212 can perform thermal monitoring operations, such as by checking the ambient temperature and / or transistor junction temperature. In another example, the power management system 212 may perform some function based on detecting a certain temperature level, such as causing a cooling system (e.g., one or more fans, an air conditioning system, etc.) to cool certain components of the vehicle computing system 210 (e.g., a control system 214 such as one or more ECUs), shutting down certain functions of the vehicle computing system 210 (e.g., limiting the infotainment system 216 by turning off one or more displays, disconnecting from a wireless network, etc.), among other functions.

[0081] The vehicle computing system 210 may include a communication system 222. The communication system 222 may include software and hardware components for transmitting signals to and receiving signals from a network (e.g., a gNB or other network entity) and / or for transmitting signals from other UEs (e.g., to another vehicle or UE via a PC5 interface, a WiFi interface, a Bluetooth™ interface, and / or other wireless and / or wired interfaces). For example, the communication system 222 may be configured to wirelessly transmit and receive information via any suitable wireless network (e.g., a 3G network, a 4G network, a 5G network, a WiFi network, a Bluetooth™ network, and / or other networks). The communication system 222 includes various components or devices used to perform wireless communication functions, including an original equipment manufacturer (OEM) subscriber identity module (referred to as a SIM or SIM card) 224, a user SIM 226, and a modem 228. Although the vehicle computing system 210 is shown as having two SIMs and one modem, in some implementations, the computing system 210 can have any number of SIMs (e.g., one SIM or two or more SIMs) and any number of modems (e.g., one modem, two modems, or more than two modems).

[0082] As previously mentioned, a SIM is a device (e.g., an integrated circuit) that can securely store a particular subscriber or user's International Mobile Subscriber Identity (IMSI) number and associated keys (e.g., encryption-decryption keys). The IMSI and keys may be used to identify and authenticate a subscriber with a particular UE. The OEM SIM 224 may be used by the communication system 222 to establish wireless connections for vehicle-based operations, to perform emergency call (eCall) functions, to communicate with the vehicle manufacturer's communication system (e.g., for software updates, etc.), among other operations. The OEM SIM 224 may be important for communication systems where the OEM SIM supports emergency services, such as eCall for making emergency calls in the event of a vehicle accident or other emergency situation. For example, eCall may include a service that automatically calls an emergency number (e.g., "9-1-1" in the United States, "1-1-2" in Europe, etc.) in the event of a vehicle accident to communicate the location of the vehicle to emergency services, such as the police department, fire department, etc.

[0083] The user SIM 226 may be used by the communication system 222 to perform wireless network access functions to support user data connectivity (e.g., for calling, messaging, infotainment-related services, among others). In some cases, the user's user device may connect with the vehicle computing system 210 via an interface (e.g., PC5, Bluetooth™, WiFi™, Universal Serial Bus (USB) port, and / or other wireless or wired interfaces). Once connected, the user device may transition wireless network access functions from the user device to the vehicle's communication system 222, in which case the user device may cease performing wireless network access functions (e.g., while the communication system 222 performs wireless access functions). The communication system 222 may initiate interactions with base stations to perform one or more wireless communication operations, such as facilitating calls, transmitting and / or receiving data (e.g., messaging, video, audio, etc.), among other operations. In such cases, other components of the vehicle computing system 210 may be used to output data received by the communication system 222. For example, the infotainment system 216 (described below) may display video received by the communication system 222 on one or more displays and / or output audio received by the communication system 222 using one or more speakers.

[0084] The modem 228 (and / or one or more other modems of the communication system 222) may be used for communication of data for the OEM SIM 224 and / or the user SIM 226. In some examples, the modem 228 may include a 4G (or LTE) modem and another modem (not shown) of the communication system 222 may include a 5G (or NR) modem. In some examples, the communication system 222 may include one or more Bluetooth™ modems (e.g., for Bluetooth™ Low Energy (BLE) or other types of Bluetooth communications), one or more WiFi™ modems (e.g., for Dedicated Short Range Communications (DSRC) and / or other WiFi communications), a wideband modem (e.g., an Ultra Wideband (UWB) modem), any combination of these, and / or other types of modems.

[0085] In some cases, modem 228 (and / or one or more other modems of communication system 222) may be used to perform V2X communications (e.g., with vehicles in vehicle-to-vehicle (V2V) communications, with other devices in device-to-device (D2D) communications, with infrastructure systems in vehicle-to-infrastructure (V2I) communications, with pedestrian UEs in vehicle-to-pedestrian (V2P) communications, etc.). In some examples, communication system 222 may include a V2X modem used to perform V2X communications (e.g., sidelink communications over a PC5 interface), in which case the V2X modem may be separate from one or more modems used for wireless network access functions (e.g., network communications over a network or air interface (e.g., network communications over a Universal Mobile Telecommunications System (UMTS) interface or “Uu interface,” etc.) and / or sidelink communications other than V2X communications).

[0086] In some examples, the communication system 222 may be or may include a telematics control unit (TCU). In some implementations, the TCU may include a network access device (NAD) (sometimes also referred to as a network control unit or NCU). In some cases, the NAD may include a modem 228, any other modems not shown in FIG. 2, an OEM SIM 224, a user SIM 226, and / or other components used for wireless communications. In some examples, the communication system 222 may include a global navigation satellite system (GNSS). In some cases, the GNSS may be part of one or more sensor systems 220, as described below. The GNSS may provide the vehicle computing system 210 with the capability to perform one or more location services, navigation services, and / or other services that can utilize GNSS capabilities.

[0087] In some cases, communications system 222 may include one or more wireless interfaces (e.g., including one or more transceivers and one or more baseband processors for each wireless interface) for transmitting and receiving wireless communications, one or more wired interfaces (e.g., serial interfaces such as universal serial bus (USB) inputs, lightening connectors, and / or other wired interfaces) for performing communications over one or more wired connections, and / or other components that may enable vehicle 202 to communicate with a network and / or other UEs.

[0088] The vehicle computing system 210 may also include an infotainment system 216 that may control content, and one or more output devices of the vehicle 202 that may be used to output content. The infotainment system 216 may also be referred to as an in-vehicle infotainment (IVI) system or an in-car entertainment (ICE) system. The content may include navigation content, media content (e.g., video content, music or other audio content, and / or other media content), among other content. The one or more output devices may include one or more graphical user interfaces, one or more displays, one or more speakers, one or more extended reality devices (e.g., VR, AR, and / or MR headsets), one or more haptic feedback devices (e.g., one or more devices configured to vibrate a seat, a steering wheel, and / or other parts of the vehicle 202), and / or other output devices.

[0089] In some examples, the computing system 210 may include an intelligent transportation system (ITS) 218. In some examples, the ITS 218 may be used to implement V2X communications. For example, the ITS stack of the ITS 218 may generate V2X messages based on information from an application layer of the ITS. In some cases, the application layer may determine whether certain conditions are met to generate messages for use by the ITS 218 and / or to generate messages to be transmitted to other vehicles (e.g., in the case of V2V communications), pedestrian UEs (e.g., in the case of V2P communications), and / or infrastructure systems (e.g., in the case of V2I communications). In some cases, the communication system 222 and / or the ITS 218 may obtain car access network (CAN) information (e.g., from other components of the vehicle via a CAN bus). In some examples, the communication system 222 (e.g., a TCU NAD) may obtain CAN information via a CAN bus and transmit the CAN information to the ITS stack. The CAN information may include vehicle related information such as vehicle heading, vehicle speed, braking information, among other information. The CAN information may be provided to the ITS 218 continuously or periodically (e.g., every 1 millisecond (ms), every 10 ms, etc.).

[0090] The conditions used to determine whether to generate a message may be determined using the CAN information based on safety-related and / or other applications, including road safety, traffic efficiency, infotainment, business-related applications, and / or other applications. In one illustrative example, the ITS 218 may perform lane change assistance or coordination. For example, using the CAN information, the ITS 218 may determine that the vehicle 202 or a driver of the vehicle 202 is attempting to change lanes from a current lane to another lane, such as an adjacent lane (e.g., based on a turn signal being activated, a user turning or steering into the adjacent lane, etc.). Based on a determination that the vehicle 202 is attempting to change lanes, the ITS 218 may determine that a lane change condition is met that is associated with a message to be sent to other vehicles in the adjacent lane near the vehicle. The ITS 218 may cause the ITS stack to generate one or more messages for sending to other vehicles, which may be used to coordinate a lane change with the other vehicles. Other example applications include forward collision warning, automatic emergency braking, lane departure warning, pedestrian avoidance or protection (e.g., when a pedestrian is detected near the vehicle 202 based on V2P communication of a user's UE, etc.), and traffic sign recognition, among others.

[0091] ITS 218 may generate messages (e.g., V2X messages) using any suitable protocol. Examples of protocols that may be used by ITS 218 include one or more Society of Automotive Engineering (SAE) standards, such as SAE J2735, SAE J2945, SAE J3161, and / or other standards.

[0092] The security layer of the ITS 218 may be used to securely sign messages from the ITS stack that are sent to and verified by other UEs configured for V2X communications, such as other vehicles, pedestrian UEs, and / or infrastructure systems. The security layer may also verify messages received from such other UEs. In some implementations, the signing and verification process may be based on the security context of the vehicle 202. In some examples, the security context may include one or more encryption-decryption algorithms, public and / or private keys used to generate a signature using the encryption-decryption algorithms, and / or other information. For example, each ITS message generated by the ITS stack may be signed by the security layer. The signature may be derived using the public key and the encryption-decryption algorithm. A vehicle, pedestrian UE, and / or infrastructure system receiving the signed message may verify the signature to ensure that the message is from an authorized vehicle. In some examples, the one or more encryption-decryption algorithms may include one or more symmetric encryption algorithms (e.g., Advanced Encryption Standard (AES), Data Encryption Standard (DES), and / or other symmetric encryption algorithms), one or more asymmetric encryption algorithms using public and private keys (e.g., Rivest-Shamir-Adleman (RSA) and / or other asymmetric encryption algorithms), and / or other encryption-decryption algorithms.

[0093] In some examples, the ITS 218 can determine some actions (e.g., V2X-based actions) to perform based on a message received from another UE, such as the mobile device 150. The actions can include, for example, but not limited to, safety-related actions, navigation actions, driving actions, and / or other actions, such as actions for road safety, traffic efficiency, navigation, infotainment, business, driving actions, and / or other applications. In some examples, the actions can include causing the vehicle 202 (e.g., the control system 214) to perform an automatic function, such as automatic braking, automatic steering (e.g., to maintain heading in a particular lane), automatic lane change coordination with other vehicles, automatic acceleration and / or deceleration, among other automatic functions. In an illustrative example, a message can be received by the communication system 222 from another vehicle (e.g., via a PC5 interface) or another UE, such as a mobile device. The message can indicate that the other vehicle is coming to a stop. In response to receiving the message, the ITS 218 can generate a message or instruction and transmit the message or instruction to the control system 214. The message or command may cause the control system 214 to automatically brake the vehicle 202 to stop the vehicle 202 (or slow the vehicle 202) before it collides with another vehicle (e.g., before it collides with another vehicle).

[0094] In other example embodiments, the operations may include triggering the presentation / display of, among other things, a message alerting an occupant (e.g., the driver) of vehicle 202 of a vehicle-related event, such as another vehicle being in the lane next to vehicle 202, a message alerting an occupant (e.g., the driver) to stop vehicle 202, a message alerting an occupant (e.g., the driver) that a pedestrian is at an approaching crosswalk (e.g., a crosswalk within a threshold proximity of vehicle 202 and / or a crosswalk estimated to be approaching vehicle 202 within a particular time period, etc.), a message alerting an occupant (e.g., the driver) that a toll booth is within a particular distance of vehicle 202 (e.g., within one mile or any other distance in any unit of measurement).

[0095] The computing system 210 may include one or more sensor systems 220 (e.g., a first sensor system through an Nth sensor system, where N is a value equal to or greater than one). In some examples, the sensor systems 220 may include various types of sensor systems that may be disposed on or in various portions of the vehicle 202. In some examples, the sensor systems 220 may include one or more camera sensor systems, LIDAR sensor systems, RADAR sensor systems, EmDAR sensor systems, SONAR sensor systems, SODAR sensor systems, GNSS receiver systems (e.g., one or more GPS receiver systems), accelerometers, speed sensors, gyroscopes, magnetometers, pressure sensor systems, IMUs, infrared sensor systems, radio frequency (RF) sensor systems, laser ranging systems, ultrasonic sensor systems, infra-red sensor systems, microphones, weight sensors, any combination thereof, and / or other sensor systems. It should be understood that any number of sensors or sensor systems may be included as part of the computing system 210 of the vehicle 202.

[0096] Although the vehicle computing system 210 is shown as including several components and / or systems, one skilled in the art will appreciate that the vehicle computing system 210 can include more or fewer components than those shown in FIG. 2. For example, the vehicle computing system 210 can also include one or more input devices and one or more output devices (not shown). In some implementations, the vehicle computing system 210 can also include at least one processor (e.g., as part of or separate from the control system 214, the infotainment system 216, the communication system 222, and / or the sensor system 220) and at least one memory having computer-executable instructions executed by the at least one processor. The at least one processor is in communication with and / or electrically connected to (referred to as "coupled" or "communicatively coupled") the at least one memory. The at least one processor may include, for example, one or more microcontrollers, one or more CPUs, one or more FPGAs, one or more ASICs, one or more GPUs, one or more NPUs, one or more DSPs, one or more ISPs, one or more VPUs, one or more application processors (Aps) (e.g., for running or executing one or more software applications), and / or other processors. The at least one memory may include, for example, a read-only memory (ROM), a random access memory (RAM) (e.g., a static RAM (SRAM)), an electrically erasable programmable read-only memory (EEPROM), a flash memory, one or more buffers, one or more databases, and / or other memories. The computer-executable instructions stored at least in or on the memory may be executed to perform one or more of the functions or operations described herein.

[0097] 3 illustrates an example system process 300 for in-vehicle localization. In this example, a mobile device 150 is located inside a vehicle 202. The mobile device 150 may include an AR device worn by an occupant of the vehicle 202, such as a driver of the vehicle 202. The mobile device 150 may use the system process 300 to locate itself within the vehicle 202.

[0098] The mobile device 150 can use in-vehicle localization to understand its posture relative to the vehicle 202, monitor occupants of the vehicle 202 to detect any occupant impairments (e.g., distraction, intoxication, drowsiness, health emergencies / conditions, stress and / or heightened emotional states, inattention, etc.), monitor occupant activities / events, monitor / detect vehicle events / activities, mitigate vehicle events and / or control the operation of the vehicle 202, determine what (if any) content (e.g., virtual content, events, data, user interfaces, etc.) to render (and / or filter) to the occupants, etc. In some cases, the content rendered / filtered by the mobile device 150 can include (or be based on) data from the sensor system 102 on the vehicle 202 and / or the mobile device 150, sensor measurements, detected events, etc.

[0099] In some cases, the vehicle 202 can also render / filter content for the occupants. For example, the vehicle 202 can render content using a screen on / in the vehicle 202, a head-up display on / in the vehicle 202, a display on the dashboard of the vehicle 202, a projector device on / in the vehicle 202, and / or any other display device. In some examples, the content rendered by the vehicle 202 can include data from the vehicle 202, data from the sensor system 102, and / or data from one or more other devices, such as a wearable device, a computing system of another vehicle, etc. In some cases, the vehicle 202 can receive the orientation of the mobile device 150 relative to the coordinate system of the vehicle 202 and use the orientation information to determine which content to render and / or filter and / or where and / or when to render / filter such content.

[0100] As shown, a vehicle application 302 on the vehicle computing system 210 can send a vehicle template 304 to the mobile device 150. The mobile device 150 can use the vehicle template 304 to locate itself within the vehicle 202, as further described herein. The vehicle template 304 can specify and / or describe landmarks / markers 306 within the vehicle 202. The vehicle template 304 can also specify coordinates of the landmarks / markers 306 relative to the coordinate system of the vehicle 202. The coordinates of the landmarks / markers 306 can include a position and / or orientation of the landmarks / markers 306 relative to the coordinate system of the vehicle 202. The landmarks / markers 306 can include any visual landmarks / markers that can be imaged and detected using one or more image sensors (e.g., image sensor 104) on the mobile device 150. For example, the landmarks / markers 306 may include one or more visual patterns within the vehicle 202, active lighting within the vehicle 202, elements or objects (movable and / or immovable elements) within the vehicle 202, devices, portions within the vehicle 202, and / or any other visual landmarks / markers. In some examples, one or more of the image sensors may be configured to perform RF and / or IR imaging. In these examples, the landmarks / markers 306 may include objects and / or patterns within the vehicle 202 that may be sensed with RF and / or IR.

[0101] In some examples, the landmarks / markers 306 may include one or more patterns or codes (e.g., patterned designs such as a Quick Response (QR) code, barcode, checkerboard pattern, shapes, symbols, etc.) positioned and / or affixed to the interior (and / or objects within) of the vehicle 202, lights (e.g., light emitting diodes (LEDs), light bulbs, etc.) positioned and / or affixed to the interior (and / or objects within) of the vehicle 202, objects within the vehicle 202 (e.g., doors, windows, seats, headrests, components of the dashboard / panel and / or center console of the vehicle 202 (e.g., instrumentation, radio / media system (and / or components thereof)), steering wheel, horn, signaling system, shift lever, cup holders, controls, etc.), movable and / or immovable elements within the vehicle 202 (e.g., portions such as interior corners of the vehicle 202 (e.g., windshield, windows, dash, doors, side panels, rear window, etc.), car seats, steering wheels, etc.).

[0102] The mobile device 150 can use one or more image sensors 104 to image the landmarks / markers 306. For example, the mobile device 150 can use one or more image sensors 104 to capture one or more images showing the landmarks / markers 306 in the vehicle 202. The localization engine 312 of the mobile device 150 can use one or more images from the one or more image sensors 104 and the vehicle template 304 to locate itself within the vehicle 202. For example, the mobile device 150 can detect the landmarks / markers 306 shown in one or more images captured by the one or more image sensors 104. The localization engine 312 can use the vehicle template 304 to determine the coordinates (and orientation) of the landmarks / markers 306 relative to the coordinate system of the vehicle 202. The localization engine 312 can implement algorithms, such as transformations, to convert the coordinates of the landmarks 306 relative to the coordinate system of the vehicle 202 to the associated coordinates relative to the coordinate system of the mobile device 150. The localization engine 312 may use the coordinates of the landmarks / markers 306 relative to the vehicle 202 and the coordinates of the landmarks / markers 306 relative to the mobile device 150 to determine the attitude of the mobile device 150 within the vehicle 202 (e.g., relative to the coordinate system of the vehicle 202).

[0103] In some examples, the localization engine 312 can use the tracked location of the mobile device 150 relative to the coordinate system of the mobile device 150 (e.g., based on sensor data from one or more sensors on the mobile device 150) to determine the pose of the mobile device 150 relative to the locations and / or orientations of the landmarks / markers 306. The localization engine 312 can use the pose of the mobile device 150 relative to the locations and / or orientations of the landmarks / markers 306 to determine the pose of the mobile device 150 relative to the coordinate system of the vehicle 202.

[0104] In some cases, the localization engine 312 can use data from other sensors (e.g., location sensor 106, IMU 108, etc.) to assist in determining the attitude of the mobile device 150 within the vehicle 202. For example, the location sensor(s) 106 can use sensor data / signals (e.g., RF signals such as WiFi or Bluetooth, ultrasonic signals, etc.) to determine the attitude of the mobile device 150 relative to one or more objects within the vehicle 202. The location sensor(s) 106 can provide the determined attitude of the mobile device 150 to the localization engine 312. The localization engine 312 can use such attitude information as well as location and / or orientation information determined based on the landmarks / markers 306, as described above, to localize the mobile device 150 within the vehicle 202.

[0105] Based on the in-vehicle localization, the localization engine 312 can generate localization data 314 indicating the attitude of the mobile device 150 relative to the coordinate system of the vehicle 202. The localization engine 312 can provide the localization data 314 to the AR application 310 on the mobile device 150. In some examples, the AR application 310 can use the localization data 314 to render a user interface for an occupant of the vehicle 202 (e.g., for a user of the mobile device 150), determine what content (if any) to render for the occupant, determine what content (if any) to filter, monitor the occupant (e.g., for obstacles, activities / events, etc.), mitigate events, and / or make any other output and / or determination as described herein. In some cases, the localization engine 312 and / or the mobile device 150 can provide localization data 314 to the vehicle computing system 210, and the vehicle application 302 on the vehicle computing system 210 can use the localization data 314 to render a user interface and / or any other data described herein. In some cases, the vehicle application 302 can additionally or alternatively use event data, data regarding the occupant's status (e.g., data indicating an occupant's impairment), vehicle instrumentation data, and / or any other data to render a user interface and / or any other virtual content for the occupant.

[0106] In some cases, the mobile device 150 can use one or more images to determine the attitude of the mobile device 150 with or without any other data and / or modalities. For example, the mobile device 150 can use one or more images of the interior of the vehicle 202 to determine the attitude of the mobile device 150 relative to the coordinate system of the vehicle 202. The one or more images can depict one or more landmarks / markers in the interior of the vehicle 202. The mobile device 150 can use the one or more images to identify the location of the one or more landmarks / markers relative to itself, which the mobile device 150 can use to determine the attitude of the mobile device 150 within the vehicle 202 (e.g., relative to the one or more landmarks / markers and / or relative to the coordinate system of the vehicle 202).

[0107] FIG. 4A illustrates an example system process 400 of augmented reality for vehicle occupant assistance. In this example, the vehicle 202 may include landmarks 306 as described above. The vehicle computing system 210 may obtain sensor data 406 from a sensor system 220 on the vehicle 202. The sensor data 406 may be used to generate at least a portion of the vehicle data 408 provided to the mobile device 150 as further described herein. In some examples, the sensor data 406 may also be used by a vehicle monitoring engine 402 of the vehicle application 302 to monitor the vehicle 202 and / or the operation of the vehicle 202, and / or by an event mitigation engine 404 of the vehicle application 302 to perform event mitigation (e.g., assist the driver, correct errors and / or actions initiated / triggered by the driver, etc.). For example, the vehicle monitoring engine 402 may use the sensor data 406 to recognize and / or monitor situational events and / or other events (e.g., impending and / or occurring events) associated with the vehicle 202 and its surroundings. As another example, the event mitigation engine 404 can use the sensor data 406 to perform driver event mitigation actions such as, for example, but not limited to, preventing and / or implementing one or more vehicle functions, operations, and / or actions, generating warnings / alerts to the driver, correcting errors and / or actions initiated / triggered by the driver, implementing safeguards, and activating one or more autonomous driving capabilities. In some examples, the event mitigation engine 404 can perform driver event mitigation actions in response to receiving one or more driver monitoring events from the mobile device 150, as described herein.

[0108] The sensor data 406 may provide information regarding the status / condition of the vehicle 202, the conditions surrounding the vehicle, navigation information, driving / operational events, safety events, etc. For example, the sensor data 406 may include data indicative of the operation of the vehicle 202 (e.g., speed, heading / direction, acceleration / deceleration, etc.), the location of the vehicle 202, driving / operational statistics associated with the vehicle 202, timestamps associated with detected activities / events associated with the vehicle 202, any conditions (e.g., objects, lane markings, traffic signals, pedestrians, other vehicles, animals, road conditions, traffic, weather, infrastructure conditions, lighting, nearby obstacles, etc.) and / or activities / events external to the vehicle 202 (e.g., along the path of the vehicle 202, occurring outside of the vehicle 202, the surroundings of the vehicle 202, the environment external to the vehicle 202, the vicinity of the vehicle 202, etc.), navigation information, etc.

[0109] As another example, the sensor data 406 may (in addition or as an alternative) include any condition and / or state of the vehicle 202 (e.g., battery charge state, fuel level, warnings and / or errors, failures or malfunctions, etc.), any condition and / or state of one or more components of the vehicle 202 (e.g., tires, brakes, vehicle sensors, engine, door locks, radio and / or sound systems, control and / or signaling systems, vehicle computing system 210, blind spot information systems, driver monitoring systems, braking systems, autonomous driving components, parking sensors, driver assistance systems, navigation systems, automotive heads up displays, light sensors, vehicle lights, vehicle communications, etc.), systems (e.g., V2V, V2I, V2X, etc.), any functions / actions implemented or to be implemented by vehicle 202 (e.g., autopilot, traction control, cruise control, collision avoidance, lane departure, lane centering, stability control, brake assist, traffic alert, lane keeping, highway assist, parking, traffic sign recognition, blind spot monitoring, driver monitoring, intersection assist, lane change assist, intelligent speed adaptation, tire pressure monitoring, turning, acceleration / deceleration, signaling, etc.), any vehicle safety related events (e.g., an imminent collision, a motion error, a violation of one or more regulations (e.g., speed limits, seat belt regulations, lane change regulations, road safety regulations, etc.), a collision and / or impact event, etc.

[0110] In some cases, the sensor data 406 may include measurements associated with an occupant of the vehicle 202. For example, the sensor data 406 may include measurements from one or more weight sensors on one or more seats of the vehicle 202. The measurements from the one or more weight sensors may indicate that an occupant is in a particular seat of the vehicle 202. In some examples, the measurements of the one or more weight sensors may be used to help determine a location of the occupant and / or the mobile device 150, as described further herein. For example, the measurements may indicate that an occupant is in a particular seat of the vehicle 202. The mobile device 150 may ascertain the location / position of the occupant using the one or more sensors, as described further herein.

[0111] The vehicle 202 may include landmarks 306 as previously described. The vehicle computing system 210 may provide vehicle data 408 to the mobile device 150. The vehicle data 408 may include a vehicle template (e.g., the vehicle template 304). In some examples, the vehicle data 408 may also include any of the sensor data 406 and / or data generated based on the sensor data 406. In some cases, the data generated based on the sensor data 406 may include, for example, without limitation, a description of the information in the sensor data 406 (e.g., a description of sensor measurements and / or vehicle instrumentation), one or more determinations and / or predictions generated from the sensor data 406 (e.g., determined and / or predicted events occurring within the environment of the vehicle 202 and / or estimated to affect the operation of the vehicle 202 (and / or associated safety risks), etc.), one or more outputs and / or inferences generated from the sensor data 406, statistics and / or metrics related to the sensor data 406, events identified and / or described by the sensor data 406, situational information related to the vehicle 202 (e.g., the status / condition of the vehicle 202), one or more attributes of the sensor data 406, etc.

[0112] For example, in some cases, the vehicle data 408 may include a vehicle template and at least one of event data included in and / or generated from the sensor data 406, and / or vehicle status information included in and / or generated from the sensor data 406. In some cases, the event data may include indications of one or more objects and / or subjects (e.g., pedestrians, animals, infrastructure objects, buildings, devices, bicycles, motorbikes, traffic, gates, signs, etc.), obstacles (e.g., potholes, curbs, cones, trees, roadwork, accidents, road blocks, electrical cables, roadside trash or objects, debris, moisture, ice, snow, etc.), incidents (e.g., collisions, close or imminent encounters, maneuvers by another vehicle, roadside activity, emergency or emergency vehicles, etc.), another vehicle, etc., an event (e.g., an entry / approach event, an exit event, a stationary event, a moving event, a predicted event, etc.) in an environment surrounding and / or associated with the vehicle 202.

[0113] In some cases, the events may include one or more events based on the location / trajectory of the vehicle and / or the relevance to the operation / navigation of the vehicle 202. For example, the events may include events detected in / along the path of the vehicle 202 and / or events predicted to be in / along the path of the vehicle 202 within a threshold time period or a threshold distance traveled by the vehicle 202, events within a threshold proximity / distance to the vehicle 202 and / or the path of the vehicle 202, events estimated to trigger / prompt a maneuver by the vehicle 202 (e.g., accelerating, decelerating, turning, braking, lane centering, collision avoidance, stopping, signaling, etc.) and / or a change in the operation (e.g., policy) of the vehicle 202 (e.g., change route, modify or implement an autonomous driving function, modify one or more driving parameters and / or thresholds, etc.), and / or the like.

[0114] The vehicle template in the vehicle data 408 may include indications of the landmarks 306 in the vehicle 202 and the coordinates and / or orientations of the landmarks / markers 306 relative to the coordinate system of the vehicle 202 (e.g., relative to the coordinate system of the vehicle computing system 210 on the vehicle 202). For example, the vehicle template may include descriptions of the landmarks / markers in the vehicle 202 and the coordinates and / or orientations of the landmarks / markers relative to the coordinate system of the vehicle 202. The mobile device 150 may use the vehicle template in the vehicle data 408 (and any other portions of the vehicle data 408) to locate itself within the vehicle 202, as described above. In some examples, the mobile device 150 may also use the vehicle data 408 to perform any of the functions described herein, such as, for example, any of the AR, monitoring, rendering, control, communication, driver assistance, and / or mitigation functions described herein.

[0115] The localization engine 312 can generate localization data 314 that locates the mobile device 150 within the vehicle 202. The localization engine 312 can provide the localization data 314 to the AR application 310 for use by the monitoring engine 410, the content filtering engine 412, and / or the vehicle user interface 414, as further described herein. The localization data 314 can include an attitude of the mobile device 150 relative to a coordinate system of the vehicle 202. In some examples, the localization engine 312 can generate the localization data 314 based on the vehicle data 408 (e.g., based on a vehicle template in the vehicle data 408) and based on image data from one or more image sensors 104 of the mobile device 150.

[0116] For example, the localization engine 312 may obtain one or more images of the landmarks / markers 306 from one or more image sensors 104. The one or more images may depict one or more of the landmarks / markers 306. The localization engine 312 may detect the landmarks / markers 306 depicted in the one or more images. The localization engine 312 may use the vehicle template to determine coordinates and / or orientations of any detected landmarks / markers referenced / described in the vehicle template. Based on the coordinates and / or orientations in the vehicle template, the localization engine 312 may determine the position and / or orientation of the detected landmarks / markers relative to the coordinate system of the vehicle 202.

[0117] In some cases, the localization engine 312 can also determine the attitude of the mobile device 150 using the tracking data 422 from the tracking engine 420. In some examples, the tracking data 422 can include the attitude of the mobile device 150 in physical space, as described further below. In some cases, the localization engine 312 can determine the attitude of the mobile device 150 relative to the detected landmarks to determine the attitude of the mobile device 150 relative to the coordinate system of the vehicle 202. For example, the localization engine 312 can use one or more images showing the landmarks / markers 306 to determine the position and / or orientation of the landmarks / markers 306 relative to the coordinate system of the mobile device 150. The localization engine 312 can determine the attitude of the mobile device 150 relative to the landmarks / markers 306 using the position and / or orientation of the landmarks / markers 306 and the attitude of the mobile device 150 as indicated in the tracking data 422 from the tracking engine 420. The localization engine 312 can use the relative pose of the mobile device 150 and the landmarks / markers 306 to determine the pose of the mobile device 150 relative to the vehicle's 202 coordinate system.

[0118] In some examples, to determine the attitude of the mobile device 150 relative to the coordinate system of the vehicle 202, the localization engine 312 may convert (e.g., convert, translate, etc.) coordinates of the detected landmarks relative to the coordinate system of the vehicle 202 into corresponding coordinates relative to the coordinate system of the mobile device 150. For example, in some cases, the localization engine 312 may determine coordinates relative to the coordinate system of the mobile device 150 that correspond to the coordinates of the detected landmarks / markers relative to the coordinate system of the vehicle 202. The localization engine 312 may determine the attitude of the mobile device 150 relative to the coordinate system of the vehicle 202 using the location (e.g., coordinates) of the detected landmarks / markers relative to the coordinate system of the vehicle 202 and the location (e.g., coordinates) of the mobile device 150 relative to the coordinate system of the mobile device 150 relative to the location (e.g., coordinates) of the detected landmarks / markers relative to the coordinate system of the mobile device 150.

[0119] In some cases, the localization engine 312 may also use data from the location sensor 106 to assist in generating the localization data 314, as described above. For example, in some cases, the location sensor 106 may determine the attitude of the mobile device 150 and / or one or more of the landmarks / markers 306 using one or more localization signals / algorithms, such as, for example, RF-based localization / positioning, ultrasound-based localization / positioning, etc.

[0120] In some cases, the sensor data 406 may include measurements from one or more weight sensors on one or more seats of the vehicle 202. Measurements from the one or more weight sensors may indicate that an occupant holding or wearing the mobile device 150 is in a particular seat of the vehicle 202. This information may be used to confirm the location / position of the mobile device 150 as determined by the mobile device 150, as previously described.

[0121] The tracking engine 420 can obtain sensor data and use the sensor data to perform tracking operations. The tracking operations can track the pose (e.g., location, orientation, etc.) of the mobile device 150, an occupant of the vehicle 202 (and / or one or more body parts of the occupant, e.g., hands, eyes, fingers, head pose, etc.), etc. The sensor data can include image data from one or more image sensors 104, position information (e.g., angular velocity, linear acceleration, orientation, and / or changes in pitch, roll, and yaw) from the IMU 108, and / or data from one or more other sensors, such as the location sensor 106 of the mobile device 150 and / or the vehicle 202. In some examples, the tracking engine 420 can use data from one or more image sensors 104, the IMU 108, and / or the location sensor 106 to perform position tracking (e.g., six degrees of freedom (6DOF) position tracking, etc.) of the mobile device 150 to determine the pose of the mobile device 150. The tracking engine 420 may generate tracking data 422 and provide the tracking data 422 to the AR application 310 for use by the monitoring engine 410, the content filtering engine 412, and / or the vehicle user interface 414. In some examples, the tracking data 422 may include an attitude of the mobile device 150 relative to a coordinate system of the mobile device 150.

[0122] The tracking engine 420 may track one or more body parts (e.g., eyes, hands, fingers, head, etc.) of the mobile device 150 and / or an occupant of the vehicle 202 using one or more tracking algorithms such as a Kalman filter, a hand tracking algorithm, a machine learning algorithm, a ray tracing algorithm, a gaze and / or eye tracking algorithm, a computer vision algorithm, a position tracking algorithm, etc. The occupant may include a driver or a passenger of the vehicle 202. The tracking engine 420 may provide tracking data 422 to the AR application 310. The tracking data 422 may include position and / or orientation information as further described herein.

[0123] In some cases, the tracking engine 420 can perform multiple tracking operations. For example, the tracking engine 420 can track the mobile device 150 and one or more body parts (e.g., hands, eyes, head, fingers, posture, etc.) of the occupant of the vehicle 202. In some examples, the tracking engine 420 can track the hand(s) of the occupant of the vehicle 202. In some examples, the tracking engine 420 can track the eyes and / or gaze of the occupant of the vehicle 202.

[0124] 4B, the tracking engine 420 may include a device tracking engine 430, a hand tracking engine 432, and / or an eye tracking engine 434. The device tracking engine 430 may track a position and / or orientation of the mobile device 150, as previously described. The device tracking engine 430 may generate device tracking data 440 for the AR application 310. In some cases, the device tracking data 440 may include or represent at least a portion of the tracking data 422 shown in FIG. 4A.

[0125] The device tracking engine 430 may use one or more images from one or more image sensors 104 and / or inertial sensor data from the IMU 108 to track the attitude of the mobile device 150. In some cases, the device tracking engine 430 may (additionally or alternatively) use data from one or more additional sensors, such as the location sensor 106, to track the attitude of the mobile device 150. For example, the location sensor 106 may acquire one or more RF signals and generate attitude information associated with the mobile device 150 based on a round trip time (RTT) associated with the one or more RF signals, a time of arrival (TOA) associated with the one or more RF signals, a received signal strength indicator (RSSI) associated with the one or more RF signals, etc. The device tracking engine 430 may use the attitude information from the location sensor 106 to track the attitude of the mobile device 150.

[0126] The hand tracking engine 432 can track one or more hands of an occupant associated with the mobile device 150. The hand tracking engine 432 can generate hand tracking data 442 for the AR application 310. In some cases, the hand tracking data 442 can include or represent at least a portion of the tracking data 422 shown in FIG. 4A. The hand tracking engine 432 can track one or more hands of the occupant using image data from one or more image sensors 104. The image data can include one or more images of the occupant's one or more hands. The hand tracking engine 432 can detect one or more hands in the one or more images and determine a posture of the one or more hands in the physical space based on the one or more images showing the one or more hands. In some cases, the hand tracking engine 432 can track a posture of the one or more hands using data from one or more additional sensors, such as (additionally or alternatively) the location sensor 106. For example, the location sensor 106 may obtain one or more RF signals and generate pose information associated with one or more hands based on an RTT associated with the one or more RF signals, a TOA associated with the one or more RF signals, an RSSI associated with the one or more RF signals, etc. The hand tracking engine 432 may use the pose information from the location sensor 106 to track the one or more hands.

[0127] The hand tracking engine 432 may track one or more hands using one or more tracking algorithms, such as a hand tracking algorithm, a machine learning algorithm, a ray tracing algorithm, a computer vision algorithm, etc. The hand tracking engine 432 may provide hand tracking data 442 to the AR application 310. The hand tracking data 442 may include a pose of one or more hands in physical space.

[0128] The eye tracking engine 434 can track the eyes and / or gaze of the occupant using image data from one or more image sensors 104. The eye tracking engine 434 can generate eye tracking data 444 for the AR application 310. The eye tracking data 444 can include the gaze of the occupant determined at one or more times or time periods. In some cases, the eye tracking data 444 can include or represent at least a portion of the tracking data 422 shown in FIG. 4A.

[0129] The image data may include one or more images depicting the eyes of the occupant. The eye tracking engine 434 may detect the eyes in the one or more images and determine the occupant's gaze based on the detected eyes in the one or more images. In some examples, the eye tracking engine 434 may detect the occupant's gaze using one or more images and one or more algorithms, such as ray tracing algorithms, machine learning algorithms, computer vision algorithms, etc. The eye tracking engine 434 may provide the eye tracking data 444 to the AR application 310, as described above.

[0130] The AR application 310 can use the location data (e.g., location data 314) and tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444) to perform various functions, such as, for example, occupant monitoring (e.g., via the monitoring engine 410), virtual content filtering (e.g., via the content filtering engine 412), content rendering (e.g., via the vehicle user interface 414), combinations thereof, and / or other functions as described herein. For example, in some cases, the AR application 310 can implement the monitoring engine 410 configured to monitor events associated with the vehicle 202 and / or occupants associated with the mobile device 150.

[0131] The AR application 310 may additionally or alternatively implement a content filtering engine 412 configured to use location data (e.g., location data 314), tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), occupant data from the monitoring engine 402, vehicle state data and / or event data from the vehicle data 408 and / or monitoring engine 410 on the vehicle application 302, sensor data, and / or any other data to filter / block virtual content from being rendered for the occupants and / or switch from virtual content rendering to live content, such as a live camera feed.

[0132] The AR application 310 may additionally or alternatively implement a vehicle user interface 414 configured to render data, such as virtual content, live content (e.g., camera feeds, etc.), and / or user interface elements. The vehicle user interface 414 may render data based on localization data (e.g., localization data 314), tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), occupant data from the monitoring engine 410, vehicle data 408 on the vehicle application 302 and / or vehicle and / or event data from the monitoring engine 402, sensor data, and / or any other data.

[0133] The monitoring engine 410 can monitor an occupant associated with the mobile device 150 and detect the occupant's state. In some examples, the occupant's state can include the occupant's gaze, the occupant's posture, the occupant's activity, any impairment of the occupant, and / or any other information regarding the occupant. The occupant's impairment can include any event, activity, distraction, state, attribute, behavior, and / or condition (e.g., cognitive, emotional or psychological, physiological, visual, audio, and / or situational event, condition, attribute, activity, behavior, distraction, and / or state) that can impact / influence the occupant's ability to safely operate (e.g., drive, control, manage, etc.) the vehicle 202. For example, impairments may include those that may negatively impact / influence the ability of an occupant (and / or associated response / reaction times) to operate (e.g., drive, control, manage, etc.) vehicle 202, detect / recognize any events / conditions encountered by vehicle 202 during operation of vehicle 202, avoid and / or react to events / conditions encountered by vehicle 202 during operation of vehicle 202, maintain a threshold amount of attention / concentration to operating vehicle 202 (and / or any events / conditions related to vehicle 202, operation of vehicle 202, environment / surroundings of vehicle 202, etc.) (and / or maintain attention / concentration for a threshold period of time), etc.

[0134] In some examples, the impairment may be a distraction (e.g., from the operation of the vehicle 202 and / or associated vehicles and / or associated events), a state of drowsiness, a state of intoxication, a health emergency (e.g., stroke, heart attack, seizure, catatonia, loss of consciousness, etc.), a state of emotional or psychological stress, a heightened emotional state, loss of consciousness, incapacitation, a physiological condition, an occupant context (e.g., occupant position / posture, occupant behavior, occupant movement, occupant activity, occupant interaction with the vehicle 202 and / or the occupant's environment, inhibiting / restricting an occupant's freedom of movement or flexibility and / or inhibiting an occupant's reaction time). These may include occupant clothing such as clothing that shortens the distance between the occupant and the vehicle, failure to wear medical or safety devices such as prescription glasses or a seat belt, items that reduce the occupant's visibility such as sunglasses in poor visibility conditions or clothing that may block or partially block the occupant's visibility, etc., that may increase safety risks and / or reduce the occupant's ability to control the vehicle 202 and / or drive and / or respond to vehicle-related events, the occupant's cognitive state, high noise levels within the vehicle 202 (e.g., that may limit the occupant's ability to concentrate and / or hear relevant sounds), etc.

[0135] In some cases, the obstacles may include an occupant's distracted state with respect to the operation of the vehicle 202 and / or events associated with the vehicle 202, an occupant's intoxicated state, an occupant's health state, an occupant's alertness state, an occupant's detected emotional state, an impaired position (of the occupant) for controlling the vehicle 202 (e.g., crouching while driving, moving away from one or more controls of the vehicle 202, one or more hands occupied by one or more objects other than the driving controls for controlling the operation / behavior of the vehicle 202, etc.), impaired visibility (e.g., obstructions to road and / or environmental visibility / visibility, impaired or reduced light and / or visibility conditions, etc.), and / or any other obstacle.

[0136] In some examples, the monitoring engine 410 can use the location data (e.g., location data 314), tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), and / or any other event, status, and / or sensor data, such as vehicle data 408 (or portions thereof), to monitor occupants (e.g., driver, passenger, etc.) and detect any impairments of the occupants. For example, the monitoring engine 410 may use the vehicle 202's context (which may relate to, characterize, and / or affect the occupant's ability to safely operate the vehicle, e.g., a state / condition, an action, a vehicle event, etc.) (e.g., determined from the vehicle data 408 and / or the sensor data 406), location data (e.g., location data (e.g., location data 314), tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), and / or one or more image sensors 104, IMUs 108, location information (e.g., location information 314), ... Any impairment of the occupant may be determined based on the occupant's posture (determined from motion sensor 106, and / or any other sensors and / or data), any virtual content rendered by mobile device 150 for the occupant, and / or any detected user interactions (e.g., occupant interactions) with the rendered virtual content (e.g., inputs, gaze and / or focus, gestures, etc.), eye tracking data, inertial sensor data, driving statistics, other sensor data, any combination thereof, and / or any other data.

[0137] For example, the context of the vehicle 202 may indicate that a particular event (e.g., affecting the safe operation of the vehicle) has occurred or is expected to occur. The monitoring engine 410 may determine that, in order to control the vehicle 202 in response to the particular event and / or to avoid safety issues and / or risks associated with the particular event, the occupant should focus and / or pay attention to the event (and / or a particular area / direction associated with the event), should not be distracted by virtual content (or other content), should not move in a particular manner, should not engage with the particular virtual content, should pay attention to the operation of the vehicle 202, should not engage in a particular activity not related to the particular operation of the vehicle 202, should be oriented in a particular direction, should be positioned in a particular manner to enable the occupant to respond to the particular event (and / or operation of the vehicle 202), etc. Alternatively, the monitoring engine 410 may determine that the occupant is in a impaired state based on the state / status of the vehicle 202 and / or the occupant's location, attention / focus, gaze, orientation, movement, content engagement, activity, etc.

[0138] In some cases, the context of the vehicle 202 may include the state / status of the vehicle 202, vehicle events, and the like. In some examples, an event associated with vehicle 202 (e.g., a vehicle event) may include an obstacle in the path of vehicle 202, an obstacle within a threshold proximity to the path of vehicle 202, a different vehicle (e.g., a car, bus, motorcycle, off-road vehicle, bicycle, train, truck, etc.) in the path of vehicle 202, a different vehicle within a threshold proximity to the path of vehicle 202, a traffic regulation (e.g., a traffic sign, a guide sign, a variable message sign, a traffic cone, an arrow board, a construction barrel, a barricade, a traffic marker, a temporary raised safety zone, a traffic signal, etc.) associated with the path of vehicle 202 (e.g., within / along the path of vehicle 202, in close proximity to the path of vehicle 202, etc.), failure of vehicle 202 to stay within a speed limit, failure of vehicle 202 to stay within a lane or lane marking, failure of vehicle 202 to stay within the road, a traffic event (e.g., an accident, a traffic reroute, a stop, a traffic increase, a road closure, etc.), and the like. In some examples, the obstacles may include pedestrians, animals, objects, another vehicle, road conditions, trees, obstructions, and the like.

[0139] In some cases, the state of the vehicle 202 may include operations and / or conditions of the vehicle 202 such as, for example, but not limited to, acceleration, deceleration, signaling (e.g., turn signals, etc.), selected / active gear, warnings (e.g., engine warning, battery warning, fuel warning, tire warning, light warning, etc.), maneuvers (e.g., turns, exits / exits, entrances / entrances, U-turns, stopping / braking, etc.), vehicle and / or vehicle component conditions / conditions (e.g., conditions / conditions of the engine, battery, one or more vehicle lights (e.g., conditions / conditions of headlights, tail lights, daytime running lights, reversing lights, emergency lights, fog lights, off-road lights, signal lights, brake lights, etc.), tires (e.g., tire pressure, tire warning, flat condition, etc.), one or more brakes or brake system conditions / conditions, fuel status and / or fuel readings, etc.), failures or specific operations of one or more vehicle components / functions, driving states, autonomous states, vehicle function states, etc.

[0140] In some cases, the monitoring engine 410 may determine an occupant's condition (e.g., an occupant's impairment) based on various cues (e.g., visual and / or audio cues) and / or information about the occupant, such as the occupant's condition, characteristics, and / or status, any occupant activities / movements, occupant behavior, occupant gaze, etc. For example, the monitoring engine 410 may determine an occupant's gaze or eye movement patterns (e.g., irregular eye movements, decreased responsiveness to stimuli, prolonged eye gaze, irregular eye gaze, focusing on a particular location or direction, etc.), eye characteristics (e.g., redness, pupil size, glazed eyes, etc.), reaction times and / or characteristics by the occupant to one or more events (e.g., exaggerated reactions, slowed reaction times, etc.), facial expressions of the occupant, occupant's head posture and / or head movements (e.g., jerking, head turning, tilting or sagging, etc.), and / or other characteristics of the occupant's head. A fault may be determined based on the following: head tilt, ...

[0141] In some cases, the monitoring engine 410 may determine an occupant's impairment based on audio cues (in addition to or instead of visual cues). For example, in some cases, the monitoring engine 410 may obtain audio data from one or more sensors, such as microphones, of the mobile device 150 and / or the vehicle 202, and determine an impairment based on the audio data. To illustrate, the monitoring engine 410 may determine an impairment based on one or more speech characteristics (e.g., slurred speech, fast speech, etc.), the occupant's voice (e.g., loudness or softness, etc.), the content of the occupant's recognized speech, etc.

[0142] In some cases, the monitoring engine 410 may determine a fault based on the state / status of the vehicle 202 and / or the content rendered by the mobile device 150. For example, if the state of the vehicle 202 indicates that the vehicle is moving at a high speed, moving in a poor visibility environment, moving in difficult conditions (e.g., wet or icy conditions, severe weather conditions, sharp turns, heavily passed areas, high volume of pedestrians and / or other traffic, etc.), performing maneuvers, etc., the monitoring engine 410 may determine that the vehicle 202 is moving at a high speed, moving in a poor visibility environment, moving in difficult conditions (e.g., wet or icy conditions, severe weather conditions, sharp turns, heavily passed areas, high volume of pedestrians and / or other traffic, etc.), performing maneuvers, etc., if the monitoring engine 410 determines that the occupants are concentrating on the virtual content rendered by the mobile device 150 for a threshold period of time or are concentrating away from a particular area / location (e.g., a road, a location of a vehicular event, etc.) for a threshold period of time (e.g., the state of the vehicle 202). An impairment of an occupant may be determined if the occupant is determined to be engaged in other activities (whether operational or safe operation (e.g., driving)), if the occupant is determined to be located away from certain vehicle controls, if the occupant is determined to have a certain posture / position determined to reduce driving safety or increase driving risk (e.g., leaning forward, bending, turning, etc.), if the occupant is not wearing a seat belt, if the occupant is operating the vehicle 202 without vehicle lights (e.g., headlights, fog lights, tail lights, brake lights, etc.) in poor visibility conditions (e.g., at night, during inclement weather conditions, during heavy fog, etc.), etc. The monitoring engine 410 may determine an impairment based on the state / status of the vehicle 202 and / or content rendered by the mobile device 150, in addition to or without any cues (visual and / or audio) regarding the occupant as described above.

[0143] In some cases, the vehicle monitoring engine 402 of the vehicle application 302 may determine any faults of the occupants as well. The vehicle monitoring engine 402 may determine any faults in addition to or in lieu of any fault information determined by the monitoring engine 410 of the AR application 310. For example, in some cases, the vehicle monitoring engine 402 may determine fault information in combination with fault information determined by the monitoring engine 410 of the AR application 310. As another example, in some cases, the vehicle monitoring engine 402 may determine fault information separately and / or without any fault information determined by the monitoring engine 410 of the AR application 310. Furthermore, the vehicle monitoring engine 402 may determine any faults in addition to any vehicle data, such as data indicative of the context of the vehicle 202, as described further below.

[0144] In some examples, the content filtering engine 412 can use location data (e.g., location data 314), tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), fault information from the monitoring engine 410 (and / or vehicle monitoring engine 402), and / or any other event, condition, and / or sensor data, such as vehicle data 408 (or portions thereof), to determine / select content to filter / block and / or render for the occupant. For example, the content filtering engine 412 can use the pose of the mobile device 150 relative to the coordinate system of the vehicle 202 (e.g., of the vehicle computing system 210) and vehicle data indicative of the context of the vehicle 202 (e.g., state / status of the vehicle 202, vehicle events / operations, etc.) to filter / block certain content that may distract the occupant from the operation and / or vehicle events of the vehicle 202, obstruct the occupant's view of the vehicle events and / or certain areas outside the vehicle 202 that the content filtering engine 412 determines should remain visible to the occupant. In some cases, instead of or in addition to filtering / blocking content, the content filtering engine 412 can replace virtual content rendered by the mobile device 150 with live content (e.g., a live camera feed) that may not obstruct the occupant's view of the vehicle events or the environment outside the vehicle 202. As another example, the content filtering engine 412 can use information about the attitude of the mobile device 150 and the state of the vehicle 202 to render virtual content that draws the occupants' attention in a particular direction and / or to a particular location, such as the direction / location of a particular event.

[0145] In some cases, the content filtering engine 412 can use the orientation of the mobile device 150 relative to the coordinate system of the vehicle 202 to correlate information from and / or associated with the vehicle 202 with the orientation of the mobile device 150. For example, the state / status of the vehicle 202 may indicate a location of a vehicle event reported by the vehicle computing system 210. The location of the vehicle event may be relative to the coordinate system of the vehicle. Thus, the orientation of the mobile device 150 can be correlated to the location of the vehicle event since both are known relative to the coordinate system of the vehicle. The content filtering engine 412 can use the orientation of the mobile device 150 relative to the coordinate system of the vehicle 202 to determine whether any virtual content rendered for the occupant obstructs the occupant's view of the event reported by the vehicle 202 relative to the coordinate system of the vehicle 202. If the content filtering engine 412 determines that content is occluded at a particular location relative to the coordinate system of the vehicle 202, the content filtering engine 412 can use the pose of the mobile device 150 relative to the coordinate system of the vehicle 202 to filter / block any content from being rendered at the particular location.

[0146] In some examples, the content filtering engine 412 can filter any virtual content determined to be distracting or distracting to the occupant. For example, the content filtering engine 412 can determine that video game content (or any other content) may be distracting to the occupant and filter any video game content to prevent such content from being rendered to the occupant while operating the vehicle 202. In other examples, the content filtering engine 412 can filter any virtual content determined to block / obstruct the occupant's view / visibility to a particular event, such as an obstacle along (or within the vicinity of) the path of the vehicle 202, a traffic event, or any vehicle event described herein. For example, the content filtering engine 412 can determine that the virtual content blocks the occupant's view of a vehicle event identified in the vehicle data 408 based on the attitude of the mobile device 150 relative to the coordinate system of the vehicle 202 and the location of the vehicle event relative to the coordinate system of the vehicle 202 (e.g., as may be indicated in the vehicle data 408). The content filtering engine 412 can then filter / block such virtual content to prevent it from obscuring the occupant's view of the vehicle event and / or can switch from rendering the virtual content to providing a live feed. In some examples, the live feed may depict the vehicle event or may not obscure the vehicle event.

[0147] The vehicle user interface 414 can render content and / or user interface elements for the occupant. In some examples, the vehicle user interface 414 can render or generate content and / or user interface elements based on the state / condition of the vehicle 202. In some examples, the vehicle user interface 414 can use the obstruction information to determine whether to render content and / or what content to render for the occupant. For example, if the obstruction information from the monitoring engine 410 indicates that the occupant is distracted, the vehicle user interface 414 can use such information to stop rendering content that may (or continues to) distract the occupant or render content configured to attract the occupant's attention to a particular location and / or action / behavior (and away from a different location and / or action / behavior that distracts the occupant).

[0148] The vehicle user interface 414 can use the orientation of the mobile device 150 relative to the coordinate system of the vehicle 202 to determine where (or whether) to render content for the occupant. For example, to render content at a particular location to draw the occupant's attention to something relative to the coordinate system of the vehicle 202, the vehicle user interface 414 can use the orientation of the mobile device 150 relative to the coordinate system of the vehicle 202 to render the content at that particular location. As another example, if the vehicle computing system 210 reports some data associated with a particular location relative to the coordinate system of the vehicle 202, the vehicle user interface 414 can use the orientation of the mobile device 150 relative to the coordinate system of the vehicle to render the data at that particular location relative to the coordinate system of the vehicle 202, or at a different location determined for that particular location relative to the coordinate system of the vehicle 202.

[0149] In some examples, the vehicle user interface 414 can use the pose of the mobile device 150 relative to the coordinate system of the vehicle 202 to determine where to render content for the occupant. For example, the vehicle user interface 414 can render virtual content (e.g., arrows, bounding boxes, images, pulsing lights, etc.) at a particular location associated with the vehicle event to draw the occupant's attention to the vehicle event. As another example, the vehicle user interface 414 can render virtual content at a location different from the location of the vehicle event to prevent the virtual content from obstructing the occupant's view of the vehicle event. In some examples, the vehicle user interface 414 can select particular content to render for the occupant based on the occupant's state and / or the context of the vehicle 202. For example, to avoid obstructing the occupant's view of a particular event, the vehicle user interface 414 may render live content (e.g., a live camera feed) instead of virtual content. As another example, to avoid distracting the occupant, the vehicle user interface 414 can render virtual content instead of live content. To illustrate, if the context of the vehicle 202 indicates that another vehicle has broken down on the side of the road or has been pulled over by police, the vehicle user interface 414 may render virtual content (or no content) rather than live content to prevent occupants from overlooking. Alternatively, the content filtering engine 412 may filter the live content to prevent overlooking, and the vehicle user interface 414 may instead render virtual content or no content at all. In some cases, the live content may include a live feed from one or more image sensors 104 on the mobile device 150 and / or one or more image sensors of the vehicle 202.

[0150] In some cases, the AR application 310 can generate event data 424. The AR application 310 can also transmit the event data 424 to the vehicle computing system 210. In some cases, the AR application 310 can generate the event data 424 based on localization data (e.g., localization data 314), tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), any output and / or data from the monitoring engine 410, the content filtering engine 412, and / or the vehicle user interface 414, data from any sensor (e.g., image sensor 104, location sensor 106, IMU 108, etc.) of the mobile device 150, application data, user input, and / or any other data. In some cases, the event data 424 can include localization data (e.g., localization data 314). The event data 424 may be generated based on tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), any output and / or data from the monitoring engine 410, the content filtering engine 412, and / or the vehicle user interface 414, data from any sensors of the mobile device 150 (e.g., the image sensor 104, the location sensor 106, the IMU 108, etc.), application data, user input, and / or any other data. In some cases, the event data 424 may additionally or alternatively include a description of the event generated from the location data (e.g., the location data 314) and / or the location data.The event data 424 may be generated based on tracking data (e.g., tracking data 422, device tracking data 440, hand tracking data 442, and / or eye tracking data 444), any output and / or data from the monitoring engine 410, the content filtering engine 412, and / or the vehicle user interface 414, data from any sensors of the mobile device 150 (e.g., the image sensor 104, the location sensor 106, the IMU 108, etc.), application data, user input, and / or any other data.

[0151] In some examples, the event data 424 may include, indicate, and / or describe the occupant status determined by the monitoring engine 410 as described above, the content from the AR application 310, any combination thereof, and / or any other occupant and / or device information. In some examples, the occupant status may include an occupant impairment determined by the monitoring engine 410. In some cases, the occupant state may additionally or alternatively include the occupant's posture (e.g., location, orientation, posture, etc.), any movement of the occupant, any activity by the occupant, the occupant's gaze, the occupant's head posture, any information regarding the occupant, and / or any other information indicative of the occupant's attention / focus, occupant activity, occupant movement, occupant visibility or field of view (FOV), occupant behavior, occupant position (e.g., location, orientation, posture, etc.), occupant reach (e.g., reach to one or more vehicle controls), occupant engagement, and / or the ability (and / or likelihood) of the occupant to control the operation of the vehicle 202 and / or respond to events associated with the vehicle 202.

[0152] As previously described, the AR application 310 can transmit the event data 424 to the vehicle 202. For example, the AR application 310 can transmit the event data 424 to the vehicle computing system 210. In some cases, the vehicle computing system 210 can receive the event data 424 and provide the event data 424 to the vehicle applications 302 for processing / analysis. In some cases, the vehicle applications 302 can control behavior, operation, functionality, and / or status of the vehicle 202 based on the event data 424 and / or the sensor data 406.

[0153] For example, in some cases, the event mitigation engine 404 of the vehicle application 302 may control (e.g., modify, implement, enable, disable, manage, etc.) the behavior, operation, function, and / or state of the vehicle 202 based on the event data 424 and / or the sensor data 406. In some examples, the event mitigation engine 404 may control and / or engage one or more vehicle functions, vehicle systems, autonomous capabilities, etc. For example, the event mitigation engine 404 may control and / or engage an autopilot function, a traction control function, a cruise control function, a collision avoidance function, a lane departure function, a lane centering function, a brake assist function, a lane keeping function, a highway assist function, a lane change assist function, a speed adaptation function, a traffic light function, an intersection assist function, a blind spot monitoring system, a driver monitoring system (e.g., the monitoring engine 402 and / or any driver monitoring system), a braking system, an autonomous driving control system, a driver assistance system, a navigation system, a steering control system, a vehicle communication system, an automotive head-up display, and / or any other vehicle system, function, and / or capability.

[0154] In some cases, the event mitigation engine 404 may control the vehicle 202 based on the event data 424 to mitigate any events and / or risks resulting from the occupant's state and / or other information about the occupant included in the event data 424. In some examples, the event mitigation engine 404 may activate one or more driver assistance functions, such as functions performed by one or more advanced driver assistance systems (ADAS) of the vehicle, and / or increase the autonomous driving policy / level of the vehicle based on the event data 424. To illustrate, if the event data 424 indicates that an occupant of the vehicle 202 is impaired (e.g., distracted, intoxicated, etc.), the event mitigation engine 404 may engage the autonomous capabilities of the vehicle 202 and / or modify the autonomous driving policy / level of the vehicle 202 to assume at least partial control of the operation of the vehicle 202 while the occupant is impaired. As another example, if the event data 424 indicates that an occupant of the vehicle 202 is not paying attention to or is unable to see a nearby obstacle identified in the sensor data 406 (e.g., because the occupant's line of sight is in a different direction, because the occupant's view is obstructed, because visibility conditions are poor, etc.), the event mitigation engine 404 may engage the autonomous capabilities of the vehicle 202 and / or modify the autonomous driving policy / level of the vehicle 202 to take at least partial control of the operation of the vehicle 202 at least until the obstacle is avoided or the potential problem is eliminated.

[0155] In some examples, the event mitigation engine 404 can notify / alert other external devices of any detected impairment of the occupant. For example, the event mitigation engine 404 can notify other vehicles, one or more remote computing devices associated with other users (e.g., pedestrians, occupants of another vehicle, traffic guards / controllers, etc.), vehicle infrastructure (e.g., traffic lights, traffic cameras, street lights, signage, parking meters, lane markers, etc.), one or more emergency response systems and / or personnel, etc.

[0156] The monitoring engine 402 of the vehicle application 302 can use the sensor data 406 to monitor and / or detect the context of the vehicle 202 (related to, characterizing, and / or affecting the occupant's ability to (safely) operate the vehicle). As previously described, in some examples, the context of the vehicle 202 can include the state of the vehicle 202 (e.g., a situation, an operation, a behavior, a trajectory, a condition, a configuration, an operation metric, an environment, a vehicle parameter, a function, etc.), a vehicle event, etc. For example, the context of the vehicle 202 can indicate that a particular vehicle event (e.g., a particular activity, a condition, an incident, an environment, an operation, etc. that affects the safe operation of the vehicle) has occurred or is expected to occur. As another example, the context of the vehicle 202 can indicate the operation, behavior, and / or a situation of the vehicle 202. In some examples, the monitoring engine 402 can use the sensor data 406 to determine any aspect of the context of the vehicle 202, such as, for example, the state of the vehicle 202, a vehicle event, etc.

[0157] For example, the monitoring engine 402 may detect, recognize, and / or locate one or more items (e.g., objects, events, people, animals, roads, vehicles, scenes, activities, traffic regulations, structures, obstacles, devices, gestures, etc.) shown on one or more images in the sensor data 406 (e.g., one or more images of a scene and / or portion of an environment external to the vehicle 202 captured by one or more image sensors of the sensor system 220), determine inertial sensor data from the sensor data 406 including one or more motion measurements (e.g., speed, angular velocity, orientation, heading, acceleration, deceleration, etc.), and / or determine inertial sensor data (e.g., GPS, GPS-based ... The context of the vehicle 202 may be determined, such as by determining location information (e.g., from one or more location devices, such as a satellite signal transceiver or a Global Navigation Satellite System (GNSS) receiver) in the sensor data 406 that indicates / describes the location / position of one or more items relative to the vehicle 202 and / or the proximity / distance of one or more items (e.g., objects, events, people, animals, roads, trees, vehicles, scenes, traffic regulations, activities, gestures, devices, structures, obstacles, etc.) relative to the vehicle 202.

[0158] In some cases, the monitoring engine 402 may also use the event data 424 from the AR application 310 of the mobile device 150 to monitor and / or detect the context of the vehicle 202. For example, the monitoring engine 402 may consider occupant conditions (e.g., obstacles, line of sight, location, activity, etc.) indicated in the event data 424 when determining the context of the vehicle 202.

[0159] 5 illustrates an example use case 500 for adjusting virtual content rendered for an occupant 502 of a vehicle (e.g., vehicle 202). As previously described, the AR application 310 can adjust / modify (e.g., via the content filtering engine 412 and / or the vehicle user interface 414) the virtual content rendered by the mobile device 150. The AR application 310 can adjust the virtual content (e.g., via the content filtering engine 412 and / or the vehicle user interface 414) based on the status of the occupant determined by the monitoring engine 410 of the AR application 310 and / or the context of the vehicle 202 identified in the vehicle data 408 and / or determined by the monitoring engine 402 of the vehicle application 302.

[0160] In some examples, the AR application 310 can adjust the virtual content by disabling, dimming, or adjusting characteristics of the virtual content (e.g., size, brightness, transparency, location, orientation, etc.) In some cases, the AR application 310 can adjust all virtual content over a period of time, all virtual content in a particular context of the vehicle 202, all virtual content in an occupant state, all virtual content except a subset of virtual content marked as excluded or required (e.g., HUD content, vehicle instrumentation content, navigation content, emergency alerts, etc.), any portion of virtual content that an occupant is concentrating on, any virtual content that is obscuring a vehicle event, etc.

[0161] For example, the AR application 310 can adjust the virtual content by filtering / blocking (e.g., via the content filtering engine 412) one or more virtual content items (or all virtual content) rendered by the mobile device 150. As another example, the AR application 310 can adjust the virtual content by controlling (e.g., via the vehicle user interface 414) which virtual content is presented, when the virtual content is presented, where the virtual content is presented, one or more characteristics (e.g., transparency, size, brightness, location, orientation, etc.) of the presented virtual content, etc. In some examples, the AR application 310 can filter / block the virtual content (e.g., via the content filtering engine 412) based on the status of the occupants determined by the monitoring engine 410 of the AR application 310 and / or the context of the vehicle 202 identified in the vehicle data 408 and / or determined by the monitoring engine 402 of the vehicle application 302. In some examples, the AR application 310 can adjust the virtual content presented by the mobile device 150 (e.g., via the vehicle user interface 414) based on the status of the occupants determined by the monitoring engine 402 and / or the context of the vehicle 202 identified in the vehicle data 408 and / or determined by the monitoring engine 410.

[0162] In the example use case 500 shown in FIG. 5, an occupant 502 of a vehicle 202 wears a mobile device 150, such as an HMD or smart glasses, while using a steering wheel 505 of the vehicle 202 to drive the vehicle 202. At time T1, the mobile device 150 is rendering virtual content 510 while the occupant 502 is driving the vehicle 202. The mobile device 150 then determines (e.g., based on the vehicle data 408 or a notification from the monitoring engine 402 of the vehicle application 302) that an event 512 is occurring or is imminent (e.g., occurs within a threshold time period and / or within a threshold amount of distance traveled by the vehicle 202) that requires the attention and / or interaction of the occupant 502. In some cases, the mobile device 150 can determine a location of the event 512 (e.g., based on the vehicle data 408 or a notification from the monitoring engine 402 of the vehicle application 302). In this example, the location of the event 512 is a location outside of the vehicle 202, and the event 512 is a stroller crossing the path of the vehicle 202. The event 512 here is just one illustrative example provided for purposes of explanation. Thus, other examples may include other events and / or types of events. In some examples, the event may be classified by the monitoring engine 402 of the vehicle application 302 and / or the vehicle user interface 414 as requiring the attention and / or interaction of the occupant 502, e.g., via a machine learning classifier trained on positive and negative examples.

[0163] The location of the virtual content 510 relative to the coordinate system of the mobile device 150 is related to the location of the event 512 relative to the coordinate system of the vehicle 202. In this example, the virtual content 510 is located within / along the line of sight from the location of the mobile device 150 to the location of the event 512. Thus, the line of sight from the mobile device 150 and / or the view / visibility of the event 512 from the mobile device 150 is blocked / obstructed by the rendered virtual content 510.

[0164] To prevent the virtual content 510 from obstructing / blocking the occupant 502's ability to see the event 512, the mobile device 150 can adjust one or more characteristics of the virtual content 510. In this example, at time T2, the mobile device 150 increases the transparency of the virtual content 510 to allow visibility of the event 512 through the virtual content 510. As shown, at time T2, the event 512 is visible through the virtual content 510 after its transparency has been increased. This allows the occupant 502 to see (and monitor / track) the event 512 and make any adjustments to the operation of the vehicle 202 that the occupant 502 deems necessary based on the event 512. For example, the occupant 502 can stop the vehicle 202 to allow a stroller associated with the event 512 to pass. The occupant 502 can drive away after the stroller has passed. As another example, the occupant 502 may accelerate / decelerate the vehicle 202 and / or perform any maneuvers that may enable the vehicle 202 to avoid the stroller associated with the event 512.

[0165] On the other hand, if the virtual content 510 was not adjusted as described above to enable the occupant 502 to see the event 512, the occupant 502 may have missed / overlooked the event 512. Without seeing the event 512 (or without seeing the event 512 in time to react), the occupant 502 may not have been able to avoid the stroller associated with the event 512 or may have reacted suddenly and / or from a closer distance, which may have created a hazard or may not have been timed enough to avoid the stroller.

[0166] 6A-6E are diagrams illustrating an example use case for adjusting virtual content. Referring to FIG. 6A, the mobile device 150 in this example determines that an occupant 602 of the vehicle 202 is distracted and an event occurs within the field of view (FOV) 610 of the occupant 602. In the present disclosure, the FOV 610 of the occupant 602 may be fixed or defined by the mobile device 150. More specifically, the FOV 610 may be defined by a field of view seen on or through the respective display area of ​​the mobile device 150. As an example, the FOV 610 of the occupant 602 wearing smart glasses may be defined by the FOV through the smart glasses, which is typically reduced compared to the occupant's naked eye FOV. In another example, the occupant's physiological / anatomical FOV may be expanded by displaying a larger FOV on the display of the HMD. In some examples, the FOV 610 may be an FOV in which the mobile device 150 can render the virtual content. To highlight an event within the FOV 610 and / or draw the occupant's 602 attention to the event within the FOV 610, the mobile device 150 can render event-related virtual content 612 within the FOV 610. In this example, the virtual content 612 includes pulsing of an edge (or contour) of the FOV 610. The pulsing of the edge can highlight an area within the pulsed edge and draw the occupant's 602 attention to the area.

[0167] 6B , the mobile device 150 has detected a vehicle event 625, based on, for example, the vehicle data 408, in which a pedestrian 620 has emerged from behind a tree 615 and is moving toward (or into) the path of the vehicle 202. The pedestrian 620 was previously occupant of the vehicle 602 or the mobile device 150 or the vehicle 202, and was not visible to the occupant 602 or the mobile device 150 or the vehicle 202, until the pedestrian 620 passed the tree 615 and moved toward (or into) the path of the vehicle 202. To draw the attention of the occupant 602 to the vehicle event 625 (including the pedestrian 620) and / or to highlight the location of the vehicle event 625 (including the pedestrian 620), the mobile device 150 has rendered a highlight 622 around the location of the vehicle event 625, and more specifically, around the location of the pedestrian 620 associated with the vehicle event 625.

[0168] In some examples, the mobile device 150 may render a bounding box (or any other shape / geometry) around the vehicle event 625 and / or a portion thereof, such as the pedestrian 620. In some cases, the mobile device 150 may render the highlighting 622 according to one or more characteristics intended to capture (or further) the attention of the occupant 602. For example, the mobile device 150 may render the highlighting 622 according to a particular size, color, pattern, shape, transparency, brightness, thickness, animation, and / or any other characteristic for rendering the highlighting 622 to better capture the attention of the occupant 602.

[0169] 6C illustrates another example for rendering content to draw the occupant 602's attention to and / or highlight the vehicle event 625 (including the pedestrian 620). In this example, the mobile device 150 renders a directional indicator 630 within the FOV 610 of the occupant 602. The directional indicator 630 orients the occupant 602's head and gaze in the direction of the vehicle event 625, including the pedestrian 620. The directional indicator 630 is shown as a head-locked arrow in this example. However, other examples may include other types and / or configurations of directional indicators.

[0170] In some cases, the mobile device 150 may render the directional indicators 630 according to one or more characteristics intended to attract / capture (or further attract / capture) the attention of the occupant 602. For example, the mobile device 150 may render the directional indicators 630 according to a particular size, color, pattern, shape, transparency, brightness, thickness, animation, and / or any other characteristic. In some cases, the mobile device 150 may pulse the periphery of the occupant's 602's vision / field of vision to further attract / capture the occupant's 602's attention.

[0171] 6D illustrates an example of live content corresponding to a vehicle event presented by the mobile device 150 for an occupant 602 of the vehicle 202. The vehicle event in this example includes a pedestrian 620 crossing / passing from behind a tree 615, as described above. However, in this case, instead of rendering virtual content for the occupant 602, the mobile device 150 is presenting a camera feed 640 acquired by the mobile device 150 from a vehicle camera (e.g., a backup camera, a side-view camera, a front camera, etc.) of the vehicle 202. The camera feed 640 may depict the vehicle event including the pedestrian 620, thus enabling the occupant 602 to observe the vehicle event.

[0172] For example, the vehicle camera may stream its feed to the mobile device 150. The mobile device 150 may then present a camera feed 640 from the vehicle camera for consumption by the occupant 602. The camera feed 640 may show a vehicle event including a pedestrian 620. Thus, the camera feed 640 presented by the mobile device 150 may enable the occupant 602 to view the vehicle event (including the pedestrian 620) from the camera feed 640 presented by the mobile device 150.

[0173] 6E shows an exterior view 650 of a vehicle event 625 rendered by a mobile device 150. The exterior view 650 in this example is locked (and / or projected) to a vehicle wall 641. In some examples, the vehicle wall 641 may be based on a vehicle template as described previously herein. The vehicle wall 641 is shown as an interior wall on the passenger side of the vehicle 202. However, the exterior view 650 may be rendered and / or locked to any wall, side, or portion of the vehicle interior.

[0174] The exterior view 650 may show a vehicle event 625 including a pedestrian 620. In some examples, the exterior view 650 may include a virtual content rendering of the vehicle event 625. In other examples, the exterior view 650 may include a live content rendering (e.g., a camera feed) of the vehicle event 625. The exterior view 650 may be rendered such that the vehicle walls 641 appear transparent or semi-transparent to the occupant 602 and the vehicle event 625 appears visible to the occupant 602 (e.g., as rendered by the exterior view 650). In some cases, the exterior view 650 may be distorted and / or perspective corrected based on the position of the mobile device 150 (and thus the occupant 602 wearing the mobile device 150) relative to the vehicle's coordinate system and / or the position of the vehicle event 625 relative to the vehicle's coordinate system.

[0175] In some cases, content rendered by mobile device 150 may include vehicle data (e.g., vehicle data 408) or portions of vehicle data. For example, content rendered in any of the examples shown in Figures 6A-6E may include vehicle data or portions of vehicle data, such as, for example, vehicle status information, vehicle instrumentation data, vehicle information, data from one or more sensors of vehicle 202, etc. To illustrate, the content rendered by the mobile device 150 may include an indication of the speed of the vehicle 202, the direction of the vehicle 202, the route of the vehicle 202, the battery status of the vehicle 202, the fuel status of the vehicle 202, the oil status of the vehicle 202, a mileage associated with the vehicle 202, navigation data associated with the vehicle 202, tire pressure (and / or tire status / status) associated with one or more tires of the vehicle 202, an indication of control signals that are active / on / enabled on the vehicle 202, information about the vehicle brakes, vehicle warnings, vehicle errors, information (e.g., status / condition, warnings, etc.) regarding one or more vehicle lights (e.g., headlights, tail lights, reverse lights, daytime running lights, fog lights, off-road lights, signal lights, brake lights, emergency lights, etc.), information about the steering system, information about one or more vehicle systems and / or autonomous functions, cruise control status information, seat belt information, and / or any other vehicle information and / or instrumentation.

[0176] In some cases, the mobile device 150 can render vehicle information (e.g., vehicle status information, instrumentation, navigation information, etc.) streamed from the vehicle 202. In some examples, the mobile device 150 can present the vehicle information as a head-locked HUD or world-locked user interface (UI) elements. In some cases, the mobile device 150 can render virtual content to replace real-world content and / or events that may distract the occupants, such as billboards, accidents, etc. In some aspects, the mobile device 150 can render any of the virtual content described herein using a deep neural network trained to generate “deepfaked” (e.g., synthetic) virtual content from a database of images. In some cases, the “deepfaked” virtual content can be rendered in a style observed by the mobile device 150 and / or the vehicle 202.

[0177] As discussed above, the monitoring engine 410 of the AR application 310 of the mobile device 150 can monitor and understand the state of one or more occupants of the vehicle 202. In some cases, the monitoring engine 402 of the vehicle application 302 of the vehicle computing system 210 can alternatively or additionally (e.g., in combination with the monitoring engine 410 of the AR application 310 or separately) monitor and understand the state of one or more occupants of the vehicle 202. In some examples, the state of the occupant can include one or more characteristics of the occupant, such as, for example, a detected obstacle, gaze, location, occupant posture (e.g., orientation, location, etc.), occupant head pose, gestures, facial expressions, hand gestures, activity, emotion, movement, intent to perform an action, and / or other characteristics of the occupant.

[0178] For example, the monitoring engine 410 and / or the monitoring engine 402 (either alone or in combination with data from the mobile device 150, such as the monitoring engine 410) may detect and / or recognize various states of an occupant of the vehicle. Figures 7A-7I illustrate different example states of an occupant 750 driving the vehicle 202, as detected by the monitoring engine 410 of the AR application 310 of the mobile device 150 and / or the monitoring engine 402 of the vehicle 202. The occupant 750 in the example illustrated in Figures 7A-7I is the driver of the vehicle 202 wearing the mobile device 150.

[0179] In the example shown in Figure 7A, an occupant 750 is wearing a mobile device 150, such as smart glasses, and is engaged in a normal driving activity 702. The monitoring engine 410 can determine that the occupant 750 is engaged in a normal driving activity 702. As shown, the normal driving activity 702 in this example includes the occupant 750 holding the steering wheel with both hands and looking directly out the windshield of the vehicle 202. Additionally, the occupant 750 is not impaired (e.g., distracted, etc.) during the normal driving activity 702.

[0180] FIG. 7B illustrates an example state 704 of an occupant 750 while driving the vehicle 202. In this example, the state 704 includes the occupant 750 using the cell phone 752 by holding the cell phone 752 to his ear while driving the vehicle 202. In some cases, the monitoring engine 410 may determine that the occupant 750 is using the cell phone 752 and determine that the use of the cell phone 752 may be harmful (e.g., distracting) to the occupant 750. Thus, in some examples, the monitoring engine 410 may determine an impairment of the occupant 750 based on the use of the cell phone 752. The impairment in this example includes distraction. The monitoring engine 410 may also determine that if the occupant 750 needs to use the hand holding the cell phone 752 to control the steering wheel and / or any other vehicle controls, the occupied hand may delay (and therefore impair) the occupant's ability to control the steering wheel and / or any other vehicle controls, thus increasing the risk of an accident. In general, the monitoring engine 410 may detect an obstacle condition based on the location and / or activity of one or both hands of the occupant, e.g., based on determining that at least one hand is not on the steering wheel. In some examples, the monitoring engine 410 may detect an obstacle condition based on the occupant operating a mobile phone.

[0181] FIG. 7C illustrates another example state 706 of an occupant 750 driving a vehicle 202. In this example, the state 706 of the occupant 750 includes the occupant 750 drinking a beverage. The monitoring engine 410 may determine that the occupant 750 is drinking a beverage, as shown in FIG. 7C. In some cases, the monitoring engine 410 may determine that drinking the beverage harms the occupant 750 (e.g., harms the occupant's 750 ability to drive the vehicle 202, harms the occupant's 750 response and / or reaction time to a vehicle event, increases the difficulty of driving the vehicle 202, increases the risk of an accident, etc.). For example, the monitoring engine 410 may determine that drinking the beverage distracts the occupant 750 and / or occupies at least one of the occupant's 750 hands, thus preventing the occupant 750 from using the hands to operate the vehicle 202 and / or respond to a vehicle event. The monitoring engine 410 may also determine that if the occupant 750 needs to use their hands to control the steering wheel and / or any other vehicle controls, the hands being occupied may delay (and thus impair) the occupant's ability to control the steering wheel and / or any other vehicle controls, thus increasing the risk of an accident. In general, the monitoring engine 410 may detect an obstruction condition based on the location and / or activity of one or both of the occupant's hands, e.g., based on determining that at least one hand is not on the steering wheel.

[0182] 7D illustrates another example state 708 of an occupant 750 driving the vehicle 202. In this example, the state 708 includes the occupant 750 adjusting his / her hair. As shown, the occupant 750 has both hands off the steering wheel of the vehicle 202 while adjusting his / her hair. The monitoring engine 410 may determine that the occupant 750 is adjusting his / her hair and has both hands off the steering wheel. In some cases, the monitoring engine 410 may determine that having both hands off the steering wheel is harming the occupant 750 (e.g., harming the occupant's 750 ability to drive the vehicle 202, harming the occupant's 750 reaction and / or reaction time to a vehicle event, increasing the difficulty of driving the vehicle 202, increasing the risk of an accident, etc.). For example, the monitoring engine 410 may determine that adjusting hair distracts the occupant 750 and / or removing both hands from the steering wheel prevents the occupant 750 from using their hands to operate the vehicle 202 and / or respond to an event. The monitoring engine 410 may also determine that if the occupant 750 needs to take control of the steering wheel (and / or any other vehicle controls), removing their hands from the steering wheel to adjust their hair may delay (and thus reduce) their ability to take control of the steering wheel (and / or any other vehicle controls), thus increasing the risk of an accident. In general, the monitoring engine 410 may detect an obstruction condition based on the location and / or activity of one or both hands of the occupant, e.g., based on determining that at least one hand is not on the steering wheel.

[0183] 7E illustrates another example state 710 of an occupant 750 driving a vehicle 202. In this example, the state 710 includes extending an arm 756 out of the driver's side window of the vehicle 202. As shown, the occupant 750 extends the arm 756 outside the driver's side window and keeps the hands of the arm 756 away from the steering wheel and any other vehicle controls. The monitoring engine 410 can determine that the occupant 750 extends the arm 756 outside the driver's side window and keeps the hands of the arm 756 away from the steering wheel and any other vehicle controls. In some cases, the monitoring engine 410 may determine that placing an arm 756 out of the driver's side window and the hands of the arm 756 away from the steering wheel and any other vehicle controls is harmful to the occupant 750 (e.g., is harmful to the occupant's 750's ability to drive the vehicle 202, is harmful to the occupant's 750's reaction and / or reaction time to a vehicle event, is increasing the difficulty of driving the vehicle 202, is increasing the risk of an accident, etc.). For example, the monitoring engine 410 may determine that placing an arm 756 out of the driver's side window and the hands of the arm 756 away from the steering wheel and any other vehicle controls is preventing the occupant 750 from quickly using their hands to operate the vehicle 202 and / or respond to an event. In some examples, the monitoring engine 410 may determine that if the occupant 750 needs to use the hands of the arms 756 to control the steering wheel and / or any other vehicle controls, making the arms 756 and the hands of the arms 756 temporarily unavailable (e.g., extending outside the driver's side window) may delay (and thus impair) the occupant's ability to control the steering wheel and / or any other vehicle controls, thus increasing the risk of an accident. In general, the monitoring engine 410 may detect an obstruction condition based on the location and / or activity of one or both of the occupant's hands, e.g., based on determining that at least one hand is not on the steering wheel.

[0184] FIG. 7F illustrates another example state 712 of an occupant 750 driving the vehicle 202. In this example, the state 712 includes yawning while driving the vehicle 202. The monitoring engine 410 may determine that the occupant 750 is yawning while driving the vehicle 202. In some cases, the monitoring engine 410 may determine that the yawning is indicative of an impairment of the occupant 750. For example, the monitoring engine 410 may determine that the yawning is indicative of a drowsy state and / or limited focus / attention that may adversely affect the occupant's 750 ability to drive the vehicle 202 and / or respond to vehicle events. In general, the monitoring engine 410 may detect the impaired state based on facial expressions such as yawning, closed / droopy eyes / eyelids, and / or the occupant's body posture indicative of a state of the occupant having limited / reduced focus / attention.

[0185] FIG. 7G illustrates another example state 714 of an occupant 750 driving the vehicle 202. In this example, the state 714 includes the occupant 750 looking down toward the occupant's 750 laps while driving the vehicle 202. The monitoring engine 410 may determine that the occupant 750 is looking down toward the occupant's 750 laps while driving the vehicle 202. In some cases, the monitoring engine 410 may determine that looking down toward the occupant's 750 laps while driving impairs the occupant's 750 ability to operate the vehicle 202 and / or respond to vehicle events. For example, the monitoring engine 410 may determine that looking down toward the occupant's 750 laps distracts the occupant 750 and prevents the occupant 750 from viewing the road and external environment and quickly detecting and responding to vehicle events associated with driving the vehicle 202. In general, the monitoring engine 410 can detect an impaired state based on facial expressions such as yawning, closed / droopy eyes / lids, and / or the body posture of the occupant indicative of a state of the occupant having limited / reduced concentration / attention.

[0186] 7H illustrates another example state 716 of an occupant 750 driving the vehicle 202. In this example, the state 716 includes the occupant 750 looking to the left and out of the driver's side window of the vehicle 202 while driving the vehicle 202. The monitoring engine 410 may determine that the occupant 750 is looking to the left and out of the driver's side window of the vehicle 202 while driving the vehicle 202. In some cases, depending on the context of the vehicle 202, the monitoring engine 410 may determine that looking to the left and out of the driver's side window of the vehicle 202 while driving impairs the occupant 750's ability to operate the vehicle 202 and / or respond to vehicle events. For example, depending on the context of the vehicle 202, the monitoring engine 410 may determine that looking to the left and out of the driver's side window of the vehicle 202 will distract the occupant 750, preventing the occupant 750 from seeing the road ahead and / or anything along the path ahead of the vehicle 202, impairing the ability of the occupant 750 to detect and respond to vehicle events along the path ahead of the vehicle 202. In general, the monitoring engine 410 may detect an obstacle condition based on the occupant's line of sight, such as looking / focusing in a direction away from the road ahead.

[0187] In other cases, depending on the context of the vehicle 202, the monitoring engine 410 may determine that looking to the left and out of the driver's side window of the vehicle 202 while driving does not impair the ability of the occupant 750 to operate the vehicle 202 and / or respond to a vehicle event. For example, if the context of the vehicle 202 indicates that a vehicle event occurred outside the driver's side window and the occupant 750 should recognize and / or respond to the vehicle event, the monitoring engine 410 may determine that by looking to the left and out of the driver's side window of the vehicle 202, the occupant 750 can see the vehicle event and determine how (or if) to respond to the vehicle event. Thus, the monitoring engine 410 may determine that in this example context, the occupant 750 looking to the left and out of the driver's side window of the vehicle 202 while driving is appropriate and should not be classified or treated as impaired.

[0188] FIG. 7I illustrates another example state 718 of an occupant 750 driving the vehicle 202. In this example, the state 718 includes the occupant 750 not having a seat belt fastened while driving the vehicle 202. The monitoring engine 410 may determine that the occupant 750 is not having a seat belt fastened while driving the vehicle 202. In some cases, the monitoring engine 410 may determine that not having a seat belt fastened while driving the vehicle 202 is an impairment if it increases a risk to the safety of the occupant 750 and others (e.g., any other passengers of the vehicle 202, and any other people in the environment of the vehicle 202, such as occupants of other vehicles, pedestrians, etc.). In other cases, the monitoring engine 410 may determine that not having a seat belt fastened while driving the vehicle 202 is not an impairment if it does not impair the occupant's 750's ability to drive the vehicle 202 or to respond to vehicle events.

[0189] 8 is a diagram illustrating an example of V2X communication 800 including an occupant monitoring event. In this example, the mobile device 150 is a wearable augmented reality (AR) device, such as an HMD or AR glasses, worn by an occupant of the vehicle 202. The occupant of the vehicle 202 is also wearing a wearable device 802, such as a smart or connected watch, a health tracker / monitor device, a smart or connected bracelet device, a wearable medical device, or any other wearable device.

[0190] The mobile device 150 and the vehicle computing system 210 can exchange data as described above, such as vehicle data (e.g., vehicle context information, vehicle templates, vehicle sensor data, etc.), event data, and / or occupant data (e.g., occupant status information, occupant sensor measurements, occupant health measurements, etc.). The wearable device 802 can include a wearable application 804 that can track / measure information about the occupant, such as health metrics, biometrics, etc. The wearable device 802 can include health and other sensors, such as, for example, but not limited to, an oximeter, skin sensors, optical heart sensors, photoplethysmography sensors, electrical heart sensors, accelerometers, gyroscopes, electrocardiogram sensors, temperature sensors, blood pressure sensors, galvanic skin response sensors, brainwave sensors, and / or any other sensors.

[0191] In some examples, the wearable device 802 may have additional and / or redundant sensor modalities (e.g., relative to those of the mobile device 150) that may assist in occupant monitoring. The wearable device 802 may transmit such data (or a digest of relevant events) to the mobile device 150 to assist in its occupant monitoring process. In some examples, the wearable device 802 may transmit sensor data to the mobile device 150, such as, for example, inertial sensor data, heart rate measurements, blood pressure measurements, galvanic skin response measurements, ECG / EKG / EEG data, temperature data, oxygen levels, motion information, sleep tracking information, etc. In some cases, data from the wearable device 802 may indicate disorders such as drowsiness, intoxication, health emergency, stress, heightened emotional state, loss of consciousness, etc., and / or may be used to determine disorders (e.g., via machine learning classifiers trained on positive and negative examples).

[0192] The wearable application 804 of the wearable device 802 can obtain health measurements and any other measurements from one or more sensors on the wearable device 802. The wearable device 802 can transmit any health measurements to the mobile device 150. The mobile device 150 can use the health measurements to monitor the status of the occupant. The mobile device 150 can use the health measurements alone or in combination with other measurements from the mobile device 150, such as any measurements previously described with respect to the monitoring engine 410. In some examples, the mobile device 150 can perform sensor data fusion and use a combination of the health measurements from the wearable device 802 and any occupant monitoring measurements and / or data from the mobile device 150 to determine the status of the occupant. The mobile device 150 can use the monitoring engine 410 as described above to determine the status of the occupant. The monitoring engine 410 can use either the data from the mobile device 150 and / or the wearable device 802 to determine the status of the occupant.

[0193] In some cases, the mobile device 150 can transmit the determined occupant status to the vehicle computing system 210. The mobile device 150 can additionally or alternatively transmit any event data to the vehicle computing system 210, as described above. In some cases, the mobile device 150 can transmit the determined occupant status to other devices, such as, for example, the vehicle 810 (e.g., the vehicle computing system 812), the infrastructure 820, and / or any other devices. The infrastructure 820 can include any infrastructure system, device, and / or component, such as, for example, but not limited to, traffic lights, traffic cameras, street lights, signage, parking meters, lane markers, one or more emergency response systems, combinations thereof, and / or any other infrastructure system, device, and / or component. In some cases, the vehicle computing system 210 can transmit / forward the determined occupant status to other devices, such as, for example, the vehicle 810 (e.g., the vehicle computing system 812), the infrastructure 820, and / or any other devices.

[0194] In some cases, the vehicle computing system 210 can also determine the occupant's state (e.g., via the monitoring engine 402) using data from one or more sensors of the vehicle 202, such as one or more image sensors, inertial sensors, weight sensors, pressure sensors, audio sensors, and / or event data 424 from the mobile device 150. For example, in some cases, the vehicle computing system 210 can determine the occupant's state in addition to the determination of the occupant's state made by the mobile device 150. In some examples, the vehicle computing system 210 can determine the occupant's state using sensor data from one or more sensors of the mobile device 150, the wearable device 802, and / or the vehicle 202. For example, the vehicle computing system 210 can fuse sensor data from one or more sensors of the mobile device 150, the wearable device 802, and / or the vehicle 202 to determine the occupant's state. In some cases, the vehicle computing system 210 can use the occupant's state received from the mobile device 150 to assist in its own determination of the occupant's state. In other cases, the vehicle computing system 210 may separately determine the occupant state without the occupant state being determined by the mobile device 150 and / or sensor data from the mobile device 150.

[0195] The vehicle computing system 210 of the vehicle 810 can communicate any vehicle and / or occupant data to other devices, such as, for example, the vehicle 202 and / or the infrastructure 820. For example, the vehicle computing system 210 can transmit the determined status of the occupant to the vehicle 810 and / or the infrastructure 820 (e.g., in addition to or instead of the mobile device 150 transmitting the occupant status to the vehicle 810 and / or the infrastructure 820). The vehicle 810 and / or the infrastructure 820 can use any occupant status information received from the mobile device 150 and / or the vehicle 202 to call for assistance for the occupant if the status information indicates an impairment of the occupant and / or to take any action to prevent any accident or collision with the vehicle 202 operated by the occupant while impaired.

[0196] For example, vehicle applications 814 of vehicle computing system 812 of vehicle 202 may use data from vehicle 810 and / or mobile device 150 to adjust the operation / behavior of vehicle 202 to avoid any accidents or collisions with vehicle 810. In some cases, vehicle applications 814 of vehicle computing system 812 of vehicle 202 may use data from vehicle 810 and / or mobile device 150 to reroute vehicle 202 to avoid vehicle 202 and / or maintain at least a certain distance from vehicle 810. Vehicle computing system 812 may be configured to include the same or similar components as vehicle computing system 210 described above in conjunction with FIG.

[0197] As another example, infrastructure applications 822 of infrastructure 820 may use data from vehicle 202 and / or mobile device 150 to adjust operations / behavior of infrastructure 820. By way of illustration, infrastructure applications 822 of infrastructure 820 may use data from vehicle 202 and / or mobile device 150 to adjust traffic restrictions / signals associated with infrastructure 820 based on a disability of an occupant of vehicle 202 and / or report a disability of an occupant of vehicle 202 to law enforcement and / or other emergency or assistance personnel and / or systems.

[0198] In some examples, vehicle computing system 210 may also transmit other information to vehicle 810 and / or infrastructure 820. For example, vehicle computing system 210 may transmit sensor data from vehicle 202 and / or determined context of vehicle 202 to vehicle 810 and / or infrastructure 820. In some cases, vehicle computing system 210 may transmit such information in addition to any state information regarding the occupants of vehicle 202.

[0199] 9 illustrates an example vehicle mitigation event based on an occupant state determined by a mobile device 150 worn by an occupant driving the vehicle 202. In this example, the mobile device 150 has determined (e.g., via the monitoring engine 410, and possibly based on data from the wearable device 802) an occupant state 910 indicating that the occupant driving the vehicle 202 is impaired by a health emergency. The mobile device 150 can transmit the occupant state 910 to a vehicle computer system (e.g., the vehicle computing system 210) of the vehicle 202 to notify the vehicle 202 that the occupant is impaired by a health emergency.

[0200] A vehicle computer system (e.g., vehicle computing system 210) of vehicle 202 determines a vehicle context 912 that describes the state of vehicle 202, such as the state of motion of vehicle 202. In this example, vehicle context 912 indicates that vehicle 202 is traveling west at 10 miles per hour (mph) and has encountered a vehicle event ahead. The vehicle event in this example is an obstacle 902 along the path of vehicle 202.

[0201] Based on the occupant state 910 and vehicle context 912 obtained from the mobile device 150, the vehicle computing system of the vehicle 202 can trigger an action 916 (e.g., via an event mitigation engine, such as the event mitigation engine 404). In this example, the action 916 includes controlling an autonomous vehicle of the vehicle 202 to control the vehicle 202 while the occupant is impaired by the health emergency. In some examples, the vehicle 202 can use the autonomous capabilities to drive the occupant to a destination route. In other examples, the vehicle 202 can use the autonomous capabilities to reroute the vehicle 202 and drive the occupant to a hospital / clinic or a location where the occupant can obtain treatment for the health emergency. In some cases, the vehicle 202 can also send a message / notification to a remote system, such as a health emergency system, indicating that the occupant is experiencing a health emergency and / or requesting assistance for the health emergency.

[0202] The actions 916 also include performing a maneuver 904 to avoid the obstacle 902. The vehicle 202 can use its autonomous capabilities to perform the maneuver 904 to avoid the obstacle 902. In this example, the maneuver 904 includes making a left turn on the road in front of the obstacle 902.

[0203] 9 are merely illustrative examples provided for purposes of explanation. Other examples may include the same or different and / or additional obstacles, maneuvers, occupant states, vehicle contexts, actions, etc.

[0204] In some examples, the vehicle context may include information regarding the operation / driving of the vehicle. The events or vehicle events may include information related to vehicle safety, vehicle safety events / conditions, etc. The occupant status may include and / or may be related to the occupant's distraction or health condition. In some cases, the distraction or health condition may include the occupant's temporary distraction or health condition.

[0205] 10 illustrates another example vehicle mitigation event based on an occupant state determined by a mobile device 150 worn by an occupant driving the vehicle 202. In this example, the mobile device 150 has determined (e.g., via the monitoring engine 410) an occupant state 1002 indicating that an occupant driving the vehicle 202 is distracted. The mobile device 150 can transmit the occupant state 1002 to a vehicle computer system (e.g., vehicle computing system 210) of the vehicle 202 to notify the vehicle 202 that the occupant is distracted.

[0206] The vehicle 202 may detect (e.g., via a monitoring engine such as monitoring engine 402) that the vehicle 202 is crossing a center lane 1012 or lane divide of a road 1010 along which the vehicle 202 is traveling. Based on the occupant state 1002 indicating that the occupant is distracted and a determination that the vehicle 202 is crossing the center lane 1012 of the road 1010, at time T1, the vehicle 202 may trigger an action 1004 (e.g., via an event mitigation engine such as event mitigation engine 404) to assist the distracted occupant. In this example, the action 1004 includes activating a lane keeping assist feature of the vehicle 202 to take corrective action to keep the vehicle 202 within its lane 1020 and to avoid the vehicle 202 crossing into another lane 1022.

[0207] At time T2, after the vehicle 202 engaged the lane keeping assist function and took corrective action, the vehicle 202 is now traveling in its lane 1020. As shown in FIG. 10, the occupant status 1006 also indicates that the vehicle occupant is in a normal state (e.g., unimpaired) at time T2. The vehicle 202 may enable the occupant to drive the vehicle 202 with or without autonomous assistance, may continue to perform one or more autonomous driving operations, may provide the occupant with an option to select whether to manually control the vehicle 202 (fully or partially), enable the vehicle 202 to operate autonomously, etc.

[0208] 11 is a flowchart illustrating an example process 1100 for controlling the presentation of virtual content during vehicle operation. At block 1102, the process 1100 can include determining an attitude of a mobile device (e.g., mobile device 150) relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle (e.g., vehicle 202).

[0209] At block 1104, the process 1100 may include receiving data from the vehicle associated with one or more sensors of the vehicle.

[0210] At block 1106, the process 1100 may include displaying virtual content using a display device of the mobile device based on data associated with the one or more sensors and an attitude of the mobile device relative to the vehicle's coordinate system.

[0211] In some aspects, the process 1100 may include determining a context for the vehicle based on data associated with one or more sensors. In some examples, the context may include an event associated with the vehicle.

[0212] In some aspects, process 1100 may include associating a location of the virtual content relative to a coordinate system of the mobile device with a location of the event relative to the coordinate system of the vehicle, and displaying a live content feed from one or more visual image sensors of the mobile device based on the association of the location of the virtual content relative to the coordinate system of the mobile device and the location of the event relative to the coordinate system of the vehicle.

[0213] In some examples, displaying the live content feed may include determining an attitude of the mobile device relative to the event based on an attitude of the mobile device relative to a coordinate system of the vehicle and an attitude of the vehicle relative to the event, and determining that the virtual content at least partially occludes the event in a field of view of an occupant of the vehicle based on the attitude of the mobile device relative to the event and an association of a location of the virtual content relative to the coordinate system of the mobile device and a location of the event relative to the coordinate system of the vehicle.

[0214] In some aspects, process 1100 may include associating a location of the virtual content item relative to the coordinate system of the mobile device with a location of the event relative to the coordinate system of the vehicle, and filtering or modifying the virtual content item based on the association of the location of the virtual content relative to the coordinate system of the mobile device and the location of the event relative to the coordinate system of the vehicle.

[0215] In some examples, filtering or modifying the virtual content item may include modifying one or more characteristics of the virtual content item. In some cases, the one or more characteristics may include at least one of the transparency, the size, the location of the virtual content item, and the brightness level of the virtual content item.

[0216] In some examples, filtering or modifying the virtual content item may include determining a line of sight of an occupant of the vehicle, the occupant associated with a mobile device, determining visibility of the event to the occupant based on the line of sight of the occupant and a location of the event, and filtering or modifying the virtual content item further based on the visibility of the event to the occupant.

[0217] In some examples, the event may include at least one of: a presence of an object in the path of the vehicle or a threshold proximity to the path of the vehicle, a traffic regulation associated with the path of the vehicle, and the vehicle failing to stay within at least one of a speed limit and a lane marking. In some cases, the object in the path of the vehicle or the threshold proximity to the path of the vehicle includes at least one of a pedestrian, an animal, and another vehicle.

[0218] In some aspects, process 1100 may include determining a line of sight of a vehicle occupant, the occupant associated with a mobile device, and rendering virtual content within the direction of the line of sight of the vehicle occupant.

[0219] In some cases, rendering the virtual content within the occupant's line of sight may include rendering a virtual content overlay comprising a virtual indicator of at least one of the event, the location of the event, and the direction of the event.

[0220] In some cases, rendering the virtual content within the direction of the occupant's line of sight may include modifying one or more characteristics of the virtual content. In some cases, the one or more characteristics may include at least one of a transparency, a size, a location of the virtual content, and a brightness level.

[0221] In some aspects, the process 1100 may include filtering at least a portion of the virtual content based on the context of the vehicle and the pose of the mobile device.

[0222] In some cases, filtering at least a portion of the virtual content can include enabling presentation of a subset of the virtual content. In some examples, the subset of the virtual content can include at least one of an indication of a vehicle condition and vehicle instrumentation information.

[0223] In some aspects, displaying the virtual content may include rendering a virtual content item associated with the vehicle, the virtual content item being rendered against a surface of the vehicle.

[0224] In some cases, the rendered virtual content item may include at least one of a first indication of an event identified in data associated with the one or more sensors, a second indication of a vehicle context, and an alert associated with the vehicle context.

[0225] In some aspects, rendering the virtual content item may include receiving a camera feed from a camera device of the vehicle and displaying at least a portion of the camera feed within a display area of ​​the mobile device.

[0226] In some cases, determining the attitude of the mobile device may include acquiring one or more radio frequency (RF) signals and determining the attitude of the mobile device based on the one or more images and at least one of a round trip time associated with the one or more RF signals, a time of arrival associated with the one or more RF signals, and a received signal strength indicator (RSSI) associated with the one or more RF signals.

[0227] In some cases, determining the pose of the mobile device may include receiving, from the vehicle, a vehicle template including one or more markers associated with the vehicle, and determining a pose of the mobile device relative to a coordinate system of the vehicle based on the one or more images and the vehicle template.

[0228] In some examples, the one or more markers may include at least one of a visual pattern of an area within the interior of the vehicle and / or an object attached to the interior of the vehicle, an element within the interior of the vehicle, a surface within the interior of the vehicle, and an illuminated object inside the vehicle.

[0229] In some cases, determining the pose of the mobile device may include detecting one or more markers in the one or more images, and determining the pose of the mobile device relative to a coordinate system of the vehicle based on the detected one or more markers and the vehicle template.

[0230] In some aspects, process 1100 may include acquiring a set of images of an interior of a vehicle using one or more image sensors of a mobile device, where the set of images depict one or more markers associated with the vehicle, and generating a vehicle template including the one or more markers based on the set of images.

[0231] 12 is a flow chart illustrating an example process 1200 for monitoring an occupant of a vehicle. At block 1202, the process 1200 may include determining an attitude of the mobile device relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle. In some cases, the obstacles may include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an obstacle location for controlling the vehicle, and an obstacle visibility with respect to at least one of an operation of the vehicle and an event associated with the vehicle.

[0232] At block 1204, the process 1200 may include determining a state of an occupant of the vehicle. In some examples, the state of the occupant may include an impairment of the occupant with respect to operating the vehicle.

[0233] At block 1206, the process 1200 may include transmitting data to the vehicle indicative of the state of the occupants and the attitude of the mobile relative to the vehicle's coordinate system.

[0234] In some examples, determining the occupant state may include receiving, from the vehicle, data associated with one or more sensors of the vehicle, and determining the occupant state based on the data associated with the one or more sensors of the vehicle and an attitude of the mobile device.

[0235] In some cases, data associated with one or more sensors of the vehicle is indicative of at least one of a state of the vehicle and an event associated with the vehicle.

[0236] In some cases, the events associated with the vehicle may include at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0237] In some examples, objects within the vehicle's path or threshold proximity to the vehicle's path may include at least one of a pedestrian, an animal, and another vehicle.

[0238] In some cases, an occupant impairment may include any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0239] In some aspects, the process 1200 can include determining a line of sight of an occupant of the vehicle. In some cases, the state of the occupant can include a line of sight of the occupant, and the occupant can be associated with a mobile device. For example, the occupant can be a user wearing the mobile device.

[0240] In some cases, determining the occupant's status may include receiving one or more health measurements associated with the occupant from one or more sensors associated with at least one of the mobile device and a wearable device worn by the occupant, and determining the occupant's status based on the one or more health measurements.

[0241] In some examples, the one or more health measurements may include at least one of heart rate, blood pressure, body temperature, galvanic skin response, a measurement of electrical signals from the occupant's heart, a measurement of electrical activity in the occupant's brain, the amount of redness in the eyes, and pupil size.

[0242] In some cases, determining the occupant's condition may include determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time, and determining an impairment condition of the occupant based on a determination that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time and that the period of time exceeds a threshold period of time.

[0243] In some examples, determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time may include determining that the occupant's gaze is focused on virtual content rendered by the mobile device for at least a portion of the period of time.

[0244] In some cases, determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time may include determining that the occupant's gaze is focused in the direction of an obstacle in the vehicle's path or in a direction other than a threshold proximity to the vehicle's path.

[0245] In some aspects, process 1200 may include transmitting an indication of the occupant's status to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0246] In some cases, determining the occupant state may include determining a line of sight of an occupant wearing a mobile device, and determining the occupant state based on the attitude of the mobile device and the occupant's line of sight.

[0247] In some cases, determining the pose of the mobile device may include receiving, from the vehicle, a vehicle template including one or more markers associated with the vehicle, and determining a pose of the mobile device relative to a coordinate system of the vehicle based on the one or more images and the vehicle template.

[0248] In some cases, the one or more markers may include visual patterns of at least one of areas within the interior of the vehicle and objects attached to the interior of the vehicle, elements within the interior of the vehicle, surfaces within the interior of the vehicle, and illuminated objects inside the vehicle.

[0249] In some cases, the one or more images may depict one or more markers, and determining the pose of the mobile device may include detecting the one or more markers in the one or more images, and determining a pose of the mobile device relative to a coordinate system of the vehicle based on the detected one or more markers and the vehicle template.

[0250] In some aspects, process 1200 may further include acquiring a set of images of an interior of the vehicle using one or more image sensors of the mobile device, where the set of images depict one or more visual landmarks associated with the vehicle, and generating a vehicle template based on the set of images, where the vehicle template includes the one or more markers.

[0251] In some cases, determining the attitude of the mobile device may include obtaining inertial sensor data associated with the mobile device and determining the attitude of the mobile device based on one or more of the images and the inertial sensor data.

[0252] 13 is a flow chart illustrating an example process 1300 for controlling operation of a vehicle. At block 1302, the process 1300 may include receiving, from a mobile device associated with an occupant of the vehicle, a pose of the occupant relative to a coordinate system of the vehicle. At block 1304, the process 1300 may include controlling one or more functions of the vehicle based on the pose of the occupant relative to a coordinate system of the vehicle.

[0253] In some examples, controlling the one or more functions of the vehicle may include participating in one or more vehicle functions of the vehicle. In some examples, the one or more vehicle functions may include at least one of an autopilot function, a traction control function, a cruise control function, a collision avoidance function, a lane departure function, a lane centering function, a brake assist function, a lane keeping function, a highway assist function, a lane change assist function, a speed adaptation function, and an intersection assist function.

[0254] In some examples, controlling the one or more functions of the vehicle may include controlling or engaging one or more autonomous vehicle systems of the vehicle. In some examples, the one or more autonomous vehicle systems may include at least one of a blind spot monitoring system, a driver monitoring system, a braking system, an autonomous driving control system, a driver assistance system, a navigation system, a steering control system, a vehicle communication system, and an automotive head-up display.

[0255] In some aspects, process 1300 may include transmitting data associated with one or more sensors of the vehicle to at least one of the mobile device, the second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0256] In some aspects, process 1300 may include transmitting data indicative of the occupant's status to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0257] In some aspects, the process 1300 may include transmitting a vehicle template to the mobile device for determining the occupant's pose relative to the vehicle's coordinate system, the vehicle template including one or more markers associated with the vehicle.

[0258] In some examples, the one or more markers may include at least one of a visual pattern of an area within the interior of the vehicle and / or an object attached to the interior of the vehicle, an element within the interior of the vehicle, a surface within the interior of the vehicle, and an illuminated object inside the vehicle.

[0259] In some aspects, the process 1300 may include generating an output based at least in part on the occupant's attitude relative to a coordinate system of the vehicle. In some examples, the output may include at least one of a communication to a mobile device, an instruction to modify one or more functions of the vehicle, and an indication of the occupant's status.

[0260] In some cases, the communication to the mobile device may include at least one of a virtual content item and a request to display the virtual content item.

[0261] In some aspects, the process 1300 may include receiving data from a mobile device indicative of a status of an occupant of the vehicle. In some cases, the data may include at least one of sensor data from one or more sensors of the mobile device and processed data generated based on the sensor data from the one or more sensors of the mobile device.

[0262] In some cases, the processed data may include at least one of a description of the occupant's state and a classification output identifying the occupant's state.

[0263] In some aspects, the process 1300 may include receiving data from a mobile device indicative of a state of an occupant of the vehicle. In some cases, the data may include at least one of an indication of the occupant's line of sight, an attitude of the occupant relative to a coordinate system of the vehicle, and one or more health measurements associated with the occupant.

[0264] In some cases, the one or more health measurements may include at least one of heart rate, blood pressure, body temperature, galvanic skin response, a measurement of electrical signals from the occupant's heart, a measurement of electrical activity in the occupant's brain, the amount of redness in the eyes, and pupil size.

[0265] In some aspects, process 1300 may include obtaining sensor data from one or more sensors of the vehicle, the sensor data including at least one of an indication of an event related to operation of the vehicle, an indication of one or more driving patterns during one or more operations of the vehicle that are at least partially controlled by an occupant, and vehicle instrumentation data. In some cases, process 1300 may include controlling one or more functions of the vehicle further based on the sensor data.

[0266] In some aspects, process 1300 may include determining a state of the occupant. In some cases, determining the state of the occupant may include receiving one or more health measurements associated with the occupant from one or more health sensors associated with at least one of the mobile device and the wearable device, and determining the state of the occupant based on the one or more health measurements associated with the occupant.

[0267] In some aspects, the process 1300 may include obtaining a state of the occupant. In some examples, the state of the occupant may include an impairment of the occupant with respect to operating the vehicle. In some examples, the impairment may include a temporary impairment of the occupant. In some cases, the impairment may include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an impairment location to control the vehicle, and an impairment visibility with respect to at least one of the operation of the vehicle and an event associated with the vehicle. In some cases, the impairment of the occupant may include any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0268] In some examples, the event may include at least one of: a presence of an object in the path of the vehicle or a threshold proximity to the path of the vehicle, a traffic regulation associated with the path of the vehicle, and the vehicle failing to stay within at least one of a speed limit and a lane marking. In some examples, the object may include at least one of a pedestrian, an animal, and another vehicle.

[0269] In some aspects, process 1300 may include acquiring one or more images of an interior of the vehicle, where the one or more images show one or more visual landmarks associated with the vehicle, and generating a vehicle template based on the one or more images, where the vehicle template describes the one or more visual landmarks.

[0270] In some aspects, process 1300 can include transmitting a vehicle template to a mobile device and receiving from the mobile device a pose of an occupant relative to one or more coordinates defined in the vehicle template. In some examples, the one or more coordinates are relative to one or more visual landmarks and correspond to a coordinate system of the vehicle.

[0271] In some aspects, process 1300 may include receiving data indicative of a state of an occupant from a mobile device. In some cases, controlling one or more functions of the vehicle may include controlling one or more functions of the vehicle based on the data indicative of an attitude of the occupant and a state of the occupant.

[0272] In some aspects, the process 1300 may include generating an output based at least in part on a pose of the occupant relative to a coordinate system of the vehicle. In some examples, the output may include at least one of an instruction to modify one or more functions of the vehicle and an updated state of the occupant.

[0273] In some examples, one or more functions of the vehicle may be controlled via a computer system of the vehicle configured to control one or more functions of the vehicle and at least one of one or more autonomous vehicle systems of the vehicle.

[0274] 14 is a flow chart illustrating an example process 1300 for interfacing a vehicle with a mobile device associated with an occupant of the vehicle. At block 1402, the process 1400 may include receiving a request from an augmented reality (AR) device connected to a computer system of the vehicle. In some examples, the request may request data generated and / or acquired by the vehicle, such as, for example, data from one or more sensors of the vehicle.

[0275] At block 1404, process 1400 may include transmitting data associated with one or more sensors of the vehicle to the AR device in response to the request. In some examples, the data may include vehicle data indicative of a context of the vehicle. In some examples, the context of the vehicle may include at least one of a state of the vehicle and one or more events encountered by the vehicle or determined to occur during one or more operations of the vehicle.

[0276] In some examples, the one or more events may include at least one of: a presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking. In some cases, the object in the vehicle's path or the threshold proximity to the vehicle's path may include at least one of a pedestrian, an animal, and another vehicle.

[0277] In some aspects, process 1400 may include sending, to the AR device, display data for presentation at an orientation of the AR device relative to a coordinate system of the vehicle and a location relative to at least one of the respective locations of the one or more events. In some examples, the display data may include at least a portion of the vehicle data.

[0278] In some aspects, the process 1400 can include sending an indication of a location of each of the one or more events to the AR device. In some aspects, the process 1400 can include sending a vehicle template to the AR device to determine an attitude of the AR device relative to a coordinate system of the vehicle. In some examples, the vehicle template can describe one or more markers within the interior of the vehicle.

[0279] In some examples, the one or more markers may include at least one of a visual pattern of an area within the interior of the vehicle and / or an object attached to the interior of the vehicle, an element within the interior of the vehicle, a surface within the interior of the vehicle, and an illuminated object inside the vehicle.

[0280] In some aspects, the process 1400 may include receiving, from the AR device, an orientation of the AR device relative to a coordinate system of the vehicle.

[0281] In some aspects, the process 1400 can include controlling the presentation of virtual content associated with the vehicle by the computer system based on at least one of data associated with the one or more sensors and an orientation of the AR device relative to a coordinate system of the vehicle. In some cases, controlling the presentation of the virtual content can include providing a live content feed from one or more image sensors of the vehicle to the AR device based on the orientation of the AR device relative to a coordinate system of the vehicle.

[0282] In some cases, providing the live content feed may include determining an attitude of the AR device relative to the vehicle event based on an attitude of the AR device relative to a coordinate system of the vehicle and a position of the vehicle relative to the vehicle event, and determining that virtual content associated with at least one of the computer system and the AR device at least partially occludes the vehicle event within a field of view of the AR device based on the attitude of the AR device relative to the vehicle event.

[0283] In some examples, controlling the presentation of the virtual content may include associating a location of a virtual content item rendered by the computer system with a location of the vehicle event relative to the vehicle's coordinate system, and filtering or modifying the virtual content item based on the association of the location of the virtual content with the location of the vehicle event relative to the vehicle's coordinate system.

[0284] In some cases, filtering or modifying the virtual content item may include modifying one or more characteristics of the virtual content item. In some examples, the one or more characteristics may include at least one of a transparency, a size, a location of the virtual content item, and a brightness level.

[0285] In some examples, filtering or modifying the virtual content item may include receiving a line of sight of an occupant of the vehicle, determining a visibility of the vehicle event to the occupant based on the line of sight of the occupant and a location of the vehicle event, and filtering or modifying the virtual content item further based on the visibility of the vehicle event to the occupant.

[0286] In some cases, controlling the presentation of the virtual content may include receiving a line of sight of a vehicle occupant and rendering the virtual content within the direction of the line of sight of the vehicle occupant.

[0287] In some examples, rendering the virtual content may include rendering a virtual content overlay comprising a virtual indicator of at least one of the vehicle event, a location of the vehicle event, and a direction of the vehicle event.

[0288] In some cases, rendering the virtual content may include modifying one or more characteristics of the virtual content. In some examples, the one or more characteristics may include at least one of a transparency, a size, a location of the virtual content, and a brightness level.

[0289] In some aspects, the process 1400 may include transmitting instructions to one or more display devices to display virtual content items that provide information regarding the operation of the vehicle. In some cases, the instructions may indicate placement of the virtual content items relative to locations within the vehicle that are within the field of view of the occupant.

[0290] In some examples, the virtual content item may include at least one of a first indication of a vehicle event detected by the computer system, a second indication of a context of the vehicle, and an alert associated with the context of the vehicle.

[0291] In some aspects, the process 1400 may include receiving a camera feed from a camera device on the vehicle and transmitting at least a portion of the camera feed to the AR device for display.

[0292] In some aspects, the process 1400 may include transmitting an indication of the occupant's state to the AR device. In some examples, the occupant's state may include an impairment of the occupant with respect to operating the vehicle. In some examples, the impairment may include a temporary impairment. In some cases, the impairment may include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an impairment location for controlling the vehicle, and an impairment visibility with respect to at least one of the vehicle's operation and an event associated with the vehicle.

[0293] In some embodiments, the process 1400 may include controlling operation of the vehicle based on an occupant impairment. In some examples, an occupant impairment may include any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0294] In some aspects, the process 1400 may include generating display data based on the occupant's impairment. In some examples, the display data may include at least one of vehicle event and vehicle instrumentation data.

[0295] In some aspects, the process 1400 may include transmitting display data to the AR device based on the occupant's impairment. In some examples, the display data may include at least one of a vehicle event and vehicle instrumentation data.

[0296] In some examples, the computer system is configured to control at least one of one or more autonomous functions of the vehicle and one or more autonomous vehicle systems of the vehicle.

[0297] In some examples, the AR device may include a head mounted display. In some examples, the AR device may include a wearable AR device.

[0298] In some examples, any of the processes 1100, 1200, 1300, and / or 1400 may be performed by one or more computing devices or apparatuses. In an exemplary embodiment, any of the processes 1100, 1200, 1300, and / or 1400 may be performed by the mobile device 150 shown in FIG. 1 and / or the vehicle 202 (and / or the vehicle computing system 210) shown in FIG. 2. In some examples, any of the processes 1100, 1200, 1300, and / or 1400 may be performed by one or more computing devices having a computing device architecture 1500 shown in FIG. 15. In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform steps of any of the processes 1100, 1200, 1300, and / or 1400. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data and / or other sensor measurements. For example, the computing device may include a smartphone, a head mounted display, a mobile device, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying the images. In some examples, the one or more sensors and / or camera are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to communicate data.

[0299] The components of a computing device may be implemented in a circuit configuration. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform various operations described herein. The computing device may further include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0300] Processes 1100, 1200, 1300, and 1400 are illustrated as logic flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0301] Further, any of the processes 1100, 1200, 1300, and / or 1400 may be performed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0302] FIG. 15 illustrates an exemplary computing device architecture 1500 of an exemplary computing device that may implement various techniques described herein. For example, the computing device architecture 1500 may implement at least some portions of the computing system 100 illustrated in FIG. 1 or the computing system 210 illustrated in FIG. 2. The components of the computing device architecture 1500 are shown in electrical communication with each other using a connection 1505, such as a bus. The exemplary computing device architecture 1500 includes a processing unit (CPU or processor) 1510 and a computing device connection 1505 that couples various computing device components, including a computing device memory 1515, such as a read only memory (ROM) 1520 and a random access memory (RAM) 1525, to the processor 1510.

[0303] The computing device architecture 1500 may include a cache of high-speed memory that is directly connected to the processor 1510, near the processor 1510, or integrated as part of the processor 1510. The computing device architecture 1500 may copy data from the memory 1515 and / or the storage device 1530 to the cache 1512 for faster access by the processor 1510. In this manner, the cache may provide a performance boost that avoids delays to the processor 1510 while waiting for data. These and other modules may control or be configured to control the processor 1510 to perform various actions. Other computing device memories 1515 may also be available. The memory 1515 may include multiple different types of memories with different performance characteristics. The processor 1510 may include any general-purpose processor, as well as hardware or software services stored in the storage device 1530 and configured to control the processor 1510, as well as special-purpose processors with software instructions built into the processor design. The processor 1510 may be a self-contained system that includes multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0304] To enable user interaction with the computing device architecture 1500, the input device 1545 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output device 1535 may also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, a multimodal computing device may enable a user to provide multiple types of input to communicate with the computing device architecture 1500. The communication interface 1540 may generally govern and manage user input and computing device output. There is no constraint to operate on any particular hardware configuration, and therefore the basic functions herein may be easily replaced with improved hardware or firmware configurations as they are developed.

[0305] The storage device 1530 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, a random access memory (RAM) 1525, a read-only memory (ROM) 1520, and hybrids thereof. The storage device 1530 may include software, code, firmware, etc. for controlling the processor 1510. Other hardware or software modules are contemplated. The storage device 1530 may be connected to the computing device connection 1505. In one aspect, a hardware module that performs a particular function may include software components stored in a computer-readable medium that connects with necessary hardware components, such as the processor 1510, the connection 1505, the output device 1535, etc., to perform the function.

[0306] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or convey instruction(s) and / or data. Computer-readable media may include non-transitory media, which do not include carrier waves and / or transitory electronic signals on which data may be stored and which propagate wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0307] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0308] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks comprising devices, device components, and steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0309] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of steps may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0310] The processes and methods according to the examples described above may be implemented using computer executable instructions stored or otherwise available from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general purpose computer, a special purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, network-attached storage devices, etc.

[0311] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor or processors may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing within a single device, as further examples.

[0312] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0313] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application are described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative embodiments, the methods may be performed in an order different from that described.

[0314] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.

[0315] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0316] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component, and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0317] Claim language or other language in this disclosure reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items within the set of A and B.

[0318] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0319] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code including instructions, which, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0320] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0321] Illustrative examples of the present disclosure include the following:

[0322] Aspect 1. An apparatus comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to determine an attitude of the apparatus relative to a vehicle's coordinate system; receive from the vehicle data associated with one or more sensors of the vehicle; and display, using a display device of the apparatus, virtual content based on the data associated with the one or more sensors and the attitude of the apparatus relative to the vehicle's coordinate system.

[0323] Aspect 2. The apparatus of aspect 1, wherein to determine the attitude of the apparatus, the one or more processors are configured to determine the attitude of the apparatus based on one or more images of the interior of the vehicle.

[0324] Embodiment 3. The apparatus of embodiment 1 or 2, wherein the one or more processors are configured to determine a context of the vehicle based on data associated with the one or more sensors, the context including an event related to the vehicle.

[0325] Aspect 4. The apparatus of aspect 3, wherein the one or more processors are further configured to filter or modify virtual content items that distract a vehicle occupant's attention from vehicle operation or vehicle events or obstruct the occupant's view of a vehicle event or an area outside the vehicle that has been determined to be uncoverable.

[0326] Aspect 5. The device of aspect 3 or 4, wherein the one or more processors are configured to associate a location of the virtual content relative to the device's coordinate system with a location of the event relative to the vehicle's coordinate system, and display a live content feed from one or more visual image sensors of the device based on the association of the location of the virtual content relative to the device's coordinate system with the location of the event relative to the vehicle's coordinate system.

[0327] Aspect 6. The device of aspect 5, wherein to display a live content feed, the one or more processors are configured to determine a device attitude relative to the event based on the device attitude relative to the vehicle's coordinate system and the vehicle attitude relative to the event, and determine that the virtual content at least partially occludes the event in the field of view of a vehicle occupant based on the device attitude relative to the event and an association between the location of the virtual content relative to the device's coordinate system and the location of the event relative to the vehicle's coordinate system.

[0328] Embodiment 7. The apparatus of embodiment 5, wherein the one or more processors are configured to display live content based further on a determination that the virtual content at least partially obstructs the view of the one or more image sensors to the event.

[0329] Aspect 8. An apparatus as described in any of aspects 3 to 7, wherein the one or more processors are configured to associate a location of the virtual content relative to the device's coordinate system with a location of the event relative to the vehicle's coordinate system, and to filter or modify the virtual content item based on the association of the location of the virtual content item relative to the device's coordinate system and the location of the event relative to the vehicle's coordinate system.

[0330] Aspect 9. The apparatus of aspect 8, wherein to filter or modify the virtual content item, the one or more processors are configured to modify one or more characteristics of the virtual content item, the one or more characteristics including at least one of transparency, size, location of the virtual content item, and brightness level of the virtual content item.

[0331] Aspect 10. The device of aspect 8 or 9, wherein to filter or modify the virtual content item, the one or more processors are configured to determine a line of sight of a vehicle occupant, the occupant being associated with the device, determine visibility of the event to the occupant based on the line of sight of the occupant and the location of the event, and filter or modify the virtual content item further based on the visibility of the event to the occupant.

[0332] Aspect 11. The apparatus of aspect 3, wherein the events include at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0333] Aspect 12. The apparatus of aspect 11, wherein the threshold proximity of an object in the path of the vehicle or to the path of the vehicle includes at least one of a pedestrian, an animal, and another vehicle.

[0334] Aspect 13. The device of aspect 3, wherein one or more processors are configured to determine a line of sight of a vehicle occupant, the occupant being associated with the device, and render virtual content within the direction of the line of sight of the vehicle occupant.

[0335] Aspect 14. The apparatus of aspect 13, wherein the one or more processors are configured to render a virtual content overlay comprising a virtual indicator of at least one of an event, a location of the event, and a direction of the event, to render virtual content within a direction of the occupant's line of sight.

[0336] Aspect 15. The device of aspect 13 or 14, wherein to filter or modify the virtual content to render the virtual content within the direction of the occupant's line of sight, the one or more processors are configured to modify one or more characteristics of the virtual content, the one or more characteristics including at least one of transparency, size, location of the virtual content, and brightness level of the virtual content.

[0337] Aspect 16. An apparatus as described in any of aspects 1 to 15, wherein the one or more processors are configured to filter at least a portion of the virtual content based on a vehicle context and an attitude of the apparatus.

[0338] Aspect 17. The apparatus of aspect 16, wherein to filter at least a portion of the virtual content, the one or more processors are configured to enable presentation of a subset of the virtual content, the subset of the virtual content including at least one of an indication of a vehicle status and vehicle instrumentation information.

[0339] Aspect 18. The apparatus of any of aspects 1 to 17, wherein to display the virtual content, the one or more processors are configured to render a virtual content item associated with the vehicle, and the virtual content item is rendered against a surface of the vehicle.

[0340] Aspect 19. The apparatus of aspect 18, wherein the rendered virtual content item includes at least one of a first indication of an event identified in data associated with the one or more sensors, a second indication of a vehicle context, and an alert associated with the vehicle context.

[0341] Aspect 20. The device of aspect 18 or 19, wherein to render the virtual content item, one or more processors are configured to receive a camera feed from a vehicle's camera device and display at least a portion of the camera feed within a display area of ​​the device.

[0342] Aspect 21. An apparatus as described in any of aspects 1 to 20, wherein to determine the attitude of the apparatus, the one or more processors are configured to acquire one or more radio frequency (RF) signals and determine the attitude of the apparatus based on one or more images of the interior of the vehicle and at least one of a round trip time associated with the one or more RF signals, a time of arrival associated with the one or more RF signals, and a received signal strength indicator (RSSI) associated with the one or more RF signals.

[0343] Aspect 22. An apparatus as described in any of aspects 1 to 21, wherein to determine the attitude of the apparatus, the one or more processors are configured to receive from the vehicle a vehicle template including one or more markers associated with the vehicle, and determine the attitude of the apparatus relative to the vehicle's coordinate system based on the one or more images of the interior of the vehicle and the vehicle template.

[0344] Aspect 23. The apparatus of aspect 22, wherein the one or more markers include at least one of visual patterns of areas within the interior of the vehicle and objects mounted within the interior of the vehicle, elements within the interior of the vehicle, surfaces within the interior of the vehicle, and illuminated objects inside the vehicle.

[0345] Aspect 24. The apparatus of aspect 22 or 23, wherein to determine the attitude of the apparatus, one or more processors are configured to detect one or more markers in one or more images, and determine the attitude of the apparatus relative to the vehicle's coordinate system based on the detected one or more markers and the vehicle template.

[0346] Aspect 25. An apparatus as described in any of aspects 1 to 24, wherein the one or more processors are configured to acquire a set of images of an interior of a vehicle using one or more image sensors of the apparatus, the set of images depicting one or more markers associated with the vehicle, and generate a vehicle template including the one or more markers based on the set of images.

[0347] Aspect 26. An apparatus described in any one of aspects 1 to 25, comprising a head-mounted display.

[0348] Embodiment 27. An apparatus described in any of embodiments 1 to 26, further comprising one or more image sensors.

[0349] Aspect 28. A method includes determining an attitude of a mobile device relative to a coordinate system of a vehicle, receiving from the vehicle data associated with one or more sensors of the vehicle, and displaying virtual content using a display device of the mobile device based on the data associated with the one or more sensors and the attitude of the mobile device relative to the coordinate system of the vehicle.

[0350] Aspect 29. The method of aspect 28, wherein determining the attitude of the mobile device includes determining the attitude of the mobile device based on one or more images of the interior of the vehicle.

[0351] Aspect 30. The method of aspect 28 or 29, further comprising determining a context of the vehicle based on data associated with one or more sensors, the context including an event related to the vehicle.

[0352] Aspect 31. The method of aspect 30, further comprising filtering or modifying virtual content items that distract a vehicle occupant's attention from vehicle operation or vehicle events or that obstruct the occupant's view of a vehicle event or an area outside the vehicle that has been determined to be uncoverable.

[0353] Aspect 32. The method of aspect 30 or 31, further comprising: associating a location of the virtual content relative to the coordinate system of the mobile device with a location of the event relative to the coordinate system of the vehicle; and displaying a live content feed from one or more visual image sensors of the mobile device based on the association of the location of the virtual content relative to the coordinate system of the mobile device and the location of the event relative to the coordinate system of the vehicle.

[0354] Aspect 33. The method of aspect 32, wherein displaying the live content feed includes determining an attitude of the mobile device relative to the event based on an attitude of the mobile device relative to the vehicle's coordinate system and an attitude of the vehicle relative to the event, and determining that the virtual content at least partially occludes the event in the field of view of an occupant of the vehicle based on the attitude of the mobile device relative to the event and an association between a location of the virtual content relative to the mobile device's coordinate system and a location of the event relative to the vehicle's coordinate system.

[0355] Aspect 34. The method of aspect 32 or 33, further comprising displaying the live content based further on a determination that the virtual content at least partially obstructs a view of one or more image sensors of the mobile device to the event.

[0356] Aspect 35. The method of aspect 30, further comprising associating a location of the virtual content item relative to the coordinate system of the mobile device with a location of the event relative to the coordinate system of the vehicle, and filtering or modifying the virtual content item based on the association of the location of the virtual content relative to the coordinate system of the mobile device and the location of the event relative to the coordinate system of the vehicle.

[0357] Aspect 36. The method of aspect 35, wherein to filter or modify the virtual content item, the one or more processors are configured to modify one or more characteristics of the virtual content item, the one or more characteristics including at least one of transparency, size, location of the virtual content item, and brightness level of the virtual content item.

[0358] Aspect 37. The method of aspect 35 or 36, wherein to filter or modify the virtual content item, the one or more processors are configured to determine a line of sight of an occupant of the vehicle, the occupant being associated with a mobile device, determine visibility of the event to the occupant based on the occupant's line of sight and the location of the event, and filter or modify the virtual content item further based on the visibility of the event to the occupant.

[0359] Aspect 38. The method of aspect 30, wherein the event includes at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0360] Aspect 39. The method of aspect 38, wherein the object in the vehicle's path or the threshold proximity to the vehicle's path includes at least one of a pedestrian, an animal, and another vehicle.

[0361] Aspect 40. The method of aspect 30, further comprising: determining a line of sight of an occupant of the vehicle, the occupant being associated with a mobile device; and rendering the virtual content within a direction of the line of sight of the occupant of the vehicle.

[0362] Aspect 41. The method of aspect 40, wherein rendering virtual content within the direction of the occupant's line of sight includes rendering a virtual content overlay comprising a virtual indicator of at least one of the event, the location of the event, and the direction of the event.

[0363] Aspect 42. The method of aspect 40 or 41, wherein to filter or modify the virtual content to render the virtual content within the direction of the occupant's line of sight, the one or more processors are configured to modify one or more characteristics of the virtual content, the one or more characteristics including at least one of transparency, size, location of the virtual content, and brightness level of the virtual content.

[0364] Aspect 43. The method of any of aspects 28 to 42, further comprising filtering at least a portion of the virtual content based on a vehicle context and a mobile device pose.

[0365] Aspect 44. The method of aspect 43, wherein filtering at least a portion of the virtual content includes enabling presentation of a subset of the virtual content, the subset of the virtual content including at least one of an indication of a vehicle status and vehicle instrumentation information.

[0366] Aspect 45. The method of any of aspects 28 to 44, wherein displaying the virtual content includes rendering a virtual content item associated with the vehicle, the virtual content item being rendered against a surface of the vehicle.

[0367] Aspect 46. The method of aspect 45, wherein the rendered virtual content item includes at least one of a first indication of an event identified in data associated with the one or more sensors, a second indication of a vehicle context, and an alert associated with the vehicle context.

[0368] Aspect 47. The method of aspect 45 or 46, wherein rendering the virtual content item includes receiving a camera feed from a camera device of the vehicle and displaying at least a portion of the camera feed within a display area of ​​the mobile device.

[0369] Aspect 48. The method of any of aspects 28 to 47, wherein determining the attitude of the mobile device includes acquiring one or more radio frequency (RF) signals and determining the attitude of the mobile device based on one or more images of the interior of the vehicle and at least one of a round trip time associated with the one or more RF signals, a time of arrival associated with the one or more RF signals, and a received signal strength indicator (RSSI) associated with the one or more RF signals.

[0370] Aspect 49. The method of any of aspects 28 to 48, wherein determining the pose of the mobile device includes receiving, from the vehicle, a vehicle template including one or more markers associated with the vehicle, and determining the pose of the mobile device relative to a coordinate system of the vehicle based on the one or more images of the interior of the vehicle and the vehicle template.

[0371] Aspect 50. The method of aspect 49, wherein the one or more markers include at least one of visual patterns of areas within the interior of the vehicle and objects mounted within the interior of the vehicle, elements within the interior of the vehicle, surfaces within the interior of the vehicle, and illuminated objects inside the vehicle.

[0372] Aspect 51. The method of aspect 49 or 50, wherein determining the attitude of the mobile device includes detecting one or more markers in one or more images, and determining the attitude of the mobile device relative to the vehicle's coordinate system based on the detected one or more markers and the vehicle template.

[0373] Aspect 52. A method as described in any of aspects 28 to 51, further comprising: acquiring a set of images of an interior of a vehicle using one or more image sensors of a mobile device, the set of images depicting one or more markers associated with the vehicle; and generating a vehicle template including the one or more markers based on the set of images.

[0374] Aspect 53. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 28 to 52.

[0375] Example 54. An apparatus comprising means for performing the method according to any of examples 28 to 52.

[0376] Aspect 55. An apparatus comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to: determine an attitude of the apparatus relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle; determine a state of an occupant of the vehicle; and transmit data to the vehicle indicative of the state of the occupant and the attitude of the apparatus relative to the coordinate system of the vehicle.

[0377] Aspect 56. The apparatus of aspect 55, wherein the occupant status includes an occupant impairment with respect to operating the vehicle.

[0378] Aspect 57. The apparatus of aspect 56, wherein the occupant impairment includes any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0379] Aspect 58. The apparatus of aspect 56 or 57, wherein the obstacles include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an obstacle location for controlling the vehicle, and an obstacle visibility with respect to at least one of the operation of the vehicle and events associated with the vehicle.

[0380] Aspect 59. An apparatus as described in any of aspects 56 to 58, wherein to determine an occupant state, the one or more processors are configured to receive from the vehicle data associated with one or more sensors of the vehicle, and determine the occupant state based on the data associated with the one or more sensors of the vehicle and the attitude of the apparatus.

[0381] Aspect 60. The apparatus of aspect 59, wherein the data associated with one or more sensors of the vehicle indicates at least one of a state of the vehicle and an event associated with the vehicle.

[0382] Aspect 61. The apparatus of aspect 60, wherein the events associated with the vehicle include at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0383] Aspect 62. The apparatus of aspect 61, wherein the threshold proximity of an object in the vehicle's path or to the vehicle's path includes at least one of a pedestrian, an animal, and another vehicle.

[0384] Aspect 63. The apparatus of any of aspects 56 to 62, wherein the occupant impairment includes any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0385] Aspect 64. An apparatus as described in any of aspects 55 to 63, wherein the one or more processors are configured to determine a line of sight of an occupant of the vehicle, the occupant state including the occupant's line of sight, and the occupant is associated with the apparatus.

[0386] Aspect 65. An apparatus as described in any of aspects 55 to 64, wherein to determine the state of the occupant, the one or more processors are configured to receive one or more health measurements associated with the occupant from one or more sensors associated with at least one of the apparatus and a wearable device worn by the occupant, and determine the state of the occupant based on the one or more health measurements.

[0387] Aspect 66. The device of aspect 65, wherein the one or more health measurements include at least one of heart rate, blood pressure, body temperature, galvanic skin response, a measurement of electrical signals from the occupant's heart, a measurement of electrical activity in the occupant's brain, the amount of redness in the eyes, and pupil size.

[0388] Aspect 67. The apparatus of any of aspects 55 to 66, wherein to determine the occupant's state, the one or more processors are configured to determine that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time, and determine an impairment state of the occupant based on a determination that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time and that the period of time exceeds a threshold period of time.

[0389] Aspect 68. The device described in aspect 67, wherein to determine that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time, the one or more processors are configured to determine that the occupant's gaze is focused on virtual content rendered by the device for at least a portion of the period of time.

[0390] Aspect 69. The apparatus of aspect 67 or 68, wherein to determine that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time, the one or more processors are configured to determine that the occupant's gaze is focused in a direction other than the direction of an obstacle in the vehicle's path or a threshold proximity to the vehicle's path.

[0391] Embodiment 70. The apparatus of any of embodiments 55 to 69, wherein the one or more processors are configured to transmit an indication of the occupant's status to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0392] Aspect 71. A device described in any of aspects 55 to 70, wherein to determine the state of an occupant, the one or more processors are configured to determine the line of sight of an occupant wearing the device, and determine the state of the occupant based on the attitude of the device and the line of sight of the occupant.

[0393] Aspect 72. The apparatus of any of aspects 55 to 71, wherein to determine the attitude of the apparatus, the one or more processors are configured to receive from the vehicle a vehicle template including one or more markers associated with the vehicle, and determine the attitude of the apparatus with respect to the vehicle's coordinate system based on the one or more images and the vehicle template.

[0394] Aspect 73. The apparatus of aspect 72, wherein the one or more markers include at least one of visual patterns of areas within the interior of the vehicle and objects mounted within the interior of the vehicle, elements within the interior of the vehicle, surfaces within the interior of the vehicle, and illuminated objects inside the vehicle.

[0395] Aspect 74. The apparatus of aspect 72, wherein the one or more images depict one or more markers, and to determine the attitude of the device, the one or more processors are configured to detect the one or more markers in the one or more images and determine the attitude of the device relative to the vehicle's coordinate system based on the detected one or more markers and a vehicle template.

[0396] Aspect 75. An apparatus as described in any of aspects 55 to 74, wherein the one or more processors are configured to acquire a set of images of an interior of the vehicle using one or more image sensors of the apparatus, the set of images depicting one or more visual landmarks associated with the vehicle, and generate a vehicle template based on the set of images, the vehicle template including the one or more visual landmarks.

[0397] Aspect 76. An apparatus as described in any of aspects 55 to 75, wherein to determine the attitude of the apparatus, the one or more processors are configured to acquire inertial sensor data associated with the apparatus and determine the attitude of the apparatus based on the one or more images and the inertial sensor data.

[0398] Embodiment 77. An apparatus described in any of embodiments 55 to 76, comprising a head-mounted display.

[0399] Aspect 78. A method, comprising: determining an attitude of a mobile device relative to a vehicle coordinate system based on one or more images of an interior of the vehicle; determining a state of an occupant of the vehicle; and transmitting data to the vehicle indicative of the state of the occupant and the attitude of the mobile device relative to the vehicle coordinate system.

[0400] Aspect 79. The method of aspect 78, wherein the occupant condition includes an occupant impairment with respect to operating the vehicle.

[0401] Aspect 80. The method of any of aspects 78 or 79, wherein the occupant impairment includes any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0402] Aspect 81. The method of any of aspects 78 to 80, wherein the obstacles include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an obstacle location for controlling the vehicle, and an obstacle visibility with respect to at least one of the vehicle operation and events associated with the vehicle.

[0403] Aspect 82. The method of any of aspects 78 to 81, wherein in some examples, determining the occupant state includes receiving, from the vehicle, data associated with one or more sensors of the vehicle, and determining the occupant state based on the data associated with the one or more sensors of the vehicle and an attitude of the mobile device.

[0404] Aspect 83. The method of aspect 82, wherein the data associated with one or more sensors of the vehicle indicates at least one of a state of the vehicle and an event associated with the vehicle.

[0405] Aspect 84. The method of aspect 83, wherein the event associated with the vehicle includes at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0406] Aspect 85. The method of aspect 84, wherein the object in the vehicle's path or the threshold proximity to the vehicle's path includes at least one of a pedestrian, an animal, and another vehicle.

[0407] Aspect 86. The method of any of aspects 78 to 85, further comprising determining a line of sight of an occupant of the vehicle, wherein the occupant state includes the occupant's line of sight, and the occupant is associated with a mobile device.

[0408] Aspect 87. A method as described in any of aspects 78 to 86, wherein determining the occupant's state includes receiving one or more health measurements associated with the occupant from one or more sensors associated with at least one of the mobile device and a wearable device worn by the occupant, and determining the occupant's state based on the one or more health measurements.

[0409] Aspect 88. The method of aspect 87, wherein the one or more health measurements include at least one of heart rate, blood pressure, body temperature, galvanic skin response, a measurement of electrical signals from the occupant's heart, a measurement of electrical activity in the occupant's brain, the amount of redness in the eyes, and pupil size.

[0410] Aspect 89. The method of any of aspects 78 to 87, wherein determining the occupant's state includes determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time, and determining an impairment state of the occupant based on a determination that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time and that the period of time exceeds a threshold period of time.

[0411] Aspect 90. The method of aspect 89, wherein determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time includes determining that the occupant's gaze is focused on virtual content rendered by the mobile device for at least a portion of the period of time.

[0412] Aspect 91. The method of aspect 89, wherein determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time includes determining that the occupant's gaze is focused in a direction other than the direction of an obstacle in the vehicle's path or a threshold proximity to the vehicle's path.

[0413] Aspect 92. The method of any of aspects 78 to 91, further comprising transmitting an indication of the occupant's status to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0414] Aspect 93. A method as described in any of aspects 78 to 92, wherein determining the state of the occupant includes determining a line of sight of an occupant wearing the mobile device, and determining the state of the occupant based on the attitude of the mobile device and the line of sight of the occupant.

[0415] Aspect 94. The method of any of aspects 78 to 93, wherein determining the attitude of the mobile device includes receiving, from the vehicle, a vehicle template including one or more markers associated with the vehicle, and determining an attitude of the mobile device relative to a coordinate system of the vehicle based on the one or more images and the vehicle template.

[0416] Aspect 95. The method of aspect 94, wherein the one or more markers include at least one of visual patterns of areas within the interior of the vehicle and objects mounted within the interior of the vehicle, elements within the interior of the vehicle, surfaces within the interior of the vehicle, and illuminated objects inside the vehicle.

[0417] Aspect 96. The method of aspect 94, wherein the one or more images depict one or more markers, and determining the attitude of the mobile device includes detecting the one or more markers in the one or more images, and determining the attitude of the mobile device relative to a coordinate system of the vehicle based on the detected one or more markers and the vehicle template.

[0418] Aspect 97. A method as described in any of aspects 78 to 96, further comprising: acquiring a set of images of an interior of a vehicle using one or more image sensors of a mobile device, the set of images depicting one or more visual landmarks associated with the vehicle; and generating a vehicle template based on the set of images, the vehicle template including the one or more visual landmarks.

[0419] Aspect 98. The method of any of aspects 78 to 97, wherein determining the attitude of the mobile device includes acquiring inertial sensor data associated with the mobile device and determining the attitude of the mobile device based on the one or more images and the inertial sensor data.

[0420] Aspect 99. The method of any of aspects 78 or 98, wherein the occupant impairment includes any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0421] Aspect 100. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 78-99.

[0422] Example 101. An apparatus comprising means for performing the method according to any of examples 78 to 99.

[0423] Aspect 102. A method for controlling one or more operations of a vehicle, the method including: receiving, from a mobile device associated with an occupant of the vehicle, an attitude of the occupant relative to a coordinate system of the vehicle; and controlling one or more functions of the vehicle based on the attitude of the occupant relative to a coordinate system of the vehicle.

[0424] Aspect 103. The method of aspect 102, wherein controlling one or more functions of the vehicle includes participating in one or more vehicle functions of the vehicle.

[0425] Aspect 104. The method of aspect 103, wherein the one or more vehicle functions include at least one of an autopilot function, a traction control function, a cruise control function, a collision avoidance function, a lane departure function, a lane centering function, a brake assist function, a lane keeping function, a highway assist function, a lane change assist function, a speed adaptation function, and an intersection assist function.

[0426] Aspect 105. The method of any of aspects 102 to 104, wherein controlling the operation of the vehicle includes controlling or engaging one or more autonomous vehicle systems of the vehicle.

[0427] Aspect 106. The method of aspect 105, wherein the one or more autonomous vehicle systems include at least one of a blind spot monitoring system, a driver monitoring system, a braking system, an autonomous driving control system, a driver assistance system, a navigation system, a steering control system, a vehicle communication system, and an automotive head-up display.

[0428] Aspect 107. The method of any of aspects 102 to 106, further comprising transmitting data associated with one or more sensors of the vehicle to at least one of the mobile device, the second vehicle, the vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0429] Aspect 108. The method of any of aspects 102 to 107, further comprising transmitting data indicative of the occupant's status to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with the pedestrian.

[0430] Aspect 109. The method of any of aspects 102 to 108, further comprising transmitting a vehicle template to the mobile device for determining the occupant's posture relative to the vehicle's coordinate system, the vehicle template including one or more markers associated with the vehicle.

[0431] Aspect 110. The method of aspect 109, wherein the one or more markers include at least one of visual patterns of areas within the interior of the vehicle and objects attached to the interior of the vehicle, elements of the interior of the vehicle, surfaces within the interior of the vehicle, and illuminated objects inside the vehicle.

[0432] Aspect 111. The method of aspect 109, generating an output based at least in part on an attitude of the occupant relative to a coordinate system of the vehicle, the output further including at least one of a communication to a mobile device, an instruction to modify one or more functions of the vehicle, and an indication of an occupant's state.

[0433] Aspect 112. The method of aspect 111, wherein the communication to the mobile device includes at least one of a virtual content item and a request to display the virtual content item.

[0434] Aspect 113. The method of any of aspects 102 to 112, further comprising receiving data from the mobile device indicative of a status of an occupant of the vehicle, the data comprising at least one of sensor data from one or more sensors of the mobile device and processed data generated based on the sensor data from the one or more sensors of the mobile device.

[0435] Aspect 114. The method of aspect 113, wherein the processed data includes at least one of a description of the occupant's state and a classification output identifying the occupant's state.

[0436] Aspect 115. The method of any of aspects 102 to 114, further comprising receiving data from the mobile device indicative of a state of an occupant of the vehicle, the data comprising at least one of an indication of the occupant's gaze, an attitude of the mobile device relative to a coordinate system of the vehicle, and one or more health measurements associated with the occupant.

[0437] Aspect 116. The method of aspect 115, wherein the one or more health measurements include at least one of heart rate, blood pressure, body temperature, galvanic skin response, a measurement of electrical signals from the occupant's heart, a measurement of electrical activity in the occupant's brain, the amount of redness in the eyes, and pupil size.

[0438] Aspect 117. The method of any of aspects 102 to 116, further comprising: acquiring sensor data from one or more sensors of the vehicle, the sensor data including at least one of an indication of an event related to operation of the vehicle, an indication of one or more driving patterns during one or more operations of the vehicle that are at least partially controlled by an occupant, and vehicle instrumentation data; and controlling one or more functions further based on the sensor data.

[0439] Aspect 118. The method of aspect 117, further including determining a state of an occupant, wherein determining the state of the occupant includes receiving one or more health measurements associated with the occupant from one or more health sensors associated with at least one of the mobile device and the wearable device, and determining the state of the occupant based on the one or more health measurements associated with the occupant.

[0440] Aspect 119. The method of any of aspects 102 to 118, further comprising obtaining an occupant status, the occupant status including an occupant impairment with respect to operating the vehicle.

[0441]

[0046] Aspect 120. The method of aspect 119, wherein the occupant impairment comprises an occupant impairment. In some examples, the impairment can include a temporary impairment.

[0442] Aspect 121. The method of aspect 119, wherein the obstacles include at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an obstacle location for controlling the vehicle, and an obstacle visibility with respect to at least one of the vehicle operation and events associated with the vehicle.

[0443] Aspect 122. The method of aspect 121, wherein the event includes at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0444] Aspect 123. The method of aspect 122, wherein the object includes at least one of a pedestrian, an animal, and another vehicle.

[0445] Aspect 124. A method as described in any of aspects 102 to 123, further comprising: acquiring one or more images of an interior of a vehicle, the one or more images depicting one or more visual landmarks associated with the vehicle; and generating a vehicle template based on the one or more images, the vehicle template describing the one or more visual landmarks.

[0446] Aspect 125. The method of aspect 124, further comprising: transmitting a vehicle template to a mobile device; and receiving from the mobile device an occupant's pose relative to one or more coordinates defined in the vehicle template, the one or more coordinates being relative to one or more visual landmarks and corresponding to a coordinate system of the vehicle.

[0447] Aspect 126. The method of aspect 125, further comprising receiving data indicative of an occupant state from the mobile device, and controlling one or more functions of the vehicle comprises controlling one or more functions of the vehicle based on the data indicative of the occupant's posture and the occupant's state.

[0448] Aspect 127. The method of aspect 126, generating an output based at least in part on an attitude of an occupant relative to a coordinate system of the vehicle, the output including at least one of instructions to modify one or more functions of the vehicle and an updated state of the occupant.

[0449] Aspect 128. A method according to any of aspects 102 to 128, wherein the one or more functions of the vehicle are controlled via a computer system of the vehicle configured to control the one or more functions of the vehicle and at least one of the one or more autonomous vehicle systems of the vehicle.

[0450] Example 129. The method of any of examples 102 to 128, wherein the mobile device includes a wearable augmented reality device.

[0451]

[0046] Aspect 130. The method of any of aspects 102 to 129, wherein the mobile device comprises a head-mounted display.

[0452] Aspect 131. The method of any of aspects 119 to 121, wherein the occupant impairment includes any event, activity, distraction, state, attribute, behavior, and / or condition that adversely affects the occupant's ability to safely operate the vehicle.

[0453] Aspect 132. An apparatus comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors are configured to perform the method of any of aspects 102 to 131.

[0454] Example 133. An apparatus comprising means for performing the method according to any of examples 102 to 131.

[0455] Aspect 134. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any of aspects 102 to 131.

[0456] Aspect 135. A method for providing vehicle data to a device associated with an occupant of a vehicle, the method including receiving a request from an augmented reality (AR) device connected to a computer system of the vehicle, and in response to the request, transmitting data associated with one or more sensors of the vehicle to the AR device.

[0457] Aspect 136. The method of aspect 135, wherein the data includes vehicle data indicative of a context of the vehicle, the context of the vehicle including at least one of a state of the vehicle and one or more events encountered by the vehicle or determined to occur during one or more operations of the vehicle.

[0458] Aspect 137. The method of aspect 136, wherein the one or more events include at least one of the presence of an object in the vehicle's path or a threshold proximity to the vehicle's path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking.

[0459] Aspect 138. The method of aspect 137, wherein the threshold proximity of an object in the vehicle's path or to the vehicle's path includes at least one of a pedestrian, an animal, and another vehicle.

[0460] Aspect 139. The method of aspect 136, further comprising transmitting to the AR device display data for presentation at a location relative to an attitude of the AR device relative to the vehicle's coordinate system and at least one of the respective locations of the one or more events, wherein the display data includes at least a portion of the vehicle data.

[0461] Aspect 140. The method of aspect 139, further comprising transmitting an indication of a location of each of the one or more events to the AR device.

[0462] Aspect 141. A method according to any o...

Claims

1. A head-mounted display (HMD) for use in a vehicle, comprising: Memory and one or more processors coupled to the memory, wherein the one or more processors: determining a pose of the HMD relative to a coordinate system of the vehicle based on one or more images of the interior of the vehicle; Displaying virtual content using a display device of the HMD; determining a state of an occupant of the vehicle wearing the HMD; transmitting data to the vehicle indicative of the state of the occupant and the attitude of the HMD relative to the coordinate system of the vehicle; receiving data from the vehicle indicative of a change in one or more autonomous functions of the vehicle in response to the state of the occupant; Displaying updated virtual content including information regarding the change in one or more of the autonomous functions of the vehicle using the display device of the HMD. An HMD configured as follows.

2. The change in one or more of the autonomous functions of the vehicle comprises: activating an autonomous function of the vehicle; increasing the level of confidence in an autonomous function already operating in said vehicle; activating an autonomous function of the vehicle to stop the vehicle; activating an autonomous function of the vehicle to navigate the vehicle to a predefined location; activating an autonomous function of the vehicle to reduce the speed of the vehicle; including one or more of the following: The HMD according to claim 1 .

3. 2. The HMD of claim 1, wherein the state of the occupant includes an impairment of the occupant related to operating the vehicle, and optionally the impairment includes at least one of a distraction state, an intoxication state, a health state, an alertness state, a detected emotional state, an impairment location for controlling the vehicle, and an impaired visibility related to at least one of the operation of the vehicle and an event associated with the vehicle.

4. To determine the state of the occupant, the one or more processors: receiving, from the vehicle, data associated with one or more sensors of the vehicle; determining the state of the occupant based on the data associated with the one or more sensors of the vehicle and the pose of the HMD; The HMD according to claim 3 , configured as follows:

5. The HMD of claim 4 , wherein the data associated with the one or more sensors of the vehicle indicates at least one of a state of the vehicle and an event associated with the vehicle.

6. 6. The HMD of claim 5, wherein the event associated with the vehicle includes at least one of the presence of an object in the vehicle's path or a threshold proximity of the vehicle to the path, a traffic regulation associated with the vehicle's path, and the vehicle's failure to stay within at least one of a speed limit and a lane marking, and optionally, the object in the vehicle's path or the threshold proximity to the vehicle's path includes at least one of a pedestrian, an animal, and another vehicle.

7. the one or more processors: The HMD of claim 1 configured to determine a line of sight of an occupant of the vehicle, wherein the state of the occupant includes the line of sight of the occupant.

8. To determine the state of the occupant, the one or more processors: receiving one or more health measurements associated with the occupant from one or more sensors associated with the HMD; Determining the condition of the occupant based on the one or more health measures.

2. The HMD of claim 1, configured to:

9. To determine the state of the occupant, the one or more processors: determining that the occupant's gaze is focused away from the road ahead of the vehicle for a period of time; determining an impairment state of the occupant based on the occupant's gaze being focused away from the road ahead of the vehicle for the period of time and a determination that the period of time exceeds a threshold period of time; The HMD according to claim 1 , configured as follows:

10. To determine that the occupant's gaze is focused away from the road ahead of the vehicle for the period of time, the one or more processors: configured to determine that the occupant's line of sight is focused on the virtual content for at least a portion of the period of time; or 10. The HMD of claim 9, configured to determine that the line of sight of the occupant is centered in a direction other than a direction of an obstacle within the path of the vehicle or within a threshold proximity to the path of the vehicle.

11. the one or more processors: configured to transmit the indication of the condition of the occupant to at least one of a second vehicle, a vehicle infrastructure system, a first remote device associated with a second occupant of the second vehicle, and a second remote device associated with a pedestrian; or To determine the state of the occupant, the one or more processors: determining a line of sight of the occupant wearing the HMD; The state of the occupant is determined based on the posture of the HMD and the line of sight of the occupant. The HMD according to claim 1 , configured as follows:

12. To determine the pose of the HMD, the one or more processors: receiving a vehicle template from the vehicle, the vehicle template including one or more markers associated with the vehicle; determining the pose of the HMD relative to the coordinate system of the vehicle based on the one or more images and the vehicle template; The HMD according to claim 1 , configured as follows:

13. the one or more markers include at least one of a visual pattern of an area within the interior of the vehicle and / or an object attached to the interior of the vehicle, an element of the interior of the vehicle, a surface within the interior of the vehicle, and an illuminated object inside the vehicle; or The one or more images depict the one or more markers, and to determine the pose of the HMD, the one or more processors: Detecting the one or more markers in the one or more images; determining the pose of the HMD relative to the coordinate system of the vehicle based on the detected one or more markers and the vehicle template; The HMD according to claim 12, configured as follows:

14. the one or more processors: acquiring a set of images of the interior of the vehicle using one or more image sensors of the HMD, the set of images depicting one or more visual landmarks associated with the vehicle; generating a vehicle template based on the set of images, the vehicle template including the one or more visual landmarks; or To determine the pose of the HMD, the one or more processors: acquiring inertial sensor data associated with the HMD; determining the pose of the HMD based on the one or more images and the inertial sensor data; The HMD according to claim 1 , configured as follows:

15. 1. A method comprising: determining a pose of a head mounted display (HMD) worn by an occupant of the vehicle relative to a coordinate system of the vehicle based on one or more images of an interior of the vehicle; displaying virtual content using a display device of the HMD; determining a state of the occupant of the vehicle; transmitting data to the vehicle indicative of the state of the occupant and the attitude of the HMD relative to the coordinate system of the vehicle; receiving data from the vehicle indicative of a change in one or more autonomous functions of the vehicle in response to the state of the occupant; displaying updated virtual content using the display device of the HMD, the updated virtual content including information regarding the change in one or more of the autonomous functions of the vehicle; A method comprising:

Images