Systems and methods for augmented reality visualization based on sensor data

Abstract

The present application relates to systems and methods for augmented reality visualization based on sensor data. A device is configured to estimate a gaze target of a user of an augmented reality headset based on vehicle sensor data and headset sensor data. The device is configured to generate visualization data based on the gaze target. In response to determining that the gaze target is inside a first vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the first vehicle. The first point of interest includes at least a portion of a particular route of a particular vehicle. The particular vehicle includes the first vehicle or a second vehicle. In response to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the first vehicle.

Description

Technical Field

[0001] This disclosure generally relates to augmented reality visualization. Background Technology

[0002] Augmented reality (AR) is rapidly transforming how people interact with computer systems and their environments. The technology is expected to have a wide-ranging impact on aerospace and defense. Crew members—such as those on commercial aircraft, military aircraft, ships, and vehicles—typically maintain situational awareness between two distinct contexts: information presented primarily on a fixed, two-dimensional computer monitor, and the three-dimensional external environment. Crew members transition between these contexts by redirecting their gaze or by physically moving between consoles and windows. For example, a mental shift also occurs when a crew member attempts to map between two-dimensional graphics and three-dimensional terrain. Size and weight constraints may limit the number of fixed monitors, so some crew members may not be able to access information as needed or when required. Summary of the Invention

[0003] In one particular embodiment, a device for augmented reality visualization includes an interface and one or more processors. The interface is configured to receive vehicle sensor data from one or more vehicle sensors coupled to a first vehicle. The interface is also configured to receive headphone sensor data from one or more headphone sensors coupled to an augmented reality headset. The one or more processors are configured to determine the orientation and position of the augmented reality headset relative to the first vehicle based on the vehicle sensor data and the headphone sensor data. The one or more processors are further configured to estimate a user's gaze target for the augmented reality headset based on the headphone sensor data and the orientation and position of the augmented reality headset relative to the first vehicle. The one or more processors are further configured to generate visualization data based on the gaze target. In response to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest outside the first vehicle. The first point of interest includes at least a portion of a particular route of the particular vehicle. The particular vehicle includes either the first vehicle or a second vehicle. In response to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest inside the first vehicle. The one or more processors are further configured to send the visualization data to a display of the augmented reality headset.

[0004] In another specific embodiment, an augmented reality visualization method includes receiving vehicle sensor data at a device from one or more vehicle sensors coupled to a first vehicle. The method also includes receiving headphone sensor data at the device from one or more headphone sensors coupled to an augmented reality headset. The method further includes determining the orientation and position of the augmented reality headset relative to the first vehicle based on the vehicle sensor data and the headphone sensor data. The method also includes estimating a gaze target of a user of the augmented reality headset at the device based on the headphone sensor data and the orientation and position of the augmented reality headset relative to the first vehicle. The method further includes generating visualization data at the device based on the gaze target. In response to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest outside the first vehicle. The first point of interest includes at least a portion of a specific route of the specific vehicle. The specific vehicle includes either the first vehicle or a second vehicle. In response to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest inside the first vehicle. The method also includes transmitting the visualization data from the device to a display of the augmented reality headset.

[0005] In another specific embodiment, a computer-readable storage device stores instructions that, when executed by one or more processors, cause one or more processors to receive vehicle sensor data from one or more vehicle sensors coupled to a first vehicle. When executed by one or more processors, these instructions also cause one or more processors to receive headphone sensor data from one or more headphone sensors coupled to an augmented reality headset. When executed by one or more processors, these instructions further cause one or more processors to determine the orientation and position of the augmented reality headset relative to the first vehicle based on the vehicle sensor data and the headphone sensor data. When executed by one or more processors, these instructions further cause one or more processors to estimate a gaze target of a user of the augmented reality headset based on the headphone sensor data and the orientation and position of the augmented reality headset relative to the first vehicle. When executed by one or more processors, these instructions further cause one or more processors to generate visualization data based on the gaze target. In response to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest outside the first vehicle. The first point of interest includes at least a portion of a specific route of a particular vehicle. The particular vehicle includes either the first vehicle or a second vehicle. In response to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest inside the first vehicle. When executed by one or more processors, these instructions also cause one or more processors to send visualization data to the display of the augmented reality headset.

[0006] In another specific embodiment, a device for augmented reality visualization includes an interface and one or more processors. The interface is configured to receive vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The interface is also configured to receive headphone sensor data from one or more headphone sensors coupled to an augmented reality headset. The one or more processors are configured to determine movement of the vehicle based on the vehicle sensor data. The one or more processors are also configured to determine movement of the augmented reality headset based on the headphone sensor data. The one or more processors are further configured to estimate a user portion of the movement of the augmented reality headset based on a comparison of the vehicle movement and the movement of the augmented reality headset, the user portion being caused by movement of the user's head, rather than by the vehicle movement. The one or more processors are further configured to determine the orientation and position of the augmented reality headset relative to the vehicle based on the user portion of the augmented reality headset movement. The one or more processors are further configured to estimate a user's gaze target based on the headphone sensor data and the orientation and position of the augmented reality headset relative to the vehicle. The one or more processors are further configured to generate visualization data based on the gaze target. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle. One or more processors are also configured to send the visualization data to the display of the augmented reality headset.

[0007] In another specific embodiment, an augmented reality visualization method includes receiving vehicle sensor data at a device from one or more vehicle sensors coupled to a vehicle. The method also includes receiving headphone sensor data at the device from one or more headphone sensors coupled to an augmented reality headset. The method further includes determining the orientation and position of the augmented reality headset relative to the vehicle based on the vehicle sensor data and the headphone sensor data. The method also includes estimating a gaze target of a user of the augmented reality headset based at least in part on the orientation and position of the augmented reality headset relative to the vehicle. The method further includes generating visualization data at the device based on the gaze target. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle. The method also includes transmitting the visualization data from the device to a display of the augmented reality headset.

[0008] In another specific embodiment, a computer-readable storage device stores instructions that, when executed by a processor, cause the processor to perform operations including receiving vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The operations also include receiving headphone sensor data from one or more headphone sensors coupled to an augmented reality headset. The operations further include estimating a gaze target of a user of the augmented reality headset based on the vehicle sensor data and the headphone sensor data. The operations also include generating visualization data based on the gaze target. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle. The operations also include sending the visualization data to a display of the augmented reality headset.

[0009] The features, functions and advantages described herein can be implemented independently in various embodiments or in combination in other embodiments, and further details can be found in the following description and figures. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating a system operable for performing augmented reality visualizations;

[0011] Figure 2A This is a diagram illustrating an example of augmented reality visualization.

[0012] Figure 2B This is a diagram illustrating another example of augmented reality visualization;

[0013] Figure 3A This is a diagram illustrating another example of augmented reality visualization;

[0014] Figure 3B This is a diagram illustrating another example of augmented reality visualization;

[0015] Figure 4A This is a diagram illustrating another example of augmented reality visualization;

[0016] Figure 4B This is a diagram illustrating another example of augmented reality visualization;

[0017] Figure 5A This is a diagram illustrating another example of augmented reality visualization;

[0018] Figure 5B This is a diagram illustrating another example of augmented reality visualization;

[0019] Figure 6 This is a diagram illustrating an example of another system operable to perform augmented reality visualizations;

[0020] Figure 7 This is a flowchart illustrating an example of a method for augmented reality visualization;

[0021] Figure 8 This is a flowchart illustrating another example of a method for augmented reality visualization; and

[0022] Figure 9 It is a block diagram depicting a computing environment including a computing device configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. Detailed Implementation

[0023] The embodiments described herein are for augmented reality visualization. In a particular example, a user in a vehicle wears an augmented reality (AR) headset. A device receives headset sensor data from one or more headset sensors coupled to the AR headset. The device also receives vehicle sensor data from one or more vehicle sensors coupled to the vehicle. The device's user movement estimator determines the user portion of the AR headset movement caused by movement of the user's head, rather than movement of the vehicle. In a particular example, the user turns 5 degrees to look toward the dashboard in the vehicle, and the vehicle turns left. In this example, the vehicle sensor data indicates that the vehicle has turned 90 degrees. The headset sensor data indicates that the AR headset has turned 95 degrees (90 degrees due to vehicle movement + 5 degrees due to user movement). The user movement estimator determines the user portion of the AR headset movement based on a comparison of the AR headset movement (e.g., 95 degrees) and the vehicle movement (e.g., 90 degrees). For example, the user movement estimator determines the user portion of the movement based on determining that the net value of the AR headset movement and the vehicle movement is 5 degrees, indicating a 5-degree movement relative to the vehicle. In another example, the user movement estimator determines that the user part indicates no movement based on matching the movement of the AR headset with the movement of the vehicle.

[0024] A gaze target estimator estimates a user's gaze target based on the movement of the user portion of the AR headset. The gaze target indicates where the user is looking at a specific time. In one particular example, the gaze target estimator determines the orientation and position of the AR headset relative to a vehicle based on the movement of the user portion of the AR headset. The gaze target estimator determines that the user is looking towards the vehicle's dashboard based on the orientation and position of the AR headset. In some implementations, the headset sensor data includes image sensor data. In such implementations, the gaze target estimator performs image recognition on the image sensor data to determine that the gaze target includes a specific instrument on the dashboard. In another example, the gaze target may indicate that the user is looking towards a location outside the vehicle (e.g., out a window).

[0025] The visualization data generator produces visualization data for AR headsets based on the gaze target and the context of a specific time period. For example, the context indicates whether the user is inside or outside a vehicle. In some examples, the context indicates the user's role. For the same gaze target, the visualization data can vary depending on the context. For example, for a user looking at a ship from inside an airplane, the visualization data could include indicators identifying the ship, and for a user (e.g., a paratrooper) looking at the ship from outside an airplane, the visualization data could include a visual depiction of an anemometer. As another example, when the user is gazing at the target inside a vehicle, the visualization data could include a 3D rotatable map with a visual depiction of the object, and when the user is gazing at the target outside a vehicle, the visualization data could include indicators marking the location of the object.

[0026] In some examples, visualization data may include indicators marking the location of objects outside a vehicle, even though those objects are not visible to the user (e.g., behind clouds or behind another object).

[0027] In some examples, the visualization data indicates at least a portion of a vehicle's route. For instance, the visualization data could include a 3D rotatable map with a visual depiction of the route of one or more vehicles when the user is viewing the target inside the vehicle. As another example, the visualization data could include indicators marking the route of one or more vehicles when the user is viewing the target outside the vehicle.

[0028] In a specific example, visualization data reduces the mental effort required for a user to visually move from one context to another. For instance, visualization data includes a visual depiction of an object (e.g., a boat, a portion of a boat's route, or both) in a first context (e.g., a map displayed inside a vehicle), and indicators for marking the location of the object (e.g., a boat, a portion of a boat's route, or both) in a second context (e.g., outside the vehicle). The indicators, together with the visual depiction, include at least one visual element, such as color, image, video, animation, symbol, text, pattern, or a combination thereof. Having similar visual elements for the same object helps users detect the corresponding object in different contexts.

[0029] The accompanying drawings and the following description illustrate specific exemplary embodiments. It should be understood that those skilled in the art will be able to devise various arrangements that, while not expressly described or shown herein, embody the principles set forth herein and are included within the scope of the claims appended to this specification. Furthermore, any examples described herein are intended to aid in understanding the principles of this disclosure and should be construed as not limiting. Therefore, this disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0030] This document describes specific embodiments with reference to the accompanying drawings. Throughout the specification, common features are indicated by common reference numerals. As used herein, various terms are used only for the purpose of describing particular embodiments and are not intended to be limiting. For example, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some embodiments and plural in others. For illustration, Figure 1 It describes a system that includes one or more processors ( Figure 1 The device 102 (of which "(one or more) processors" 134) indicates that in some embodiments, device 102 includes a single processor 134, and in other embodiments, device 102 includes multiple processors 134. For ease of reference herein, such features are generally described as "one or more" features and are subsequently referred to in the singular unless aspects relating to multiple features are described.

[0031] The terms “comprising” and its variations are used interchangeably with “including” and its variations. Additionally, the terms “wherein” and “in the case of” are used interchangeably. As used herein, “exemplary” indicates an example, implementation, and / or aspect, and should not be construed as limiting or indicating a preference or preferred implementation. As used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., “first,” “second,” “third,” etc.) do not themselves indicate any priority or order of the element relative to another element, but merely distinguish the element from another element having the same name (except for the use of ordinal numbers). As used herein, the term “set” refers to a grouping of one or more elements, and the term “multiple” refers to multiple elements.

[0032] As used herein, unless the context otherwise indicates, “generate,” “calculate,” “use,” “select,” “access,” and “determine” are interchangeable. For example, “generate,” “calculate,” or “determine” a parameter (or signal) can refer to actively generating, calculating, or determining a parameter (or signal), or it can refer to using, selecting, or accessing a parameter (or signal) that has already been generated, such as by other components or devices. As used herein, “coupling” can include “communication coupling,” “electrical coupling,” or “physical coupling,” and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communication coupling, electrical coupling, or physical coupling) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative and non-limiting example, two electrically coupled devices (or components) may be included in the same device or different devices and may be connected via electronics, one or more connectors, or inductive coupling. In some implementations, two devices (or components) that are communicatively coupled (e.g., in an electrical communication manner) can directly or indirectly (e.g., via one or more wires, buses, networks, etc.) send and receive electrical signals (digital or analog signals). As used herein, "direct coupling" is used to describe two devices coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intermediate components.

[0033] Figure 1This is a block diagram of a system 100 operable for performing augmented reality visualization. System 100 includes a device 102 coupled to an augmented reality (AR) headset 108 and a vehicle 104 via a network 190. Network 190 includes a wired network, a wireless network, or both. AR headset 108 includes (or is coupled to) one or more headset sensors 131. Headset sensors 131 include accelerometers, gyroscopes, magnetometers, inertial measurement units, image sensors, global positioning system (GPS) receivers, beacons, or combinations thereof. Headset sensors 131 are configured to generate headset sensor (HS) data 141. Vehicle 104 includes (or is coupled to) one or more vehicle sensors 162. Vehicle sensors 162 include radar receivers, sonar hydrophones, surface search radar receivers, acoustic sensors, seismic sensors, ground surveillance radar receivers, passive sonar hydrophones, passive radar receivers, Doppler radar receivers, accelerometers, gyroscopes, magnetometers, inertial measurement units, image sensors, GPS receivers, beacons, or combinations thereof. The vehicle sensor 162 is configured to generate vehicle sensor (VS) data 159.

[0034] It should be noted that in the following description, by Figure 1 The various functions performed by System 100 are described as being executed by certain components or modules. However, this division of components and modules is for illustrative purposes only. Alternatively, the functions described herein, performed by specific components or modules, may be divided among multiple components or modules. Furthermore, alternatively... Figure 1 Two or more components or modules are integrated into a single component or module. Figure 1 Each component or module shown may be implemented using hardware (e.g., field-programmable gate array (FPGA) devices, application-specific integrated circuits (ASICs), digital signal processors (DSPs), controllers, etc.), software (e.g., instructions executable by a processor), or any combination thereof.

[0035] Device 102 includes memory 132, one or more processors 134, and interface 130. Interface 130 includes a communication interface, a network interface, an application programming interface, or a combination thereof. Memory 132 is configured to store data 103 used (e.g., generated) by processor 134. In some aspects, a portion of data 103 is stored in memory 132 at any given time. Interface 130 is configured to communicate with network 190. Processor 134 includes a user mobility estimator 140, a gaze target estimator 142, a route estimator 144, a context estimator 152, a visualization data generator 154, or a combination thereof. User mobility estimator 140 is configured to estimate headset movement 109 of AR headset 108 based on HS data 141 and vehicle movement 105 of vehicle 104 based on VS data 159. User mobility estimator 140 is configured to estimate user movement 111 based on a comparison of headset movement 109 and vehicle movement 105. User movement 111 corresponds to the user portion of the headset movement 109 caused by the movement of the user 101's head on the AR headset 108, rather than by the movement of the vehicle 105.

[0036] The gaze target estimator 142 is configured to estimate the gaze target 127 of user 101 (e.g., the location of gaze) based on user movement 111. In a particular example, the gaze target estimator 142 is configured to determine the headset orientation 115, headset position 113, or both of AR headset 108 relative to vehicle 104 based on user movement 111. For illustration, user movement 111 indicates that AR headset 108 has moved a specific angle (e.g., 5 degrees), a specific distance, or both relative to vehicle 104. Headset orientation 115 indicates that user 101 is looking in a specific direction relative to vehicle 104 (e.g., 10 degrees in front of vehicle 104). Headset position 113 indicates the position of AR headset 108 relative to vehicle 104 (e.g., within vehicle 104).

[0037] In one particular aspect, the gaze target estimator 142 is configured to determine the gaze target 127 based on headphone orientation 115, headphone position 113, or both. For example, the gaze target estimator 142 determines that the gaze target 127 includes a specific dashboard of the vehicle 104. As another example, the gaze target estimator 142 determines that the gaze target 127 includes a window of the vehicle 104.

[0038] In one particular aspect, the gaze target estimator 142 is configured to determine (e.g., refine) the gaze target 127 by performing image recognition on the HS data 141. For example, the gaze target estimator 142 is configured to determine, based on image recognition, that the gaze target 127 includes a specific instrument on a dashboard. As another example, the gaze target estimator 142 is configured to determine, based on image recognition, that the gaze target 127 includes a specific object on the exterior of the vehicle 104 that is visible through a window.

[0039] Route estimator 144 is configured to determine one or more routes 178 for vehicle 104, one or more routes for vehicle 106, or a combination thereof. For example, route 178 includes one or more planned routes 180, one or more traversed routes 182, one or more interpolated routes 184, one or more predicted routes 186, one or more recommended routes 188, or a combination thereof. For example, the vehicle intends to travel along planned route 180 from the origin to the destination. A traversed route 182 from a first location to a second location is detected. In some examples, the first location includes the origin, and the second location is located between the origin and the destination. In other examples, the second location is not between the origin and the destination if the vehicle has rerouted from planned route 180 for various reasons (such as traffic, weather, obstacles, etc.). In a particular example, VS data 159 (e.g., GPS data, radar data, sonar data, or a combination thereof) indicates that a vehicle has been detected at the estimated location. An interpolated route 184 is determined for the vehicle from the second location to the estimated location. The predicted route 186 of the vehicle indicates the expected route of the vehicle from the estimated location to the predicted location on the planned route 180. The recommended route 188 of the vehicle satisfies route objectives 194. Route objectives 194 include obstacle avoidance, collision avoidance, severe weather avoidance, area avoidance, time objectives, fuel consumption objectives, cost objectives, or combinations thereof.

[0040] Context estimator 152 is configured to determine context 137. For example, context 137 includes user role 139 of user 101, user location context 121 (e.g., whether user 101 is inside or outside vehicle 104), gaze target context 123 (e.g., the location of the object the user is looking at), or a combination thereof. Visualization data generator 154 is configured to generate visualization data 155 based on gaze target 127 (e.g., what user 101 is looking at), context 137, route 178, or a combination thereof. For example, visualization data 155 includes one or more visual elements 163 (e.g., virtual elements) selected based on gaze target 127, context 137, route 178, or a combination thereof. Visualization data generator 154 is configured to send visualization data 155 to the display of AR headset 108.

[0041] During operation, user 101 activates (e.g., powers on) AR headset 108. Headset sensor 131 generates HS data 141. For example, headset sensor 131 generates HS data 141 during a first time range. For illustration, one or more sensors in headset sensor 131 continuously generate data at various time intervals during the first time range in response to detected events, or combinations thereof. Events may include receiving a request from device 102, receiving user input, detecting movement of AR headset 108, or combinations thereof.

[0042] Device 102 receives HS data 141 from AR headset 108. For example, AR headset 108 continuously sends HS data 141 to device 102 at various time intervals, in response to the detection of an event, or a combination thereof. Events may include receiving a request from device 102, receiving user input, detecting movement of AR headset 108, detecting an update of HS data 141, or a combination thereof.

[0043] Vehicle sensor 162 generates VS data 159. For example, vehicle sensor 162 generates VS data 159 during a second time range. In one particular aspect, the second time range is the same as, overlaps with, or falls within a threshold duration (e.g., 5 minutes) of the first time range. For illustration, one or more sensors of vehicle sensor 162 continuously generate data during the second time range at various time intervals in response to the detection of an event, or a combination thereof. Events may include receiving a request from device 102, receiving user input, detecting movement of vehicle 104, or a combination thereof.

[0044] Device 102 receives VS data 159 from vehicle 104. For example, vehicle 104 continuously sends VS data 159 to device 102 at various time intervals, in response to the detection of an event, or a combination thereof. Events may include receiving a request from device 102, receiving user input, detecting movement of vehicle 104, detecting an update of VS data 159, or a combination thereof.

[0045] In one particular aspect, device 102 receives planned route data 191, location data 193, or a combination thereof from one or more data sources 192, vehicles 104, vehicles 106, or combinations thereof via interface 130. One or more data sources 192, vehicles 104, vehicles 106, or combinations thereof continuously transmit planned route data 191, location data 193, or combinations thereof to device 102 at various time intervals in response to the detection of an event, or a combination thereof. Events may include receiving a request from device 102, receiving user input, or a combination thereof. In one particular aspect, one or more data sources 192 include shore stations, satellites, Aircraft Communications Addressing and Reporting Systems (ACARS), Blue Force Trackers (BFT), or combinations thereof.

[0046] In one particular example, device 102 receives planned route data 191A for vehicle 104 from vehicle 104, via user input (e.g., from the operator of vehicle 104), or both. Planned route data 191A indicates a planned route 180A for vehicle 104 from a first origin at a first time (e.g., the estimated departure time) to a first destination at a second time (e.g., the estimated arrival time). For example, planned route 180A indicates that vehicle 104 is expected to be at the first origin at a first time, at a second location at a specific time, at one or more additional locations at one or more additional times, at the first destination at a second time, or a combination thereof. In one particular aspect, device 102 receives planned route data 191A before vehicle 104 departs from the first origin.

[0047] In one particular aspect, device 102 receives planned route data 191B from data source 192A via network 190. Planned route data 191B instructs vehicle 106 to take a planned route 180B from a second origin at a first time (e.g., estimated departure time) to a second destination at a second time (e.g., estimated arrival time). For example, planned route 180B instructs vehicle 106 to be at the second origin at a first time, at a second location at a specific time, at one or more additional locations at one or more additional times, at the second destination at a second time, or a combination thereof. The first origin may be the same as or different from the second origin or the second destination. The first destination may be the same as or different from the second origin or the second destination.

[0048] In one particular aspect, route estimator 144 sends a planned route data request to data source 192A and receives planned route data 191B in response to the planned route data request. In one particular embodiment, the planned route data request indicates a planned route 180A for vehicle 104, and data source 192A sends planned route data 191 for one or more vehicles 106 expected to be within a threshold distance of vehicle 104 along planned route 180A. For example, data source 192A determines that vehicle 106A is expected to be within a threshold distance of vehicle 104 based on a comparison of planned route 180A and planned route 180B. In response to determining that vehicle 106A is expected to be within the threshold distance of vehicle 104, data source 192A sends planned route data 191B to device 102. In one particular embodiment, the planned route data request indicates one or more domains associated with (e.g., traversed by) planned route 180A, and data source 192A sends planned route 180B in response to determining that planned route 180B is associated with at least one of the one or more domains (e.g., traversed by one or more areas).

[0049] In one particular aspect, device 102 receives location data 193A of vehicle 104 from vehicle 104. Location data 193A indicates a specific location of vehicle 104 detected at a specific time. In one particular embodiment, device 102 sends a request for planned route data indicating a specific location and a specific time to data source 192A, and data source 192A sends planned route data 191 of one or more vehicles 106 expected to be within a threshold distance of a specific location within a threshold time period at the specific time.

[0050] In one particular aspect, route estimator 144 determines the route 182A traversed by vehicle 104 from a first location to a specific location. For example, route estimator 144 receives location data 193A from vehicle 104 at various time intervals. To illustrate, route estimator 144 receives location data 193A indicating that vehicle 104 was detected at a first location at a first time, location data 193A indicating that vehicle 104 was detected at a second location at a second time, location data 193A indicating that vehicle 104 was detected at one or more additional locations at one or more additional times, location data 193A indicating that vehicle 104 was detected (e.g., most recently detected) at a specific location at a specific time, or a combination thereof. Route estimator 144 determines the route 182A traversed by vehicle 104 from the first location via a second location, one or more additional locations, or a combination thereof to the specific location. In one particular aspect, route 182A indicates points (and corresponding times) along the route traversed by vehicle 104 from the first location to the specific location. In one particular implementation, route 182A includes straight-line segments between paired detection locations of vehicle 104. In one example, route 182 includes a curved route consisting of a series of straight-line segments that are curved to a specific scale. In one particular example, the first location includes the origin, and the specific location is between the origin and the destination.

[0051] In one particular aspect, device 102 receives location data 193 of one or more vehicles 106 from data source 192B. For example, vehicle 106 transmits its location information at various time intervals (e.g., at 15-minute intervals), and data source 192B stores the vehicle location information. For illustration, the vehicle location information is received by a shore station, which forwards the vehicle location information to data source 192B via satellite.

[0052] In one particular aspect, route estimator 144 determines one or more routes 182 for one or more vehicles 106 based on location data 193. For example, location data 193B indicates that vehicle 106 was detected at a first location at a first time, at one or more additional times, at a specific location at a specific time (e.g., the most recently detected location), or a combination thereof. Route estimator 144 determines a route 182B for vehicle 106 from the first location via one or more additional locations to the specific location. In one particular aspect, route 182B indicates points (and corresponding times) along the route traversed by vehicle 106 from the first location to the specific location. In one particular embodiment, route 182B includes straight segments, curved segments, or combinations thereof between pairs of detected locations of vehicle 104.

[0053] In one particular aspect, route estimator 144 determines one or more interpolated routes 184 for one or more vehicles 106. For example, there may be a delay between vehicle 106 transmitting vehicle location information indicating a specific location (e.g., the most recently detected location) and device 102 receiving the vehicle location information. Vehicle 106 may have moved from the specific location during the delay. For example, VS data 159 (e.g., radar data, sonar data, or both) indicates the estimated location of vehicle 106. In one particular aspect, route estimator 144 determines the interpolated route 184 of vehicle 106 in response to determining that the difference between the estimated location indicated by VS data 159 and the specific location indicated by location data 193 (e.g., the most recently detected location) is greater than a threshold distance. In one particular aspect, determining the interpolated route 184 of vehicle 106 includes performing interpolation based on the specific location (e.g., the most recently detected location) and the estimated location. In one particular aspect, the interpolated route 184 corresponds to the shortest traversable path, the fastest traversable path, or both between the specific location and the estimated location. In some examples, the interpolation route 184 corresponds to a straight line between a specific location and an estimated location. In other examples, the interpolation route 184 corresponds to a non-linear path between a specific location and an estimated location. For illustration, the interpolation route 184 follows a curved path (e.g., a road or river) that the vehicle 106 is traveling on, avoids obstacles (e.g., bypasses islands, mountains, or buildings), follows a legally accessible path (e.g., avoids restricted areas, one-way streets in the wrong direction, etc.), or a combination thereof.

[0054] In one particular aspect, the specific location indicated by the location data 193A of vehicle 104 is the same as the estimated location indicated by the VS data 159 of vehicle 104. For example, device 102 receives location data 193A, VS data 159, or a combination thereof from vehicle 104. The location data 193A, VS data 159, or a combination thereof indicates the same vehicle location information (e.g., GPS coordinates) of vehicle 104.

[0055] In one particular aspect, route estimator 144 determines one or more predicted routes 186 for vehicle 104, or a combination thereof. For example, route estimator 144 determines a predicted route 186 from an estimated location to a predicted location on a planned route. The predicted route 186 is based on the estimated location (e.g., indicated by VS data 159), vehicle status (e.g., indicated by VS data 159), vehicle capacity (e.g., indicated by vehicle capacity data), external conditions (e.g., indicated by external condition data), or a combination thereof. For illustration, route estimator 144 determines a predicted route 186A for vehicle 104 from an estimated location to a predicted location on a planned route 180A for vehicle 104. The predicted route 186A is based on the estimated location of vehicle 104, the vehicle state of vehicle 104 (e.g., direction, speed, acceleration, or a combination thereof), the vehicle capacity of vehicle 104 (e.g., average speed, maximum speed, turning radius, average acceleration, maximum acceleration, average deceleration, maximum deceleration), and external conditions (e.g., weather conditions, wind speed, airflow, water flow, obstacles, restricted areas), or a combination thereof. In one particular aspect, given the vehicle state, vehicle capacity, external conditions, or a combination thereof, route estimator 144 determines the predicted route 186A that vehicle 104 may take from the estimated location back to the planned route 180A. In one particular aspect, route estimator 144 receives vehicle capacity data, external condition data, or a combination thereof from one or more data sources 192, vehicle 104, or a combination thereof.

[0056] In one particular aspect, route estimator 144 determines a predicted route 186B for vehicle 106 from an estimated location to a predicted location on the planned route 180B of vehicle 106. The predicted route 186B is based on the estimated location of vehicle 106 (e.g., indicated by VS data 159), the vehicle state of vehicle 106 (e.g., indicated by VS data 159, such as direction, speed, acceleration, or a combination thereof), the vehicle capacity of vehicle 106 (e.g., indicated by vehicle capacity data, such as average speed, maximum speed, turning radius, average acceleration, maximum acceleration, average deceleration, maximum deceleration), external conditions (e.g., indicated by external condition data, such as weather conditions, wind speed, wind flow, water flow, obstacles, restricted areas), or a combination thereof. In one particular aspect, given the vehicle state, vehicle capacity, external conditions, or a combination thereof, route estimator 144 determines the predicted route 186B that vehicle 106 may take from the estimated location back to the planned route 180B. In one particular aspect, route estimator 144 receives vehicle capacity data, external condition data, or a combination thereof from one or more data sources 192, vehicle 104, vehicle 106, or a combination thereof.

[0057] In one particular aspect, route estimator 144 determines one or more recommended routes 188 for vehicle 104. For example, route estimator 144 determines recommended routes 188 from estimated locations to recommended locations on planned routes. Recommended routes 188 satisfy route objectives 194. For example, route objectives 194 include obstacle avoidance, collision avoidance, severe weather avoidance, area avoidance, time objectives, fuel consumption objectives, cost objectives, or combinations thereof. In one particular aspect, route objectives 194 are based on user input, default data, configuration settings, or combinations thereof. In one particular aspect, recommended routes 188 are based on the estimated location of vehicle 104 (e.g., indicated by VS data 159), the vehicle status of vehicle 104 (e.g., indicated by VS data 159), the vehicle capacity of vehicle 104 (e.g., indicated by vehicle capacity data), external conditions (e.g., indicated by external condition data), one or more predicted routes 186, route objectives 194, or combinations thereof. For illustration, route estimator 144 determines a recommended route 188A for vehicle 104 from an estimated location to a recommended location on the planned route 180A of vehicle 104. Recommended route 186A is based on the estimated location of vehicle 104, vehicle status of vehicle 104 (e.g., direction, speed, acceleration, or a combination thereof), vehicle capacity of vehicle 104 (e.g., average speed, maximum speed, turning radius, average acceleration, maximum acceleration, average deceleration, maximum deceleration), external conditions (e.g., weather conditions, wind speed, airflow, water flow, obstacles, restricted areas), one or more predicted routes 186 for one or more vehicles 106, route destination 194, or a combination thereof. In one particular aspect, given vehicle status, vehicle capacity, external conditions, possible routes for one or more vehicles 106, or a combination thereof, route estimator 144 determines a recommended route 188A for vehicle 104 to reach the planned route 180A from the estimated location and satisfy route destination 194.

[0058] In one particular aspect, in response to determining that the predicted route 186 fails to meet the route objective 194, the route estimator 144 generates a recommended route 188 that meets the route objective 194. The recommended route 188 indicates a recommended heading, recommended speed, recommended altitude, or a combination thereof, which differs from the predicted heading, predicted speed, predicted altitude, or a combination thereof indicated by the predicted route 186.

[0059] In a specific example, route estimator 144 detects a predicted collision between vehicle 104 and vehicle 106 based on a comparison of predicted routes 186A for vehicle 104 and 186B for vehicle 106. For illustration, route estimator 144 detects a predicted collision in response to determining that a first predicted position of vehicle 104 at a first predicted time indicated by predicted route 186A is within a threshold distance of a second predicted position of vehicle 106 at a second predicted time indicated by predicted route 186B, and that the first predicted time is within a threshold duration of the second predicted time. In response to determining that route objective 194 includes collision avoidance and detecting the predicted collision, route estimator 144 generates a recommended route 188A for vehicle 104 to avoid the predicted collision, generates an alert 181 indicating the predicted collision, and the recommended route 188A, or both. For example, recommended route 188A includes recommended speed, recommended direction, recommended altitude, or a combination thereof, which differs from the predicted speed, predicted direction, predicted altitude, or a combination thereof indicated by predicted route 186A prior to the predicted location of the predicted collision.

[0060] In one particular aspect, route estimator 144 determines multiple recommended routes 188 for vehicle 104. For example, route estimator 144 determines a recommended route 188A that satisfies route objective 194A (e.g., collision avoidance), a recommended route 188B that satisfies route objective 194B (e.g., time objective), a recommended route 188C that satisfies route objective 194C (e.g., collision avoidance and cost objective), or a combination thereof. In one particular aspect, route estimator 144 stores data indicating one or more routes 178, route objectives 194, alarms 181, planned route data 191, location data 193, or a combination thereof, in memory 132.

[0061] Context estimator 152 retrieves HS data 141 from memory 132 corresponding to a first time range (e.g., 10:10 AM – 10:11 AM). For example, HS data 141 includes sensor data timestamped with timestamps during the first time range (e.g., 10:10:03 AM). Context estimator 152 retrieves VS data 159 from memory 132 corresponding to a second time range (e.g., 10:09 AM – 10:11 AM). For example, VS data 159 includes sensor data timestamped with timestamps during the second time range (e.g., 10:10:07 AM). In response to determining that the first and second time ranges match (e.g., overlap), context estimator 152 determines that HS data 141 corresponds to VS data 159. In a particular example, in response to determining that the first and second time ranges are the same, context estimator 152 determines that the first and second time ranges match. In another example, in response to determining that the first time range overlaps with the second time range, the context estimator 152 determines that the first time range matches the second time range. In a particular example, in response to determining that the duration between the end of one of the first time range or the second time range and the start of the other of the first time range or the second time range is within a threshold duration (e.g., 10 seconds or 1 minute), the context estimator 152 determines that the first time range matches the second time range.

[0062] Context estimator 152 determines context 137 based on HS data 141, VS data 159, or both. For example, in response to determining that HS data 141 corresponds to VS data 159, context estimator 152 determines context 137 based on HS data 141 and VS data 159. In one particular aspect, context estimator 152 determines a user location context 121 indicating whether user 101 is in any vehicle. In one particular embodiment, in response to determining that VS data 159 indicates that a tag associated with user 101 has been detected in the vicinity (e.g., less than 6 inches) of a first sensor in vehicle 104, context estimator 152 generates a user location context 121 indicating that user 101 is in vehicle 104. Alternatively, in response to determining that VS data 159 indicates that a tag associated with user 101 has been detected in the vicinity (e.g., less than 6 inches) of a second sensor outside vehicle 104 (e.g., on an external surface of vehicle 104), context estimator 152 generates a user location context 121 indicating that user 101 is not in vehicle 104. In an alternative implementation, in response to determining that the first user location of user 101 indicated by HS data 141 is within a threshold distance (e.g., 6 inches) of the first vehicle location of vehicle 104 indicated by VS data 159, image recognition performed on the image sensor data of HS data 141 and internal matching of vehicle 104, or both, the context estimator 152 generates a user location context 121 indicating the user 101 in vehicle 104.

[0063] In one particular aspect, user 101 is considered to be "in" vehicle 104 when the movement of vehicle 104 is likely to cause movement of AR headset 108 worn by user 101 independently of any movement caused by user 101. In one particular example, user 101 is standing on the roof of vehicle 104 and is considered to be "in" vehicle 104. In another example, user 101 is standing on the ground outside vehicle 104 with their feet and looking into vehicle 104 with their head through an open window, and user 101 is not considered to be "in" vehicle 104.

[0064] Context estimator 152 determines user role 139 for user 101. In one particular example, user role 139 is relatively static and is indicated by context configuration data 167. In one particular aspect, context configuration data 167 is based on default data, configuration data, user input, or a combination thereof. In an alternative example, user role 139 is dynamic and is at least partially based on user location context 121. For illustration, context configuration data 167 indicates that user 101 has a first role inside vehicle 104 and a second role outside vehicle 104.

[0065] In response to the determination that user location scenario 121 indicates user 101 is in vehicle 104, user movement estimator 140 determines user movement 111 based on a comparison of HS data 141 and VS data 159. In a particular example, user movement estimator 140 determines headset movement 109 indicated by HS data 141. For example, HS data 141 indicates that AR headset 108 moves relative to the environment (e.g., geographic coordinates) in a specific direction, a specific distance, a specific rotation angle, or a combination thereof. Headset movement 109 indicates a specific direction, a specific distance, a specific rotation angle, or a combination thereof. In one particular aspect, the specific rotation angle indicates yaw, pitch, roll, or a combination thereof of user 101's head relative to the environment. User movement estimator 140 determines vehicle movement 105 indicated by VS data 159. For example, VS data 159 indicates that vehicle 104 moves relative to the environment (e.g., geographic coordinates) in a specific direction, a specific distance, a specific rotation angle, or a combination thereof. In one particular aspect, the specific rotation angle indicates yaw, pitch, roll, or a combination thereof of vehicle 104 relative to the environment. The movement of a vehicle 105 indicates a specific direction, a specific distance, a specific rotation angle, or a combination thereof.

[0066] In response to the determination of user location scenario 121 indicating that user 101 is in vehicle 104, user movement estimator 140 determines user movement 111 based on a comparison of headset movement 109 and vehicle movement 105. For example, user movement estimator 140 determines user movement 111 based on the net of headset movement 109 and vehicle movement 105. In one particular aspect, headset movement 109 includes a first portion caused by movement of user 101 and a second portion caused by movement of vehicle 104. For example, user 101 wearing AR headset 108 moves their head (e.g., turns 5 degrees to the left), and vehicle 104 rotates (e.g., turns 90 degrees to the left). Headset movement 109 indicates that AR headset 108 has moved relative to the environment (e.g., 95 degrees). User movement 111 (e.g., 5 degrees to the left) indicates the first portion (e.g., caused by movement of user 101's head rather than by vehicle movement 105).

[0067] In one particular aspect, in response to determining that user location scenario 121 indicates that user 101 is not in vehicle 104 (e.g., not in any vehicle), user movement estimator 140 determines user movement 111 based on HS data 141 and independently of VS data 159. For example, in response to determining that user location scenario 121 indicates that user 101 is not in vehicle 104 (e.g., not in any vehicle), user movement estimator 140 designates headset movement 109 as user movement 111.

[0068] In response to determining that user location scenario 121 indicates that user 101 is in vehicle 104, user movement estimator 140 determines the headphone orientation 115 and headphone position 113 of AR headset 108 relative to vehicle 104. For example, user movement estimator 140 determines headphone orientation 115, headphone position 113, or both, based on user movement 111. In a particular example, user movement 111 indicates a specific rotation angle (e.g., net rotation angle) relative to vehicle 104, and user movement estimator 140 determines headphone orientation 115 based on the specific rotation angle. To illustrate, in response to determining that AR headset 108 previously had a first headphone orientation relative to vehicle 104 (e.g., user 101 is looking forward towards vehicle 104), user movement estimator 140 determines headphone orientation 115 (e.g., -5 degrees or 355 degrees) by applying a specific rotation angle (e.g., 5 degrees to the left) to the first headphone orientation (e.g., 0 degrees). In a particular example, user movement 111 indicates a specific distance, a specific direction, or both relative to vehicle 104, and user movement estimator 140 determines earphone position 113 based on the specific distance, specific direction, or both. For illustration, in response to determining that AR earphone 108 previously had a first earphone position (e.g., first coordinates) relative to vehicle 104, user movement estimator 140 determines earphone position 113 (e.g., second coordinates) relative to vehicle 104 by applying a specific distance, specific direction, or both (e.g., coordinate increments) to the first earphone position (e.g., first coordinates). User movement estimator 140 stores earphone position 113, earphone orientation 115, or both in memory 132.

[0069] In a specific example, HS data 141 indicates that user 101 is looking in a first global direction (e.g., west), and VS data 159 indicates that vehicle 104 is oriented in a second global direction (e.g., north). User motion estimator 140 determines, based on a comparison of the first and second global directions, that user 101 is looking in a specific user-vehicle direction relative to vehicle 104 (e.g., to the left of vehicle 104). Headphone orientation 115 indicates the specific user-vehicle direction.

[0070] In a specific example, HS data 141 indicates that user 101 is located at a first global location, and VS data 159 indicates that the center of vehicle 104 is located at a second global location. User mobility estimator 140 determines a specific user-vehicle location of user 101 relative to the center of vehicle 104 based on a comparison of the first and second global locations. Headset location 113 indicates this specific user-vehicle location.

[0071] In response to the determination of user location scenario 121 indicating that user 101 is not in vehicle 104, gaze target estimator 142 determines gaze target 127 based on HS data 141, user movement 111, or both. In one particular example, HS data 141 indicates user 101's global location (e.g., GPS coordinates), the global direction user 101 is looking in (e.g., compass direction, elevation angle, or both), or both. In another example, HS data 141 indicates user movement 111. In this example, gaze target estimator 142 determines global location, global direction, or both by applying user movement 111 to user 101's previous global location, previous global direction, or both. In one particular aspect, gaze target estimator 142 identifies gaze target 127 in response to determining that map data 135 indicates that gaze target 127 is associated with global location, global direction, or both. In one particular aspect, gaze target estimator 142 performs image recognition by comparing image sensor data from HS data 141 with an image of an object associated with global location, global direction, or both. In this respect, the gaze target estimator 142 identifies the gaze target 127 in response to matching the image of the gaze target 127 determined based on image recognition with image sensor data.

[0072] In one particular aspect, in response to determining that user location scenario 121 indicates that user 101 is in vehicle 104, gaze target estimator 142 determines gaze target 127 based on HS data 141, headphone position 113, headphone orientation 115, or a combination thereof. In one particular example, gaze target 127 indicates what user 101 is looking at (e.g., an object). Gaze target estimator 142 performs image recognition on image sensor data (e.g., captured by a camera of headphone sensor 131) from HS data 141 to identify a specific object. For illustration, gaze target estimator 142 determines that vehicle map data 133 indicates that a gaze target area (e.g., a specific dashboard) corresponds to headphone position 113, headphone orientation 115, or both. Vehicle map data 133 includes images of objects (e.g., specific gauges) located in the gaze target area (e.g., a specific dashboard) within vehicle 104. For example, vehicle map data 133 includes one or more first images of a specific object. The gaze target estimator 142 identifies a specific object as gaze target 127 in response to determining that image recognition indication image sensor data matches a first image. In response to determining that vehicle map data 133 indicates that a specific object is in vehicle 104, the gaze target estimator 142 generates a gaze target context 123 indicating that gaze target 127 is in vehicle 104.

[0073] In one particular example, gaze target estimator 142 determines that user 101 is looking at a specific object outside vehicle 104. For illustration, in response to determining that vehicle map data 133 indicates that earphone location 113, earphone orientation 115, or both correspond to an open or perspective portion (e.g., a window) of vehicle 104, gaze target estimator 142 determines that user 101 is looking outside. In one particular example, gaze target estimator 142 identifies the specific object (outside vehicle 104) as gaze target 127 by performing image recognition. In one particular example, gaze target estimator 142 identifies the specific object outside vehicle 104 based on user 101's global location (e.g., geographic coordinates) and global direction (e.g., east). For example, based on map data 135, gaze target estimator 142 identifies a global region (e.g., a segment of road to the left of a specific road) based on global location and global direction. The gaze target estimator 142 designates a specific object as gaze target 127 in response to a match between image sensor data determined based on image recognition of HS data 141 and an image of the specific object. The gaze target estimator 142 generates a gaze target context 123 indicating that gaze target 127 is outside vehicle 104. In one particular aspect, gaze target context 123 indicates the location (e.g., GPS coordinates) of gaze target 127.

[0074] The visualization data generator 154 generates visualization data 155 based on HS data 141, gaze target 127, scenario 137, scenario configuration data 167, one or more routes 178, alarm 181, or a combination thereof. For example, scenario configuration data 167 indicates data corresponding to gaze target 127, scenario 137, or both. In a particular example, scenario configuration data 167 indicates that first data (e.g., a 3D map having the locations of one or more vehicles 106, the location of vehicle 104, route 178, or a combination thereof) corresponds to user role 139 (e.g., navigator), user location scenario 121 (e.g., inside vehicle 104), gaze target 127 (e.g., the interior portion of vehicle 104), gaze target scenario 123 (e.g., inside vehicle 104), or a combination thereof. In one particular aspect, a first portion of the first data is relatively static (e.g., the location of a landmark in a 3D map), and a second portion of the first data is dynamic (e.g., location and route). For example, a second portion of the first data is received from one or more systems of the vehicle 104 (e.g., radar system, communication system, or both), generated by the route estimator 144, or a combination thereof.

[0075] The visualization data generator 154 generates visualization data 155 based on the first data to include one or more visual elements 163. For example, in response to determining that the gaze target context 123 indicates that the gaze target 127 is inside the vehicle 104, the visualization data generator 154 generates visualization data 155 to include visual elements 163 that will be displayed to a user in the vehicle 104 who has a user role 139 (e.g., a navigator) and is looking in the direction of a specific object (e.g., a specific instrument) in the vehicle 104.

[0076] In one particular aspect, visual element 163 includes a first visual depiction (e.g., a virtual representation) of a point of interest (POI) 164 (e.g., vehicle 106) located outside vehicle 104, a second visual indicator indicating the location of a POI 166 (e.g., a specific meter) located inside vehicle 104, a visual depiction of alarm 181, or a combination thereof. As used herein, “POI” means an object, landmark, location, person, vehicle, at least a portion of a route of a vehicle, data, or a combination thereof to be indicated in visualization data 155. In one particular example, contextual configuration data 167 indicates an object, landmark, location, person, vehicle, portion of a route, or a combination thereof as a POI.

[0077] exist Figure 2A In the example shown, visual element 163 includes a visual depiction 203 (e.g., a virtual representation) of POI 164 (e.g., vehicle 106) outside vehicle 104, a visual indicator 221 indicating the location of POI 166 (e.g., a specific instrument) inside vehicle 104, or both. Visual depiction 203, visual indicator 221, or both include one or more visual elements. In one aspect, visual elements include symbols, shapes, fillers, labels, images, animations, videos, text, patterns, or combinations thereof. In one aspect, visual elements indicate information (e.g., identifier, specifications, remaining fuel, passenger information, cargo information, location, or combinations thereof) of the corresponding POI (e.g., POI 164 or POI 166). In one aspect, visual elements include selectable options associated with the corresponding POI (e.g., POI 164 or POI 166). For example, user 101 can select selectable options associated with POI 164, for example, to communicate with vehicle 106.

[0078] In one particular aspect, visual depictions 203 are generated to display a virtual representation of a point of interest (POI) 164 (e.g., vehicle 106) outside vehicle 104 within vehicle 104. In another particular aspect, visual elements 163 include visual depictions of multiple POIs outside vehicle 104, and visual elements 163 are generated to display the relative positions of the multiple POIs inside vehicle 104. For example, if the distance of POI 164 (e.g., vehicle 106) from a first POI (e.g., vehicle 106) is twice the distance from a second POI (e.g., a mountain) outside vehicle 104, then the visual depiction 203 of POI 164 is generated to be displayed inside vehicle 104 such that the distance from the first visual depiction of the first POI is twice the distance from the second visual depiction of the second POI.

[0079] In one specific aspect, when POI 166 (e.g., a specific meter) is within the field of vision of user 101 wearing headset 108, a visual indicator 221 is generated for display by AR headset 108 to indicate the location of POI 166 within vehicle 104. For example, visualization data generator 154 determines the location of POI 166 within vehicle 104 based on vehicle map data 133, HS data 141, or both. For illustration, visualization data generator 154 performs image recognition on image sensor data from HS data 141 and determines the location of POI 166 within the field of vision of user 101 wearing AR headset 108. Visualization data generator 154 generates visual indicator 221 such that visual indicator 221 indicates the location of POI 166 within the field of vision of user 101. For example, visual indicator 221 is superimposed on the vicinity of POI 166 (e.g., within one inch of POI 166) within the field of vision of user 101.

[0080] exist Figure 3A In the example shown, visual element 163 includes one or more visual depictions of one or more points of interest (POIs) outside vehicle 104 (e.g., one or more portions of one or more routes of vehicle 106). For example, visual element 163 includes a visual depiction 302 (e.g., a virtual representation) of a first POI (e.g., one or more portions of planned route 180B), a visual depiction 304 of a second POI (e.g., passing through one or more portions of route 182B), a visual depiction 306 of a third POI (e.g., interpolating one or more portions of route 184B), a visual depiction 308 of a fourth POI (e.g., predicting one or more portions of route 186B), or a combination thereof.

[0081] Visual depictions 302, 304, 306, and 308 include one or more visual elements. In one particular aspect, visual elements include symbols, shapes, fills, labels, images, animations, videos, text, or combinations thereof. In one particular aspect, visual elements indicate information (e.g., identifiers, remaining fuel, location, time, or combinations thereof) corresponding to a POI (e.g., a portion of a route). For example, visual elements include the time at which vehicle 106 is located (e.g., planned location, passed location, interpolated location, or predicted location) along a corresponding route (e.g., planned route 180B, route 182B, interpolated route 184B, or predicted route 186B). In one particular aspect, visual elements include optional options associated with the corresponding POI (e.g., a portion of a route). For example, user 101 can select optional options associated with the indicated route, such as to communicate with vehicle 106. In one particular aspect, visual depictions 302, 304, 306, and 308 include at least one common element (e.g., the same arrow) to indicate that visual depictions 302, 304, 306, and 308 are related (e.g., related to vehicle 106). In another particular aspect, visual depictions 302, 304, 306, and 308 include at least one distinct element (e.g., dashed line type) to indicate that visual depictions 302, 304, 306, and 308 correspond to portions of different types of routes (e.g., planned, passing, interpolated, or predicted).

[0082] exist Figure 4A In the example shown, visual element 163 includes a visual depiction 403 (e.g., a virtual representation) of a POI (e.g., vehicle 104) and one or more visual depictions of one or more POIs outside vehicle 104 (e.g., one or more portions of one or more routes of vehicle 104). For example, visual element 163 includes a visual depiction 402 (e.g., a virtual representation) of a first POI (e.g., one or more portions of planned route 180A), a visual depiction 404 of a second POI (e.g., one or more portions of route 182A), a visual depiction 408 of a third POI (e.g., one or more portions of predicted route 186A), or a combination thereof.

[0083] Visual depictions 402, 403, 404, and 408 include one or more visual elements. In one aspect, the visual elements indicate information (e.g., identifier, remaining fuel, location, time, or a combination thereof) corresponding to a POI (e.g., vehicle 104, or a portion of a route). In one aspect, the visual elements include optional options associated with the corresponding POI (e.g., a portion of a route, vehicle 104, or both). For example, user 101 can select an optional option associated with predicted route 186A (e.g., the indicated route) to set the course of vehicle 104 to predicted route 186A. In one aspect, visual depictions 402, 404, and 408 include at least one common element (e.g., the same arrow) to indicate that visual depictions 402, 404, and 408 are relevant (e.g., related to vehicle 104). In one aspect, at least one common element of visual depictions 402, 404, and 408 is different from at least one common element of visual depictions 302, 304, 306, and 308. In one particular aspect, visual depiction 302 and visual depiction 402 include at least a first common element (e.g., a first dashed line type) to indicate a common route type (e.g., a planned route), visual depiction 304 and visual depiction 404 include at least a second common element (e.g., a second dashed line type) to indicate a common route type (e.g., a route that has been taken), and visual depiction 308 and visual depiction 408 include at least a third common element (e.g., a third dashed line type) to indicate a common route type (e.g., a predicted route) or a combination thereof.

[0084] exist Figure 5A In the example shown, visual element 163 includes a visual depiction 510 (e.g., a virtual representation) of a POI (e.g., one or more portions of recommended route 188A), a visual depiction 572 of an alert 181, or both. In one particular aspect, visual element 163 includes multiple depictions of multiple recommended routes.

[0085] Visual depiction 510 includes one or more visual elements. In one particular aspect, the visual elements include optional options associated with a corresponding POI (e.g., a portion of a route, vehicle 104, or both). For example, user 101 can select an optional option associated with recommended route 188A (e.g., an indicated route) to set the route of vehicle 104 to recommended route 188A. In one particular aspect, visual depiction 510 includes at least one common element (e.g., the same arrow) of visual depictions 402, 404, 408, and 510 to indicate that visual depictions 402, 404, 408, and 510 are relevant (e.g., related to vehicle 104). In one particular aspect, at least one common element of visual depictions 402, 404, 408, and 510 is different from at least one common element of visual depictions 302, 304, 306, and 308.

[0086] return Figure 1 In response to determining that the gaze target 127 is outside the vehicle 104, the visualization data generator 154 generates visualization data 155 to include visual elements 163 to be displayed to a user in the vehicle 104 who has a user role 139 (e.g., a navigator) and is looking in the direction of a specific object (e.g., vehicle 106) outside the vehicle 104. In one particular aspect, the visual element 163 includes a first visual indicator indicating the location of a POI 166 (e.g., vehicle 106) located outside the vehicle 104, a second visual depiction of a POI 166 (e.g., a specific instrument) located inside the vehicle 104, a visual depiction of an alarm 181, or a combination thereof.

[0087] exist Figure 2B In the example shown, visual element 163 includes a visual depiction 205 of a POI 166 (e.g., a specific instrument) inside vehicle 104, a visual indicator 223 indicating the location of a POI 164 (e.g., vehicle 106) outside vehicle 104, or both. Visual depiction 205, visual indicator 223, or both include one or more visual elements.

[0088] exist Figure 3BIn the example shown, visual element 163 includes a visual indicator 362 indicating the location of a first POI (e.g., one or more portions of planned route 180B), a visual indicator 364 indicating the location of a second POI (e.g., passing through one or more portions of route 182B), a visual indicator 366 indicating the location of a third POI (e.g., interpolating one or more portions of route 184B), and a visual indicator 368 indicating the location of a fourth POI (e.g., predicting one or more portions of route 186B). Visual indicators 362, 364, 366, 368, or combinations thereof comprise one or more visual elements.

[0089] exist Figure 4B In the example shown, visual element 163 includes a visual indicator 462 indicating the location of a first POI (e.g., one or more portions of planned route 180A) and a visual indicator 464 indicating the location of a second POI (e.g., passing through one or more portions of route 182A). Visual indicator 462, visual indicator 464, or both comprise one or more visual elements. In one particular aspect, visual element 163 includes visual indicators corresponding to one or more portions of the route within the user 101's field of vision. Figure 4B In the example shown, part of planned route 180A and part of route 182A are within the field of view of user 101, while predicted route 186A is not within the field of view of user 101.

[0090] exist Figure 5B In the example shown, visual element 163 includes a visual depiction 574 of alarm 181, a visual indicator 568 indicating the location of a first POI (e.g., one or more portions of predicted route 186A), and a visual indicator 570 indicating the location of a second POI (e.g., one or more portions of recommended route 188A). Visual depiction 574, visual indicator 568, visual indicator 570, or a combination thereof, comprise one or more visual elements. Figure 5B In the example shown, portions of predicted route 186A and recommended route 188A are within the user 101's field of vision, while planned route 180A and via route 182A are not. In one particular aspect, visual element 163 includes visual indicators of portions of multiple recommended routes 188 within the user 101's field of vision.

[0091] In one particular implementation, the visualization data generator 154 generates visualization data 155 to reduce the mental workload of the user 101 when switching between the output of the AR headset 108 corresponding to the interior of the vehicle 104 and the output of the AR headset 108 corresponding to the exterior of the vehicle 104. For example, the visual depiction 203, together with the visual indicator 223, includes at least one visual element. Figures 2A-2B In the example, common visual elements include shape and fill (e.g., diagonal fill). As another example, visual depiction 205 and visual indicator 221 together include at least one visual element. Figures 2A-2B In the examples, common visual elements include shape and text (e.g., wind speed). In a particular example, visual depiction 302 and visual indicator 362 together include at least one visual element. Figures 3A-3B In the example, common visual elements include arrows and dashed lines. When user 101 switches between looking at a target inside vehicle 104 and looking at a target outside vehicle 104, common visual elements reduce the mental workload of user 101 in identifying visual depictions and corresponding visual indicators.

[0092] The visualization data generator 154 sends visualization data 155 to the display of the AR headset 108. The AR headset 108 generates AR output indicating visual elements 163 based on the visualization data 155. In one aspect, the AR output can be manipulated based on user input. For example, user 101 can rotate, zoom in, or zoom out of the 3D map. In an illustrative example, the 3D map includes a visual depiction of moving objects in low-visibility terrain, and user 101 (e.g., a navigator) can use the 3D map to consider moving objects and plan a route for vehicle 104 through the terrain.

[0093] System 100 thus enables the determination of the user portion of movement in the AR headset, caused by movement of the user's head rather than by movement of the vehicle in which the user is located. System 100 also enables the visualization data to include context-selective visual elements. These visual elements can be displayed within the user's field of vision, allowing the user to access relevant information without having to switch between contexts or access a fixed display. When the user does switch between contexts, this reduces the mental workload associated with those transitions.

[0094] refer to Figure 6A system operable for performing augmented reality visualizations is shown, and is generally indicated as 600. In system 600, processor 134 is integrated into AR headset 608. For example, visualization data generator 154 generates visualization data 155 based at least in part on HS data generated by AR headset 608 (e.g., headset sensor 131), VS data 159 received from vehicle 104 (e.g., vehicle sensor 162), planned route data 191, location data 193, or combinations thereof, as referenced. Figure 1 As stated above.

[0095] and Figure 1 Compared to system 100, integrating one or more of the user movement estimator 140, gaze target estimator 142, route estimator 144, context estimator 152, or visualization data generator 154 into AR headset 608 reduces the number of devices used for augmented reality visualization. Alternatively, integrating one or more of the user movement estimator 140, gaze target estimator 142, route estimator 144, context estimator 152, or visualization data generator 154 into AR headset 608 reduces the number of devices used for augmented reality visualization. Figure 1 The device 102, distinct from the AR headset 108, enables the AR headset 108 to offload some augmented reality visualization tasks and reduce the resource consumption of the AR headset 108 (e.g., memory, processing cycles, or both). It should be understood that the specific device configured to perform a particular operation is provided as an illustrative example. In other examples, one or more operations described herein may be performed by one or more devices.

[0096] Figure 7 An example of an augmented reality visualization method 700 is shown. In one particular aspect, method 700 is... Figure 1 User movement estimator 140, gaze target estimator 142, route estimator 144, context estimator 152, visualization data generator 154, processor 134, interface 130, device 102. Figure 6 The AR headset 608 or a combination thereof is used for execution.

[0097] Method 700 includes receiving vehicle sensor data at 702 from one or more vehicle sensors coupled to the vehicle. For example, as referenced... Figure 1 As described, Figure 1 Interface 130 receives VS data 159 from vehicle sensor 162. Vehicle sensor 162 is coupled to... Figure 1 Transportation vehicle 104.

[0098] Method 700 also includes receiving headphone sensor data at 704 from one or more headphone sensors coupled to the augmented reality headphones. For example, as referenced Figure 1 As described, Figure 1 The interface 130 receives HS data 141 from the headphone sensor 131. The headphone sensor 131 is coupled to... Figure 1 AR headphones 108.

[0099] Method 700 also includes determining the orientation and position of the augmented reality headphones relative to the vehicle at point 706 based on vehicle sensor data and headphone sensor data. For example, as referenced... Figure 1 As described, Figure 1 The gaze target estimator 142 determines the headphone orientation 115 and headphone position 113 of the AR headphone 108 relative to the vehicle 104 based on VS data 159 and HS data 141.

[0100] In one particular aspect, method 700 includes determining the movement of a vehicle based on vehicle sensor data. For example, as referenced... Figure 1 As described, Figure 1 The user mobility estimator 140 determines the vehicle movement 104 based on VS data 159.

[0101] Method 700 also includes determining the movement of the augmented reality headphones based on headphone sensor data. For example, as referenced... Figure 1 As described, Figure 1 The user movement estimator 140 determines the headphone movement 109 of the AR headphone 108 based on HS data 141.

[0102] Method 700 also includes estimating the user portion of the augmented reality headset movement caused by the user's head movement, rather than the vehicle movement, based on a comparison between the movement of the vehicle and the movement of the augmented reality headset. For example, as referenced Figure 1 As described, Figure 1 The user movement estimator 140 estimates user movement 111 (e.g., user portion of earphone movement 109) caused by head movement of user 101 of AR earphone 108, but not by vehicle movement 105, based on a comparison of vehicle movement 105 and earphone movement 109. See reference... Figure 1 As described, the headphone orientation 115 and headphone position 113 are determined based on user movement 111.

[0103] Method 700 also includes estimating, at 708, the gaze target of the augmented reality headset user based at least in part on the orientation and position of the augmented reality headset relative to the vehicle. For example, as referenced Figure 1 As described, Figure 1 The gaze target estimator 142 determines the gaze target 127 of the user 101 of the AR headset 108 based at least in part on the headset orientation 115 and headset position 113.

[0104] In one particular aspect, method 700 includes determining whether a user is in a vehicle based on headphone sensor data. For example, as referenced... Figure 1 As described, Figure 1 The context estimator 152 determines whether user 101 is in vehicle 104 based on HS data 141. In response to determining that user 101 is in vehicle 104, gaze target estimator 142 estimates gaze target 127 based on headphone orientation 115 and headphone position 113.

[0105] Method 700 also includes generating visualization data based on the gaze target at point 710. For example, as referenced... Figure 1 As described, Figure 1 The visualization data generator 154 generates visualization data 155 based on the gaze target 127. (See reference...) Figure 1 As depicted in Figure 2, in response to determining that the gaze target 127 is inside the vehicle 104, visualization data 155 includes a visual depiction 203 of the POI 164 outside the vehicle 104. (See reference...) Figure 1 As depicted in Figure 2, in response to determining that the gaze target 127 is outside the vehicle 104, visualization data 155 includes a visual depiction 205 of the POI 166 inside the vehicle 104.

[0106] Method 700 also includes sending visualization data to the display of the augmented reality headset at 712. For example, as referenced... Figure 1 As described, Figure 1 The visualization data generator 154 sends visualization data 155 to the display of the AR headset 108.

[0107] Therefore, method 700 enables the determination of the user portion of the AR headset's movement, caused by movement of the user's head rather than by movement of the vehicle in which the user is located. Method 700 also enables the visualization data to include visual elements selected based on context (e.g., whether the gaze target is inside or outside the vehicle). These visual elements can be displayed within the user's field of vision, allowing the user to access relevant information without having to switch between contexts or access a fixed display. When the user does switch between contexts, this reduces the mental workload associated with those transitions.

[0108] Figure 8 An example of an augmented reality visualization method 800 is shown. In one particular aspect, method 800 comprises a user movement estimator 140, a gaze target estimator 142, a route estimator 144, a context estimator 152, a visualization data generator 154, a processor 134, and an interface 130. Figure 1 Equipment 102 Figure 6 The AR headset 608, or a combination thereof, is used to perform this.

[0109] Method 800 includes receiving vehicle sensor data at 802 from one or more vehicle sensors coupled to a first vehicle. For example, Figure 1 Interface 130 receives VS data 159 from vehicle sensor 162, as referenced. Figure 1 The vehicle sensor 162 is coupled to... Figure 1 Transportation vehicle 104.

[0110] Method 800 also includes receiving headphone sensor data at 804 from one or more headphone sensors coupled to the augmented reality headphones. For example, Figure 1 Interface 130 receives HS data 141 from headphone sensor 131, as referenced. Figure 1 The headphone sensor 131 is coupled to... Figure 1 AR headphones 108.

[0111] Method 800 further includes determining the orientation and position of the augmented reality headphones relative to the first vehicle at point 706 based on vehicle sensor data and headphone sensor data. For example, Figure 1 The gaze target estimator 142, based on VS data 159 and HS data 141, determines the headphone orientation 115 and headphone position 113 of the AR headset 108 relative to the vehicle 104, as shown in the reference. Figure 1 As stated above.

[0112] In one particular aspect, method 800 includes determining the movement of a first vehicle based on vehicle sensor data. For example, Figure 1 The user mobility estimator 140 determines the vehicle movement of vehicle 104 based on VS data 159, as referenced in the following example. Figure 1 As stated above.

[0113] Method 800 also includes determining the movement of the augmented reality headphones based on headphone sensor data. For example, Figure 1 The user movement estimator 140 determines the headphone movement 109 of the AR headphone 108 based on HS data 141, as referenced. Figure 1 As stated above.

[0114] Method 800 also includes estimating the user portion of the movement of the augmented reality headset caused by the movement of the user's head, rather than the movement of the first vehicle, based on a comparison of the movement of the first vehicle and the movement of the augmented reality headset. For example, Figure 1The user movement estimator 140 estimates user movement 111 (e.g., user portion of earphone movement 109) caused by head movement of user 101 of AR earphone 108, but not by vehicle movement 105, based on a comparison of vehicle movement 105 and earphone movement 109, as referenced. Figure 1 As described above. Based on user movement 111, the headphone orientation 115 and headphone position 113 are determined, as shown in reference... Figure 1 As stated above.

[0115] Method 800 also includes estimating, at 808, the gaze target of the augmented reality headset user based on headset sensor data and the orientation and position of the augmented reality headset relative to the first vehicle. For example, Figure 1 The gaze target estimator 142 determines the gaze target 127 of the user 101 of the AR headset 108 based on HS data 141, as well as headset orientation 115 and headset position 113, as referenced. Figure 1 As stated above.

[0116] In one particular aspect, method 800 includes determining whether a user is in the first mode of transportation based on headphone sensor data. For example, Figure 1 The context estimator 152 determines whether user 101 is in vehicle 104 based on HS data 141, as shown in the reference. Figure 1 The gaze target estimator 142 estimates the gaze target 127 based on the headphone orientation 115 and headphone position 113 in response to determining that the user 101 is in the vehicle 104.

[0117] Method 800 also includes generating visualization data based on the gaze target at point 810. For example, Figure 1 The visualization data generator 154 generates visualization data 155 based on the gaze target 127, as shown in the reference. Figure 1 As described above. In response to determining that the gaze target 127 is inside the vehicle 104, the visualization data 155 includes a visual depiction 302 of a first POI (e.g., one or more portions of planned route 180B) outside the vehicle 104, a visual depiction 402 of a second POI (e.g., one or more portions of planned route 180A), or both, as referenced. Figure 1 As shown in Figures 3-4. The first POI includes at least a portion of a specific route of vehicle 106. The second POI includes at least a portion of a specific route of vehicle 104. In response to determining that the gaze target 127 is outside vehicle 104, visualization data 155 includes a visual depiction 205 of the POI 166 inside vehicle 104, as referenced. Figure 1 -As shown in Figure 2.

[0118] Method 800 also includes sending visualization data to the display of the augmented reality headset at point 812. For example, Figure 1 The visualization data generator 154 sends visualization data 155 to the display of the AR headset 108, as shown in the reference. Figure 1 As stated above.

[0119] Therefore, Method 800 enables the visualization data to include visual elements selected based on context (e.g., whether the gaze target is inside or outside a vehicle). The visual elements indicate one or more routes for a vehicle, another vehicle, or both. These visual elements can be displayed within the user's field of vision, allowing the user to access relevant information without having to switch between contexts or access a fixed display. The mental workload associated with switching between contexts is reduced.

[0120] Figure 9 This is an illustration of a block diagram of a computing environment 900 including a computing device 910 configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. For example, the computing device 910 or portions thereof is configured to execute instructions to initiate, execute, or control references. Figures 1 to 8 One or more operations are described. In one particular aspect, computing device 910 corresponds to... Figure 1 Device 102, AR headset 108, Vehicle 104 Figure 6 AR headphones 608 or combinations thereof.

[0121] Computing device 910 includes a processor 134. Processor 134 is configured to communicate with system memory 930, one or more storage devices 940, one or more input / output interfaces 950, transceiver 922, one or more communication interfaces 960, or a combination thereof. System memory 930 includes volatile memory devices (e.g., random access memory (RAM) devices), non-volatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory) or both. System memory 930 stores operating system 932, which may include a basic input / output system for booting computing device 910 and a complete operating system enabling computing device 910 to interact with users, other programs, and other devices. System memory 930 stores system (program) data 936. In one particular aspect, Figure 1 The memory 132 includes system memory 930, one or more storage devices 940, or a combination thereof. In one particular aspect, system (program) data 936 includes... Figure 1Data 103. In one particular aspect, data 103 includes (e.g., indication) alarms 181, visualization data 155, context configuration data 167, route destination 194, map data 135, headset orientation 115, vehicle map data 133, gaze target 127, context 137, one or more routes 178, headset location 113, user movement 111, vehicle movement 105, headset movement 109, VS data 159, HS data 141, planned route data 191, location data 193, or combinations thereof.

[0122] System memory 930 includes one or more application programs 934 executable by processor 134. As an example, the one or more application programs 934 include those executable by processor 134 to initiate, control, or execute references. Figures 1 to 8 Instructions for one or more operations described herein. For illustration, one or more applications 934 include instructions executable by processor 134 to initiate, control, or perform one or more operations described by reference user movement estimator 140, gaze target estimator 142, route estimator 144, context estimator 152, visualization data generator 154, or combinations thereof.

[0123] Processor 134 is configured to communicate with one or more storage devices 940. For example, the one or more storage devices 940 include non-volatile storage devices such as disks, optical discs, or flash memory devices. In a particular example, storage device 940 includes both removable and non-removable memory devices. Storage device 940 is configured to store an operating system, images of the operating system, applications, and program data. In one particular aspect, system memory 930, storage device 940, or both include tangible computer-readable media. In one particular aspect, one or more of the storage devices 940 are external to computing device 910.

[0124] Processor 134 is configured to communicate with one or more input / output interfaces 950, which enable computing device 910 to communicate with one or more input / output devices 970 to facilitate user interaction. Processor 134 is configured to detect interaction events based on user input received via input / output interfaces 950. Processor 134 is configured to communicate with device or controller 980 via one or more communication interfaces 960. For example, one or more communication interfaces 960 include... Figure 1The interface 130, and the device or controller 980 include an AR headset 108, a headset sensor 131, a vehicle 104, a vehicle sensor 162, one or more data sources 192, or a combination thereof. In an illustrative example, a non-transitory computer-readable storage medium (e.g., system memory 930) includes instructions that, when executed by a processor (e.g., processor 134), cause the processor to initiate, perform, or control operations. These operations include referencing... Figures 1 to 8 One or more operations described.

[0125] although Figure 1-9 One or more of the examples illustrate systems, apparatuses, and / or methods according to the teachings of this disclosure, but this disclosure is not limited to these illustrated systems, apparatuses, and / or methods. As shown or described herein... Figures 1-9 One or more functions or components of any one of them can be with Figures 1-9 Another combination of one or more other parts. For example, Figure 7 Method 700, one or more elements Figure 8 One or more elements of method 800, or combinations thereof, may be performed in conjunction with other operations described herein. Therefore, no single implementation described herein should be construed as limiting, and implementations of this disclosure may be appropriately combined without departing from the teachings of this disclosure. As an example, see reference to... Figures 1 to 9 One or more operations described may be optional, may be performed at least partially simultaneously, and / or may be performed in an order different from the order shown or described.

[0126] The examples described above are illustrative and do not limit this disclosure. It should be understood that many modifications and variations are possible based on the principles of this disclosure. The examples described herein are intended to provide a general understanding of the structure of various embodiments. These descriptions are not intended to be used as a complete description of all elements and features of apparatuses and systems utilizing the structures or methods described herein. Many other embodiments may become apparent to those skilled in the art after reviewing this disclosure. Other embodiments may be utilized and other embodiments may be derived from this disclosure, allowing for structural and logical substitutions and changes without departing from the scope of this disclosure. For example, method operations may be performed in a different order than that shown in the drawings, or one or more method operations may be omitted. Therefore, this disclosure and the drawings should be considered illustrative rather than restrictive.

[0127] Furthermore, although specific examples have been shown and described herein, it should be understood that any subsequent arrangements designed to achieve the same or similar results may replace the specific embodiments shown. This disclosure is intended to cover any and all subsequent modifications or variations of the various embodiments. After reviewing the specification, combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art.

[0128] Furthermore, this disclosure includes embodiments pursuant to the following provisions:

[0129] Clause 1. A device (102, 608) for augmented reality visualization, said device (102, 608) comprising:

[0130] Interfaces (130, 960) are configured as follows:

[0131] Vehicle sensor data (159) is received from one or more vehicle sensors (162, 980) coupled to the first vehicle (104); and

[0132] One or more headphone sensors (131, 980) coupled to the augmented reality headphones (108, 608, 980) receive headphone sensor data (141); and

[0133] One or more processors (134) are configured to:

[0134] Based on the vehicle sensor data (159) and the headphone sensor data (141), the orientation (115) and position (113) of the augmented reality headphones (108, 608, 980) relative to the first vehicle (104) are determined;

[0135] Based on the headphone sensor data (141) and the orientation (115) and position (113) of the augmented reality headphones (108, 608, 980) relative to the first vehicle (104), the gaze target (127) of the user (101) of the augmented reality headphones (108, 608, 980) is estimated.

[0136] Visualization data (155) is generated based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the first vehicle (104), the visualization data (155) includes a first visual depiction (203, 302, 304, 306, 308, 402, 403, 404, 408, 510) of a first point of interest outside the first vehicle (104), wherein the first point of interest includes at least a portion of a specific route (180, 182, 184, 186, 188) of a specific vehicle (104, 106), wherein the specific vehicle (104, 106) includes the first vehicle (104) or the second vehicle (106), and wherein, in response to determining that the gaze target (127) is outside the first vehicle (104), the visualization data (155) includes a second visual depiction (205) of a second point of interest inside the first vehicle (104); and

[0137] The visualization data (155) is sent to the display of the augmented reality headset (108, 608, 980).

[0138] Clause 2. The device (102, 608) according to Clause 1, wherein the one or more processors (134) are further configured to:

[0139] The movement (105) of the first vehicle (104) is determined based on the vehicle sensor data (159);

[0140] The movement (109) of the augmented reality headphones (108, 608, 980) is determined based on the headphone sensor data (141);

[0141] Based on a comparison of the movement (105) of the first vehicle (104) and the movement (109) of the augmented reality headset (108, 608, 980), estimate the user portion (111) of the movement (109) of the augmented reality headset (108, 608, 980) caused by the movement of the user's (101) head of the augmented reality headset (101), rather than by the movement (105) of the first vehicle (104); and

[0142] Based on the user portion (111) of the movement (109) of the augmented reality headsets (108, 608, 980), the orientation (115) and position (113) of the augmented reality headsets (108, 608, 980) relative to the first vehicle (104) are determined.

[0143] Clause 3. The device (102, 608) according to any of the preceding clauses, wherein the particular vehicle (104, 106) includes the second vehicle (106), wherein the interface (130, 960) is further configured to receive location data (193) indicating a route (182) of the second vehicle (106) from a first location to a second location.

[0144] Clause 4. The device (102, 608) as described in Clause 3, wherein the particular route (180, 182, 184, 186, 188) includes the route (182).

[0145] Clause 5. The device (102, 608) according to Clause 3, wherein the one or more processors (134) are configured to determine an interpolation route (184) of the second vehicle (106) from the second location to an estimated location, wherein determining the interpolation route (184) includes performing interpolation based on the second location and the vehicle sensor data (159), and wherein the particular route (180, 182, 184, 186, 188) includes the interpolation route (184).

[0146] Clause 6. The device (102, 608) according to any of the preceding clauses further includes a memory (132, 930) configured to store location data (193) indicating the route (182) of the first vehicle (104) from the first location to the estimated location.

[0147] Clause 7. The device (102, 608) as described in Clause 6, wherein the particular route (180, 182, 184, 186, 188) includes the route (182).

[0148] Clause 8. The device (102, 608) according to any of the preceding clauses, wherein the interface (130, 960) is further configured to receive planned route data (191) instructing the particular vehicle (104, 106) to take a planned route (180) from the origin to the destination.

[0149] Clause 9. The equipment (102, 608) as described in Clause 8, wherein the particular route (180, 182, 184, 186, 188) includes the planned route (180).

[0150] Clause 10. The device (102, 608) according to Clause 8, wherein the one or more processors (134) are configured to determine a predicted route (186) for the particular vehicle (104, 106) from an estimated position of the particular vehicle (104, 106) to a predicted position on the planned route (180), wherein the predicted route (186) is based on the estimated position, vehicle sensor data (159), vehicle capacity data, external condition data, or a combination thereof, and wherein the particular route (180, 182, 184, 186, 188) includes the predicted route (186).

[0151] Clause 11. The device (102, 608) according to any of the preceding clauses, wherein the one or more processors (134) are configured to determine a recommended route (188) for the first vehicle (104) to satisfy a route objective, the recommended route (188) being a recommended location from an estimated position of the first vehicle (104) to a planned route (180) of the first vehicle (104), and wherein the particular route (180, 182, 184, 186, 188) includes the recommended route (188).

[0152] Clause 12. The equipment (102, 608) pursuant to Clause 11, wherein the route objective (194) includes obstacle avoidance, collision avoidance, severe weather avoidance, area avoidance, time objective, fuel consumption objective, cost objective, or a combination thereof.

[0153] Clause 13. The device (102, 608) according to any of the preceding clauses, wherein the one or more vehicle sensors (162, 980) include radar, sonar, accelerometer, gyroscope, magnetometer, inertial measurement unit, image sensor, global positioning system (GPS) receiver, beacon, or a combination thereof.

[0154] Clause 14. The device (102, 608) according to any of the preceding clauses, wherein the first visual depiction (203, 302, 304, 306, 308, 402, 403, 404, 408, 510) of the first point of interest outside the first vehicle (104) indicates the time at which the particular vehicle (104, 106) is positioned along the particular route (180, 182, 184, 186, 188).

[0155] Clause 15. The device (102, 608) as described in Clause 14, wherein the location of the particular vehicle (104, 106) includes a detected location, an interpolated location, a predicted location, or a recommended location.

[0156] Clause 16. A method (800) for augmented reality visualization, said method (800) comprising:

[0157] At the device (102, 608), vehicle sensor data (159) is received from one or more vehicle sensors (162, 980) coupled to the first vehicle (104);

[0158] At the device (102, 608), headphone sensor data (141) is received from one or more headphone sensors (131, 980) coupled to the augmented reality headphone (108, 608, 980);

[0159] Based on the vehicle sensor data (159) and the headphone sensor data (141), the orientation (115) and position (113) of the augmented reality headphones (108, 608, 980) relative to the first vehicle (104) are determined;

[0160] At the device (102, 608), based on the headphone sensor data (141) and the orientation (115) and position (113) of the augmented reality headphones (108, 608, 980) relative to the first vehicle (104), the gaze target (127) of the user (101) of the augmented reality headphones (108, 608, 980) is estimated;

[0161] At the device (102, 608), visualization data (155) is generated based on the gaze target (127), wherein in response to determining that the gaze target (127) is inside the first vehicle (104), the visualization data (155) includes a first visual depiction (203, 302, 304, 306, 308, 402, 403, 404, 408, 510) of a first point of interest outside the first vehicle (104), wherein the first point of interest includes a specific vehicle. At least a portion of a specific route (180, 182, 184, 186, 188) of a vehicle (104, 106), wherein the specific vehicle (104, 106) includes the first vehicle (104) or the second vehicle (106), and wherein, in response to determining that the gaze target (127) is outside the first vehicle (104), the visualization data (155) includes a second visual depiction (205) of a second point of interest inside the first vehicle (104); and

[0162] The visualization data (155) is sent from the device (102, 608) to the display of the augmented reality headset (108, 608, 980).

[0163] Clause 17. The method (800) according to Clause 16, wherein in response to determining that the gaze target (127) is outside the first vehicle (104), the visualization data (155) further includes visual indicators (223, 362, 364, 366, 368, 462, 464, 568, 570) of the location of the first point of interest outside the first vehicle (104).

[0164] Clause 18. The method (800) according to Clause 17, wherein the first visual depiction (203, 302, 304, 306, 308, 402, 403, 404, 408, 510) and the visual indicator (223, 362, 364, 366, 368, 462, 464, 568, 570) comprises at least one common visual element, and wherein the visual element comprises color, image, video, animation, symbol, text, pattern, or combination thereof.

[0165] Clause 19. A computer-readable storage device (930) storing instructions (934), said instructions (934), when executed by one or more processors (134), causing said one or more processors (134) to:

[0166] Vehicle sensor data (159) is received from one or more vehicle sensors (162, 980) coupled to the first vehicle (104);

[0167] One or more headphone sensors (131, 980) coupled to the augmented reality headphones (108, 608, 980) receive headphone sensor data (141);

[0168] Based on the vehicle sensor data (159) and the headphone sensor data (141), the orientation (115) and position (113) of the augmented reality headphones (108, 608, 980) relative to the first vehicle (104) are determined;

[0169] Based on the headphone sensor data (141) and the orientation (115) and position (113) of the augmented reality headphones (108, 608, 980) relative to the first vehicle (104), the gaze target (127) of the user (101) of the augmented reality headphones (108, 608, 980) is estimated.

[0170] Visualization data (155) is generated based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the first vehicle (104), the visualization data (155) includes a first visual depiction (203, 302, 304, 306, 308, 402, 403, 404, 408, 510) of a first point of interest outside the first vehicle (104), wherein the first point of interest includes at least a portion of a specific route (180, 182, 184, 186, 188) of a specific vehicle (104, 106), wherein the specific vehicle (104, 106) includes the first vehicle (104) or the second vehicle (106), and wherein, in response to determining that the gaze target (127) is outside the first vehicle (104), the visualization data (155) includes a second visual depiction (205) of a second point of interest inside the first vehicle (104); and

[0171] The visualization data (155) is sent to the display of the augmented reality headset (108, 608, 980).

[0172] Clause 20. The computer-readable storage device (930) pursuant to Clause 19, wherein the particular means of transport (104, 106) includes the second means of transport (106), and wherein the instructions (934), when executed by the one or more processors (134), further cause the one or more processors (134) to:

[0173] Receive location data (193) indicating the route (182) taken by the second vehicle (106) from the first location to the second location; and

[0174] Determine an interpolated route (184) for the second vehicle (106) from the second location to the estimated location, wherein determining the interpolated route (184) includes performing interpolation based on the second location and the vehicle sensor data (159), wherein the specific routes (180, 182, 184, 186, 188) include the interpolated route (184), and

[0175] The vehicle sensor data (159) mentioned therein includes radar data, sonar data, or a combination thereof.

[0176] The abstract of this disclosure submitted is to be understood as not intended to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing detailed descriptions, various features may be combined together or described in a single embodiment for the purpose of simplifying the intent of this disclosure. The examples described above are illustrative but not limiting of this disclosure. It should also be understood that many modifications and variations are possible based on the principles of this disclosure. As reflected in the appended claims, the claimed subject matter may apply to all features of fewer than any of the disclosed examples. Therefore, the scope of this disclosure is defined by the appended claims and their equivalents.

Claims

1. A device for augmented reality visualization, the device comprising: The interface is configured as follows: Receive vehicle sensor data from one or more vehicle sensors coupled to the first vehicle; as well as Receive headphone sensor data from one or more headphone sensors coupled to the augmented reality headphones; as well as One or more processors, which are configured as follows: Based on the vehicle sensor data and the headphone sensor data, the orientation and position of the augmented reality headphones relative to the first vehicle are determined; Based on the headphone sensor data and the orientation and position of the augmented reality headphones relative to the first vehicle, the gaze target of the user of the augmented reality headphones is estimated; Visualization data is generated based on the gaze target, wherein in response to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest outside the first vehicle, wherein the first point of interest includes at least a portion of a specific route of a particular vehicle, wherein the particular vehicle includes the first vehicle or a second vehicle, and wherein in response to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest inside the first vehicle. as well as The visualization data is sent to the display of the augmented reality headset.

2. The device of claim 1, wherein the one or more processors are further configured to: The movement of the first vehicle is determined based on the vehicle sensor data; The movement of the augmented reality headphones is determined based on the data from the headphone sensors. Based on a comparison of the movement of the first vehicle and the movement of the augmented reality headset, the user portion of the movement of the augmented reality headset caused by the movement of the user's head, rather than by the movement of the first vehicle, is estimated; as well as Based on the movement of the user portion of the augmented reality headset, the orientation and position of the augmented reality headset relative to the first vehicle are determined.

3. The device according to any of the preceding claims, wherein the particular vehicle includes the second vehicle, and wherein the interface is further configured to receive location data indicating the route taken by the second vehicle from the first location to the second location.

4. The device according to claim 3, wherein the specific route includes the route traversed.

5. The device of claim 3, wherein the one or more processors are configured to determine an interpolated route of the second vehicle from the second location to an estimated location, wherein determining the interpolated route includes performing interpolation based on the second location and the vehicle sensor data, and wherein the specific route includes the interpolated route.

6. The device according to any one of claims 1-2, further comprising a memory configured to store location data indicating the route taken by the first vehicle from the first location to the estimated location.

7. The device of claim 6, wherein the specific route includes the route traversed.

8. The device according to any one of claims 1-2, wherein the interface is further configured to receive planned route data indicating a planned route for the particular vehicle from the origin to the destination.

9. The device of claim 8, wherein the specific route includes the planned route.

10. The device of claim 8, wherein the one or more processors are configured to determine a predicted route for the particular vehicle from an estimated position of the particular vehicle to a predicted position on the planned route, wherein the predicted route is based on the estimated position, vehicle sensor data, vehicle capability data, external condition data, or a combination thereof, and wherein the particular route includes the predicted route.

11. The device according to any one of claims 1-2, wherein the one or more processors are configured to determine a recommended route for the first vehicle to satisfy a route objective, the recommended route being a recommended location from an estimated position of the first vehicle to a planned position on the first vehicle's route, and wherein the specific route includes the recommended route.

12. The device of claim 11, wherein the route objective includes obstacle avoidance, collision avoidance, severe weather avoidance, area avoidance, time objective, fuel consumption objective, cost objective, or a combination thereof.

13. The device according to any one of claims 1-2, wherein the one or more vehicle sensors comprise radar, sonar, accelerometer, gyroscope, magnetometer, inertial measurement unit, image sensor, GPS receiver, beacon, or a combination thereof.

14. The device according to any one of claims 1-2, wherein the first visual depiction of the first point of interest outside the first vehicle indicates the time of the location of the particular vehicle along the particular route.

15. The device of claim 14, wherein the position of the particular vehicle includes a detected position, an interpolated position, a predicted position, or a recommended position.

16. A method for augmented reality visualization, the method comprising: The device receives vehicle sensor data from one or more vehicle sensors coupled to the first vehicle. The device receives headphone sensor data from one or more headphone sensors coupled to the augmented reality headphones; Based on the vehicle sensor data and the headphone sensor data, the orientation and position of the augmented reality headphones relative to the first vehicle are determined; At the device, based on the headphone sensor data and the orientation and position of the augmented reality headphones relative to the first vehicle, the gaze target of the user of the augmented reality headphones is estimated; At the device, visualization data is generated based on the gaze target, wherein in response to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest outside the first vehicle, wherein the first point of interest includes at least a portion of a specific route of a particular vehicle, wherein the particular vehicle includes the first vehicle or a second vehicle, and wherein in response to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest inside the first vehicle; as well as The visualization data is sent from the device to the display of the augmented reality headset.

17. The method of claim 16, wherein in response to determining that the gaze target is outside the first vehicle, the visualization data further includes a visual indicator of the location of the first point of interest outside the first vehicle.

18. The method of claim 17, wherein the first visual depiction and the visual indicator comprise at least one common visual element, and wherein the visual element comprises color, image, video, animation, symbol, text, pattern, or a combination thereof.

Images