System and method for augmented reality visualization based on sensor data

Abstract

The present application discloses systems and methods for augmented reality visualization based on sensor data. A device is configured to determine the movement of a vehicle based on vehicle sensor data, and to determine the movement of an augmented reality (AR) headset based on headset sensor data. The device is configured to estimate the user portion of the movement of the AR headset caused by the user's movement rather than the movement of the vehicle. The device is configured to determine a gaze target of the user based on the user portion, and to generate visualization data based on the gaze target. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle.

Description

Technical Field

[0001] The present disclosure relates generally to augmented reality visualization. Background Art

[0002] Augmented reality (AR) is rapidly changing the way people interact with computer systems and the environment. The technology is expected to have a broad impact on aerospace and defense. Aircrews (e.g., crews of commercial aircraft, military aircraft, ships, and ground transportation) typically maintain situational awareness between two different contexts: information presented primarily on fixed, two-dimensional computer displays, and the three-dimensional external environment. Crews transition between these two contexts by redirecting their gaze or by physically moving between consoles and windows. Mental transitions can also occur when crews attempt to map between two-dimensional graphics and three-dimensional terrain, for example. Size and weight constraints may limit the number of fixed displays, so some crew members may not be able to access information as needed or when needed. Summary of the invention

[0003] In a specific embodiment, a device for augmented reality visualization includes an interface and one or more processors. The interface is configured to receive vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The interface is also configured to receive headset sensor data from one or more headset sensors coupled to an augmented reality headset. The one or more processors are configured to determine the movement of the vehicle based on the vehicle sensor data. The one or more processors are also configured to determine the movement of the augmented reality headset based on the headset sensor data. The one or more processors are also configured to estimate the user portion of the movement of the augmented reality headset based on a comparison of the movement of the vehicle with the movement of the augmented reality headset, the user portion being caused by the movement of the head of the user of the augmented reality headset, rather than by the movement of the vehicle. The one or more processors are also configured to determine the orientation and position of the augmented reality headset relative to the vehicle based on the user portion of the movement of the augmented reality headset. The one or more processors are also configured to estimate the user's gaze target based on the headset sensor data and the orientation and position of the augmented reality headset relative to the vehicle. The one or more processors are further configured to generate visualization data based on the gaze target. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle. The one or more processors are further configured to send the visualization data to a display of the augmented reality headset.

[0004] In another specific embodiment, a method of augmented reality visualization includes receiving vehicle sensor data from one or more vehicle sensors coupled to a vehicle at a device. The method also includes receiving headset sensor data from one or more headset sensors coupled to an augmented reality headset at the device. The method also includes determining an orientation and position of the augmented reality headset relative to the vehicle based on the vehicle sensor data and the headset sensor data. The method also includes estimating a gaze target of a user of the augmented reality headset based at least in part on the orientation and position of the augmented reality headset relative to the vehicle. The method also includes generating visualization data based on the gaze target at the device. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle. The method also includes sending the visualization data from the device to a display of the augmented reality headset.

[0005] In another specific embodiment, a computer-readable storage device stores instructions that, when executed by a processor, cause the processor to perform operations including receiving vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The operations also include receiving headset sensor data from one or more headset sensors coupled to an augmented reality headset. The operations also include estimating a gaze target of a user of the augmented reality headset based on the vehicle sensor data and the headset sensor data. The operations also include generating visualization data based on the gaze target. In response to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest outside the vehicle. In response to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest inside the vehicle. The operations also include sending the visualization data to a display of the augmented reality headset.

[0006] The features, functions, and advantages described herein can be achieved independently in various embodiments or may be combined in yet other embodiments further details of which can be found with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram illustrating a system operable to perform augmented reality visualization;

[0008] Figure 2A is a diagram illustrating an example of augmented reality visualization.

[0009] Figure 2B is a diagram showing another example of augmented reality visualization;

[0010] Figure 3 is a diagram illustrating an example of another system operable to perform augmented reality visualization;

[0011] Figure 4 is a flow chart illustrating an example of a method of augmented reality visualization; and

[0012] Figure 5 is a block diagram depicting a computing environment including a computing device configured to support aspects of the computer-implemented method and computer-executable program instructions (or code) according to the present disclosure. DETAILED DESCRIPTION

[0013] Embodiments described herein are directed to augmented reality visualization. In one specific example, a user in a vehicle wears an augmented reality (AR) headset. A device receives headset sensor data from one or more headset sensors coupled to the AR headset. The device also receives vehicle sensor data from one or more vehicle sensors coupled to the vehicle. A user movement estimator of the device determines the user portion of the movement of the AR headset caused by the movement of the user's head rather than the movement of the vehicle. In a specific example, the user turns 5 degrees to look in the direction of the dashboard in the vehicle, and the vehicle turns left. In this example, the vehicle sensor data indicates that the vehicle has turned 90 degrees. The headset sensor data indicates that the AR headset has turned 95 degrees (90 degrees due to the movement of the vehicle + 5 degrees due to the user's movement). The user movement estimator determines the user portion of the movement of the AR headset based on a comparison of the movement of the AR headset (e.g., 95 degrees) and the movement of the vehicle (e.g., 90 degrees). For example, the user movement estimator determines that the user portion indicates a 5 degree movement relative to the vehicle based on determining that the net value of the movement of the AR headset and the movement of the vehicle is 5 degrees. In another example, the user movement estimator determines that the user portion indicates no movement based on determining that the movement of the AR headset matches the movement of the vehicle.

[0014] The gaze target estimator estimates the user's gaze target based on the moving user portion of the AR headset. The gaze target indicates where the user is looking at a particular time. In one specific example, the gaze target estimator determines the orientation and position of the AR headset relative to the vehicle based on the moving user portion of the AR headset. The gaze target estimator determines that the user is looking in the direction of the dashboard of the vehicle based on the orientation and position of the AR headset. In some embodiments, the headset sensor data includes image sensor data. In such an embodiment, the gaze target estimator performs image recognition on the image sensor data to determine that the gaze target includes a specific instrument of the dashboard. In another example, the gaze target may indicate that the user is looking in the direction of a location outside the vehicle (e.g., out of a window).

[0015] The visualization data generator generates visualization data for the AR headset based on the gaze target and the context at a specific time. For example, the context indicates whether the user is in a vehicle or outside the vehicle. In some examples, the context indicates the role of the user. For the same gaze target, the visualization data may be different based on the context. For example, for a user looking at a ship from inside an airplane, the visualization data may include an indicator identifying the ship, and for a user (e.g., a paratrooper) looking at the ship from outside an airplane, the visualization data may include a visual depiction of an anemometer. As another example, when the user's gaze target is inside a vehicle, the visualization data may include a three-dimensional rotatable map with a visual depiction of an object, and when the user's gaze target is outside a vehicle, the visualization data may include an indicator marking the location of the object. In some examples, the visualization data may include an indicator marking the location of an object outside the vehicle, even though the object is not visible to the user (e.g., behind clouds or behind another object).

[0016] In one specific example, the visualization data reduces the mental workload of the user when visually moving from one context to another. For example, the visualization data includes a visual depiction of an object (e.g., a ship) in a first context (e.g., a map displayed inside a vehicle) and includes an indicator for marking the location of the object (e.g., the ship) in a second context (e.g., outside the vehicle). The indicator and the visual depiction together include at least one visual element, such as color, image, video, animation, symbol, text, or a combination thereof. Having similar visual elements for the same object helps the user detect the corresponding object in different contexts.

[0017] The accompanying drawings and the following description illustrate specific exemplary embodiments. It should be understood that those skilled in the art will be able to design various arrangements that, although not explicitly described or shown herein, embody the principles described herein and are included within the scope of the claims appended to this specification. In addition, any examples described herein are intended to aid in understanding the principles of the present disclosure and should be interpreted as not limiting. Therefore, the present disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0018] Specific embodiments are described herein with reference to the accompanying drawings. In the specification, common features are denoted by common reference numerals throughout the drawings. As used herein, various terms are used only for the purpose of describing specific embodiments and are not intended to be limiting. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some embodiments and plural in other embodiments. For illustration, Figure 1 Depicts a system including one or more processors ( Figure 1 134), indicating that in some embodiments, the device 102 includes a single processor 134, and in other embodiments, the device 102 includes multiple processors 134. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular unless aspects relating to multiple features are being described.

[0019] The term "comprising" and its variants are used interchangeably with "including" and its variants. In addition, the term "wherein" is used interchangeably with the term "in the case of...". As used herein, "exemplary" indicates examples, embodiments and / or aspects, and should not be interpreted as limiting or indicating preferences or preferred embodiments. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) for modifying elements such as structures, components, operations, etc. do not themselves indicate any priority or order of an element relative to another element, but merely distinguish an element from another element with the same name (except for the use of ordinal numbers). As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to a plurality of elements.

[0020] As used herein, "generate", "calculate", "use", "select", "access" and "determine" are interchangeable unless the context indicates otherwise. For example, "generate", "calculate" or "determine" a parameter (or signal) may refer to actively generating, calculating or determining a parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) that has already been generated, such as by other components or devices. As used herein, "coupling" may include "communicative coupling", "electrical coupling" or "physical coupling", and may also (or alternatively) include any combination thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronics, one or more connectors, or inductive coupling. In some embodiments, two devices (or components) that are communicatively coupled (e.g., in electrical communication) can send and receive electrical signals (digital or analog) directly or indirectly (e.g., via one or more wires, buses, networks, etc.). As used herein, "directly coupled" is used to describe two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intermediate components.

[0021] Figure 1 1 is a block diagram of a system 100 operable to perform augmented reality visualization. The system 100 includes a device 102 coupled to an augmented reality (AR) headset 108 and a vehicle 104 via a network 190. The network 190 includes a wired network, a wireless network, or both. The AR headset 108 includes (or is coupled to) one or more headset sensors 131. The headset sensors 131 include accelerometers, gyroscopes, magnetometers, inertial measurement units, image sensors, global positioning system (GPS) receivers, beacons, or a combination thereof. The headset sensors 131 are configured to generate headset sensor (HS) data 141. The vehicle 104 includes (or is coupled to) one or more vehicle sensors 162. The vehicle sensors 162 include accelerometers, gyroscopes, magnetometers, inertial measurement units, image sensors, GPS receivers, beacons, or a combination thereof. The vehicle sensors 162 are configured to generate vehicle sensor (VS) data 159.

[0022] It should be noted that in the following description, Figure 1The various functions performed by the system 100 are described as being performed by certain components or modules. However, this division of components and modules is for illustration only. In alternative aspects, the functions described herein as being performed by a particular component or module are divided among multiple components or modules. Furthermore, in alternative aspects, Figure 1 Two or more components or modules of a system are integrated into a single component or module. Figure 1 Each component or module shown in the may be implemented using hardware (e.g., a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a controller, etc.), software (e.g., instructions executable by a processor), or any combination thereof.

[0023] The device 102 includes a memory 132, one or more processors 134, and an interface 130. The interface 130 includes a communication interface, a network interface, an application program interface, or a combination thereof. The memory 132 is configured to store data 103 used (e.g., generated) by the processor 134. In some aspects, a portion of the data 103 is stored in the memory 132 at any given time. The interface 130 is configured to communicate with the network 190. The processor 134 includes a user movement estimator 140, a gaze target estimator 142, a scenario estimator 152, a visualization data generator 154, or a combination thereof. The user movement estimator 140 is configured to estimate the headset movement 109 of the AR headset 108 based on the HS data 141, and to estimate the vehicle movement 105 of the vehicle 104 based on the VS data 159. The user movement estimator 140 is configured to estimate the user movement 111 based on a comparison of the headset movement 109 and the vehicle movement 105. User movement 111 corresponds to the user portion of headset movement 109 that is caused by movement of the head of user 101 of the AR headset 108 rather than by vehicular movement 105 .

[0024] The gaze target estimator 142 is configured to estimate a gaze target 127 (e.g., a location of gaze) of the user 101 based on the user movement 111. In a particular example, the gaze target estimator 142 is configured to determine a headset orientation 115, a headset position 113, or both of the AR headset 108 relative to the vehicle 104 based on the user movement 111. To illustrate, the user movement 111 indicates that the AR headset 108 has moved a particular angle (e.g., 5 degrees), a particular distance, or both relative to the vehicle 104. The headset orientation 115 indicates that the user 101 is looking in a particular direction relative to the vehicle 104 (e.g., 10 degrees in front of the vehicle 104). The headset position 113 indicates the location of the AR headset 108 relative to the vehicle 104 (e.g., in the vehicle 104).

[0025] In one particular aspect, gaze target estimator 142 is configured to determine gaze target 127 based on headset orientation 115, headset position 113, or both. For example, gaze target estimator 142 determines that gaze target 127 includes a particular instrument panel of vehicle 104. As another example, gaze target estimator 142 determines that gaze target 127 includes a window of vehicle 104.

[0026] In one particular aspect, gaze target estimator 142 is configured to determine (e.g., refine) gaze target 127 by performing image recognition on HS data 141. For example, gaze target estimator 142 is configured to determine, based on image recognition, that gaze target 127 includes a particular instrument of a dashboard. As another example, gaze target estimator 142 is configured to determine, based on image recognition, that gaze target 127 includes a particular object outside of vehicle 104 that is visible through a window.

[0027] The context estimator 152 is configured to determine the context 137. For example, the context 137 includes a user role 139 of the user 101, a user location context 121 (e.g., whether the user 101 is inside or outside the vehicle 104), a gaze target context 123 (e.g., the location of an object the user is looking at), or a combination thereof. The visualization data generator 154 is configured to generate visualization data 155 based on the gaze target 127 (e.g., what the user 101 is looking at) and the context 137. For example, the visualization data 155 includes one or more visual elements 163 (e.g., virtual elements) that are selected based on the gaze target 127 and the context 137. The visualization data generator 154 is configured to send the visualization data 155 to a display of the AR headset 108.

[0028] During operation, the user 101 activates (e.g., powers up) the AR headset 108. The headset sensors 131 generate HS data 141. For example, the headset sensors 131 generate HS data 141 during a first time range. To illustrate, one or more of the headset sensors 131 generate data during the first time range continuously, at various time intervals, in response to detecting an event, or a combination thereof. The event may include receiving a request from the device 102, receiving a user input, detecting movement of the AR headset 108, or a combination thereof.

[0029] The device 102 receives the HS data 141 from the AR headset 108. For example, the AR headset 108 sends the HS data 141 to the device 102 continuously, at various time intervals, in response to detecting an event, or a combination thereof. The event may include receiving a request from the device 102, receiving user input, detecting movement of the AR headset 108, detecting an update to the HS data 141, or a combination thereof.

[0030] The vehicle sensors 162 generate the VS data 159. For example, the vehicle sensors 162 generate the VS data 159 during a second time range. In one particular aspect, the second time range is the same as, overlaps with, or is within a threshold duration (e.g., 5 minutes) of the first time range. To illustrate, one or more sensors of the vehicle sensors 162 generate data during the second time range continuously, at various time intervals, in response to detecting an event, or a combination thereof. The event can include receiving a request from the device 102, receiving a user input, detecting movement of the vehicle 104, or a combination thereof.

[0031] The device 102 receives the VS data 159 from the vehicle 104. For example, the vehicle 104 sends the VS data 159 to the device 102 continuously, at various time intervals, in response to detecting an event, or a combination thereof. The event may include receiving a request from the device 102, receiving user input, detecting movement of the vehicle 104, detecting an update to the VS data 159, or a combination thereof.

[0032] The scenario estimator 152 retrieves HS data 141 corresponding to a first time range (e.g., 10:10 AM–10:11 AM) from the memory 132. For example, the HS data 141 includes sensor data time-stamped with a time during the first time range (e.g., 10:10:03 AM). The scenario estimator 152 retrieves VS data 159 corresponding to a second time range (e.g., 10:09 AM–10:11 AM) from the memory 132. For example, the VS data 159 includes sensor data time-stamped with a time during the second time range (e.g., 10:10:07 AM). In response to determining that the first time range matches (e.g., overlaps) the second time range, the scenario estimator 152 determines that the HS data 141 corresponds to the VS data 159. In one specific example, in response to determining that the first time range is the same as the second time range, the scenario estimator 152 determines that the first time range matches the second time range. In another example, in response to determining that the first time range overlaps with the second time range, the scenario estimator 152 determines that the first time range matches the second time range. In a specific example, in response to determining that a duration between an end of one of the first time range or the second time range and a start of the other of the first time range or the second time range is within a threshold duration (e.g., 10 seconds or 1 minute), the scenario estimator 152 determines that the first time range matches the second time range.

[0033] The context estimator 152 determines the context 137 based on the HS data 141, the VS data 159, or both. For example, in response to determining that the HS data 141 corresponds to the VS data 159, the context estimator 152 determines the context 137 based on the HS data 141 and the VS data 159. In one particular aspect, the context estimator 152 determines a user location context 121 indicating whether the user 101 is in any vehicle. In one particular embodiment, in response to determining that the VS data 159 indicates that a tag associated with the user 101 is detected in the vicinity of a first sensor in the vehicle 104 (e.g., less than 6 inches), the context estimator 152 generates a user location context 121 indicating that the user 101 is in the vehicle 104. Alternatively, in response to determining that the VS data 159 indicates that a tag associated with the user 101 is detected in the vicinity of a second sensor outside the vehicle 104 (e.g., on an exterior surface of the vehicle 104) (e.g., less than 6 inches), the context estimator 152 generates a user location context 121 indicating that the user 101 is not in the vehicle 104. In an alternative embodiment, in response to determining that a first user position of user 101 indicated by HS data 141 is within a threshold distance (e.g., 6 inches) of a first vehicle position of vehicle 104 indicated by VS data 159, image recognition performed on image sensor data of HS data 141 and matching it to an interior of vehicle 104, or both, the scenario estimator 152 generates a user position context 121 indicating that user 101 is in vehicle 104.

[0034] In one particular aspect, the user 101 is considered to be “in” the vehicle 104 when movement of the vehicle 104 is likely to cause movement of the AR headset 108 worn by the user 101 independently of any movement caused by the user 101. In one particular example, the user 101 is standing on the roof of the vehicle 104 and is considered to be “in” the vehicle 104. In another example, the user 101 is standing with their feet on the ground outside the vehicle 104 and looking inside the vehicle 104 through an open window with their head, and the user 101 is considered not to be “in” the vehicle 104.

[0035] The context estimator 152 determines a user role 139 for the user 101. In one particular example, the user role 139 is relatively static and is indicated by context configuration data 167. In one particular aspect, the context configuration data 167 is based on default data, configuration data, user input, or a combination thereof. In an alternative example, the user role 139 is dynamic and is based at least in part on the user location context 121. To illustrate, the context configuration data 167 indicates that the user 101 has a first role inside the vehicle 104, and a second role outside the vehicle 104.

[0036] In response to determining that the user location context 121 indicates that the user 101 is in the vehicle 104, the user movement estimator 140 determines the user movement 111 based on the comparison of the HS data 141 and the VS data 159. In a specific example, the user movement estimator 140 determines the headset movement 109 indicated by the HS data 141. For example, the HS data 141 indicates that the AR headset 108 moves along a specific direction, a specific distance, a specific rotation angle, or a combination thereof relative to the environment (e.g., geographic coordinates). The headset movement 109 indicates a specific direction, a specific distance, a specific rotation angle, or a combination thereof. In a specific aspect, the specific rotation angle indicates the yaw, pitch, roll, or a combination thereof of the user 101's head relative to the environment. The user movement estimator 140 determines the vehicle movement 105 indicated by the VS data 159. For example, the VS data 159 indicates that the vehicle 104 moves along a specific direction, a specific distance, a specific rotation angle, or a combination thereof relative to the environment (e.g., geographic coordinates). In one particular aspect, the particular rotation angle indicates a yaw, pitch, roll, or a combination thereof of the vehicle 104 relative to the environment. The vehicle movement 105 indicates a particular direction, a particular distance, a particular rotation angle, or a combination thereof.

[0037] In response to determining that the user location context 121 indicates that the user 101 is in the vehicle 104, the user movement estimator 140 determines the user movement 111 based on a comparison of the headset movement 109 and the vehicle movement 105. For example, the user movement estimator 140 determines the user movement 111 based on the net value of the headset movement 109 and the vehicle movement 105. In one particular aspect, the headset movement 109 includes a first portion caused by the movement of the user 101 and a second portion caused by the movement of the vehicle 104. For example, the user 101 wearing the AR headset 108 moves his head (e.g., turns 5 degrees to the left), and the vehicle 104 turns (e.g., turns 90 degrees to the left). The headset movement 109 indicates that the AR headset 108 moves relative to the environment (e.g., 95 degrees). The user movement 111 (e.g., 5 degrees to the left) indicates the first portion (e.g., caused by the movement of the user 101's head and not caused by the vehicle movement 105).

[0038] In one particular aspect, in response to determining that the user location context 121 indicates that the user 101 is not in the vehicle 104 (e.g., not in any vehicle), the user movement estimator 140 determines the user movement 111 based on the HS data 141 and independently of the VS data 159. For example, in response to determining that the user location context 121 indicates that the user 101 is not in the vehicle 104 (e.g., not in any vehicle), the user movement estimator 140 specifies the headset movement 109 as the user movement 111.

[0039] In response to determining that the user location context 121 indicates that the user 101 is in the vehicle 104, the user movement estimator 140 determines a headset orientation 115 and a headset position 113 of the AR headset 108 relative to the vehicle 104. For example, the user movement estimator 140 determines the headset orientation 115, the headset position 113, or both based on the user movement 111. In one specific example, the user movement 111 indicates a particular rotation angle (e.g., a net rotation angle) relative to the vehicle 104, and the user movement estimator 140 determines the headset orientation 115 based on the particular rotation angle. To illustrate, in response to determining that the AR headset 108 previously had a first headset orientation relative to the vehicle 104 (e.g., the user 101 was looking toward the front of the vehicle 104), the user movement estimator 140 determines a headset orientation 115 (e.g., -5 degrees or 355 degrees) by applying a specific rotation angle (e.g., 5 degrees to the left) to the first headset orientation (e.g., 0 degrees). In a specific example, the user movement 111 indicates a specific distance, a specific direction, or both relative to the vehicle 104, and the user movement estimator 140 determines the headset position 113 based on the specific distance, the specific direction, or both. To illustrate, in response to determining that the AR headset 108 previously had a first headset position (e.g., a first coordinate) relative to the vehicle 104, the user movement estimator 140 determines the headset position 113 (e.g., a second coordinate) relative to the vehicle 104 by applying the specific distance, the specific direction, or both (e.g., a coordinate increment) to the first headset position (e.g., the first coordinate). The user movement estimator 140 stores the headset position 113 , the headset orientation 115 , or both in the memory 132 .

[0040] In one specific example, the HS data 141 indicates that the user 101 is looking in a first global direction (e.g., west), and the VS data 159 indicates that the vehicle 104 is oriented in a second global direction (e.g., north). The user movement estimator 140 determines that the user 101 is looking in a specific user-vehicle direction relative to the vehicle 104 (e.g., toward the left side of the vehicle 104) based on a comparison of the first global direction and the second global direction. The headset orientation 115 indicates the specific user-vehicle direction.

[0041] In one particular example, HS data 141 indicates that user 101 is located at a first global location, and VS data 159 indicates that the center of vehicle 104 is located at a second global location. User movement estimator 140 determines that user 101 is located at a specific user-vehicle location relative to the center of vehicle 104 based on a comparison of the first global location and the second global location. Headset location 113 indicates the specific user-vehicle location.

[0042] In response to determining that the user location context 121 indicates that the user 101 is not in the vehicle 104, the gaze target estimator 142 determines the gaze target 127 based on the HS data 141, the user movement 111, or both. In a specific example, the HS data 141 indicates the global location of the user 101 (e.g., GPS coordinates), the global direction that the user 101 is looking at (e.g., compass direction, elevation, or both), or both. In another example, the HS data 141 indicates the user movement 111. In this example, the gaze target estimator 142 determines the global location, the global direction, or both by applying the user movement 111 to the previous global location, the previous global direction, or both of the user 101. In a specific aspect, the gaze target estimator 142 identifies the gaze target 127 in response to determining that the map data 135 indicates that the gaze target 127 is associated with the global location, the global direction, or both. In a specific aspect, the gaze target estimator 142 performs image recognition by comparing the image sensor data of the HS data 141 and an image of an object associated with the global location, the global direction, or both. In this regard, gaze target estimator 142 identifies gaze target 127 in response to determining, based on the image recognition, that an image of gaze target 127 matches the image sensor data.

[0043] In one particular aspect, in response to determining that user location context 121 indicates that user 101 is in vehicle 104, gaze target estimator 142 determines gaze target 127 based on HS data 141, headset location 113, headset orientation 115, or a combination thereof. In one particular example, gaze target 127 indicates what user 101 is looking at (e.g., an object). Gaze target estimator 142 performs image recognition on image sensor data of HS data 141 (e.g., captured by a camera of headset sensor 131) to identify a specific object. For illustration, gaze target estimator 142 determines that vehicle map data 133 indicates that a gaze target area (e.g., a specific instrument panel) corresponds to headset location 113, headset orientation 115, or both. Vehicle map data 133 includes images of objects (e.g., specific instruments) in a gaze target area (e.g., a specific instrument panel) located in vehicle 104. For example, vehicle map data 133 includes one or more first images of a specific object. In response to determining that the image recognition indicates that the image sensor data matches the first image, gaze target estimator 142 identifies the particular object as gaze target 127. In response to determining that vehicle map data 133 indicates that the particular object is in vehicle 104, gaze target estimator 142 generates gaze target context 123 indicating that gaze target 127 is in vehicle 104.

[0044] In one specific example, gaze target estimator 142 determines that user 101 is looking at a specific object outside vehicle 104. To illustrate, in response to determining that vehicle map data 133 indicates that headset position 113, headset orientation 115, or both correspond to an open or see-through portion (e.g., a window) of vehicle 104, gaze target estimator 142 determines that user 101 is looking outside. In one specific example, gaze target estimator 142 identifies the specific object (outside vehicle 104) as gaze target 127 by performing image recognition. In one specific example, gaze target estimator 142 identifies the specific object outside vehicle 104 based on a global location (e.g., geographic coordinates) and a global direction (e.g., east) of user 101. For example, based on map data 135, gaze target estimator 142 identifies a global area (e.g., a segment on the left side of a specific road) based on the global location and the global direction. In response to determining based on image recognition that the image sensor data of HS data 141 matches an image of the specific object, gaze target estimator 142 designates the specific object as gaze target 127. Gaze target estimator 142 generates gaze target context 123 indicating that gaze target 127 is outside of vehicle 104. In one particular aspect, gaze target context 123 indicates a location of gaze target 127 (e.g., GPS coordinates).

[0045] The visualization data generator 154 generates visualization data 155 based on the HS data 141, the gaze target 127, the context 137, the context configuration data 167, or a combination thereof. For example, the context configuration data 167 indicates data corresponding to the gaze target 127, the context 137, or both. In a specific example, the context configuration data 167 indicates that the first data (e.g., a three-dimensional map with the position of the ship within a specific distance of the vehicle 104) corresponds to the user role 139 (e.g., navigator), the user position context 121 (e.g., inside the vehicle 104), the gaze target 127 (e.g., the interior portion of the vehicle 104), the gaze target context 123 (e.g., inside the vehicle 104), or a combination thereof. In a specific aspect, the first part of the first data is relatively static (e.g., the position of a landmark in the three-dimensional map), and the second part of the first data is dynamic (e.g., the position of the ship). For example, the second part of the first data is received from one or more systems of the vehicle 104 (e.g., a radar system, a communication system, or both).

[0046] Visualization data generator 154 generates visualization data 155 based on first data to include one or more visual elements 163. For example, in response to determining that gaze target context 123 indicates that gaze target 127 is inside vehicle 104, visualization data generator 154 generates visualization data 155 to include visual element 163, which is to be displayed to a user in vehicle 104 who has user role 139 (e.g., navigator) and is looking in the direction of a specific object (e.g., a specific instrument) of vehicle 104.

[0047] In one particular aspect, the visual element 163 includes a first visual depiction (e.g., a virtual representation) of a point of interest (POI) 164 (e.g., a ship) located outside the vehicle 104, a second visual indicator indicating the location of a POI 166 (e.g., a particular instrument) located inside the vehicle 104, or both. As used herein, a "POI" refers to an object, landmark, location, person, vehicle, data, or a combination thereof to be indicated in the visualization data 155. In one particular example, the context configuration data 167 indicates an object, landmark, location, person, vehicle, or a combination thereof as a POI.

[0048] exist Figure 2AIn the example shown, the visual element 163 includes a visual depiction 203 (e.g., a virtual representation) of a POI 164 (e.g., a ship) outside the vehicle 104, a visual indicator 221 indicating the location of a POI 166 (e.g., a specific instrument) inside the vehicle 104, or both. The visual depiction 203, the visual indicator 221, or both include one or more visual elements. In a specific aspect, the visual element includes a symbol, a shape, a fill, a label, an image, an animation, a video, text, or a combination thereof. In a specific aspect, the visual element indicates information (e.g., an identifier, specifications, remaining fuel, passenger information, cargo information, location, or a combination thereof) of the corresponding POI (e.g., POI 164 or POI 166). In a specific aspect, the visual element includes a selectable option associated with the corresponding POI (e.g., POI 164 or POI 166). For example, the user 101 can select a selectable option associated with the POI 164, for example, to communicate with the ship.

[0049] In one particular aspect, visual depictions 203 are generated to display inside vehicle 104 a virtual representation of POI 164 (e.g., a boat) outside vehicle 104. In one particular aspect, visual element 163 includes visual depictions of a plurality of POIs outside vehicle 104, and visual element 163 is generated to display relative locations of the plurality of POIs inside vehicle 104. For example, if a POI 164 (e.g., a boat) is twice as far away from a first POI (e.g., the first boat) as it is from a second POI (e.g., a mountain) outside vehicle 104, visual depictions 203 of POI 164 are generated to be displayed inside vehicle 104 as a first visual depiction of the first POI that is twice as far away from the second visual depiction of the second POI.

[0050] In one particular aspect, when a POI 166 (e.g., a particular instrument) is in the field of view of a user 101 wearing the headset 108, a visual indicator 221 is generated for display by the AR headset 108 to indicate the location of the POI 166 within the vehicle 104. For example, the visualization data generator 154 determines the location of the POI 166 inside the vehicle 104 based on the vehicle map data 133, the HS data 141, or both. To illustrate, the visualization data generator 154 performs image recognition on the image sensor data of the HS data 141 and determines the location of the POI 166 in the field of view of the user 101 of the AR headset 108. The visualization data generator 154 generates the visual indicator 221 such that the visual indicator 221 indicates the location of the POI 166 in the field of view of the user 101. For example, the visual indicator 221 is superimposed near the POI 166 (e.g., within one inch of the POI 166) in the field of view of the user 101.

[0051] return Figure 1 In response to determining that gaze target 127 is outside vehicle 104, visualization data generator 154 generates visualization data 155 to include visual element 163 to be displayed to a user having user role 139 (e.g., navigator) in vehicle 104 and looking in the direction of a particular object (e.g., a boat) outside vehicle 104. In one particular aspect, visual element 163 includes a first visual indicator indicating the location of POI 166 (e.g., a boat) located outside vehicle 104, a second visual depiction of POI 166 (e.g., a particular instrument) located inside vehicle 104, or both.

[0052] exist Figure 2B In the example shown, the visual element 163 includes a visual depiction 205 of a POI 166 (e.g., a particular instrument) inside the vehicle 104, a visual indicator 223 indicating the location of a POI 164 (e.g., a boat) outside the vehicle 104, or both. The visual depiction 205, the visual indicator 223, or both include one or more visual elements. In a particular embodiment, the visualization data generator 154 generates the visualization data 155 to reduce the mental workload of the user 101 when transitioning between the output of the AR headset 108 corresponding to the interior of the vehicle 104 and the output of the AR headset 108 corresponding to the exterior of the vehicle 104. For example, the visual depiction 203 and the visual indicator 223 together include at least one visual element. Figure 2A-2B In the example of , the common visual elements include shape and fill (e.g., diagonal fill). As another example, the visual depiction 205 and the visual indicator 221 include at least one visual element in common. Figure 2A-2BIn the example of , the common visual elements include shapes and text (e.g., wind speed). When the user 101 switches between looking at a gaze target inside the vehicle 104 and looking at a gaze target outside the vehicle 104, the common visual elements reduce the mental workload of the user 101 in identifying the visual depiction and the corresponding visual indicator.

[0053] The visualization data generator 154 sends the visualization data 155 to the display of the AR headset 108. The AR headset 108 generates an AR output indicating the visual element 163 based on the visualization data 155. In one particular aspect, the AR output can be manipulated based on user input. For example, the user 101 can rotate, zoom in, or zoom out of the three-dimensional map. In an illustrative example, the three-dimensional map includes a visual depiction of a moving object in low-visibility terrain, and the user 101 (e.g., a navigator) can use the three-dimensional map to account for the moving object and plan a path for the vehicle 104 to traverse the terrain.

[0054] The system 100 thus enables determination of the user portion of the movement of the AR headset that is caused by movement of the head of the user of the AR headset rather than by movement of the vehicle in which the user is located. The system 100 also enables visualization data to include visual elements selected based on context. The visual elements can be displayed in the user's field of view, thereby enabling the user to access relevant information without having to transition between contexts and without having to access a fixed display. When the user does transition between contexts, the mental workload associated with the transition is reduced.

[0055] refer to Figure 3 , a system operable to perform augmented reality visualization is shown and generally designated 300. In system 300, processor 134 is integrated into an AR headset 308. For example, visualization data generator 154 generates visualization data 155 based on HS data generated by AR headset 308 (e.g., headset sensor 131), VS data 159 received from vehicle 104 (e.g., vehicle sensor 162), or both.

[0056] and Figure 1 Integrating one or more of the user movement estimator 140, the gaze target estimator 142, the context estimator 152, or the visualization data generator 154 in the AR headset 308 reduces the number of devices used for augmented reality visualization compared to the system 100 of FIG. 308. Alternatively, integrating one or more of the user movement estimator 140, the gaze target estimator 142, the context estimator 152, or the visualization data generator 154 in the AR headset 308 reduces the number of devices used for augmented reality visualization. Figure 1The AR headset 108 may be configured to offload some augmented reality visualization tasks to the AR headset 108 and reduce resource consumption (e.g., memory, processing cycles, or both) of the AR headset 108. It should be understood that specific devices configured to perform specific operations are provided as illustrative examples. In other examples, one or more operations described herein may be performed by one or more devices.

[0057] Figure 4 An example of a method 400 for augmented reality visualization is shown. In one particular aspect, the method 400 comprises Figure 1 The user movement estimator 140, the gaze target estimator 142, the scene estimator 152, the visualization data generator 154, the processor 134, the interface 130, the device 102, Figure 3 The AR headset 308 or a combination thereof is performed.

[0058] The method 400 includes receiving vehicle sensor data from one or more vehicle sensors coupled to a vehicle at 402. For example, as shown in FIG. Figure 1 As described, Figure 1 The interface 130 receives VS data 159 from a vehicle sensor 162. The vehicle sensor 162 is coupled to Figure 1 Means of transportation 104.

[0059] The method 400 also includes receiving headset sensor data from one or more headset sensors coupled to the augmented reality headset at 404. For example, as described with reference to Figure 1 As described, Figure 1 The interface 130 receives HS data 141 from the headphone sensor 131. The headphone sensor 131 is coupled to Figure 1 AR headset 108.

[0060] The method 400 also includes determining an orientation and position of the augmented reality headset relative to the vehicle based on the vehicle sensor data and the headset sensor data at 406. For example, as shown in FIG. Figure 1 As described, Figure 1 The gaze target estimator 142 determines a headset orientation 115 and a headset position 113 of the AR headset 108 relative to the vehicle 104 based on the VS data 159 and the HS data 141 .

[0061] In one particular aspect, method 400 includes determining movement of a vehicle based on vehicle sensor data. Figure 1 As described, Figure 1The user movement estimator 140 determines the vehicle movement 105 of the vehicle 104 based on the VS data 159 .

[0062] The method 400 also includes determining movement of the augmented reality headset based on the headset sensor data. Figure 1 As described, Figure 1 The user movement estimator 140 determines the headset movement 109 of the AR headset 108 based on the HS data 141 .

[0063] The method 400 also includes estimating the portion of the user's movement of the augmented reality headset that is caused by movement of the user's head of the augmented reality headset and not caused by movement of the vehicle based on a comparison of the movement of the vehicle with the movement of the augmented reality headset. Figure 1 As described, Figure 1 The user movement estimator 140 estimates the user movement 111 (e.g., the user portion of the headset movement 109) caused by the movement of the head of the user 101 of the AR headset 108 but not caused by the vehicle movement 105 based on the comparison of the vehicle movement 105 and the headset movement 109. Figure 1 As described, the headset orientation 115 and the headset position 113 are determined based on the user movement 111 .

[0064] The method 400 also includes estimating a gaze target of a user of the augmented reality headset based at least in part on the orientation and position of the augmented reality headset relative to the vehicle at 408. Figure 1 As described, Figure 1 The gaze target estimator 142 determines a gaze target 127 of the user 101 of the AR headset 108 based at least in part on the headset orientation 115 and the headset position 113 .

[0065] In one particular aspect, method 400 includes determining whether the user is in a vehicle based on the headset sensor data. Figure 1 As described, Figure 1 Context estimator 152 of HS determines whether user 101 is in vehicle 104 based on HS data 141 . In response to determining that user 101 is in vehicle 104 , gaze target estimator 142 estimates gaze target 127 based on headset orientation 115 and headset position 113 .

[0066] Method 400 also includes generating visualization data based on the gaze target at 410. For example, as shown in FIG. Figure 1 As described, Figure 1The visualization data generator 154 generates visualization data 155 based on the gaze target 127. Figure 1 2 , in response to determining that gaze target 127 is inside vehicle 104 , visualization data 155 includes visual depiction 203 of POI 164 outside vehicle 104 . Figure 1 As described in FIG. 2 , in response to determining that gaze target 127 is outside of vehicle 104 , visualization data 155 includes visual depiction 205 of POI 166 inside vehicle 104 .

[0067] The method 400 also includes sending the visualization data to a display of the augmented reality headset at 412. For example, as shown in FIG. Figure 1 As described, Figure 1 The visualization data generator 154 sends the visualization data 155 to the display of the AR headset 108 .

[0068] Thus, method 400 enables determination of a user portion of the movement of the AR headset that is caused by movement of the head of the user of the AR headset rather than by movement of the vehicle in which the user is located. Method 400 also enables visualization data to include visual elements selected based on context (e.g., whether the gaze target is inside or outside the vehicle). Visual elements can be displayed in the user's field of view, enabling the user to access relevant information without having to transition between contexts and without having to access a fixed display. When the user does transition between contexts, the mental workload associated with the transition is reduced.

[0069] Figure 5 is an illustration of a block diagram of a computing environment 500 including a computing device 510 that is configured to support various aspects of computer-implemented methods and computer-executable program instructions (or codes) according to the present disclosure. For example, the computing device 510 or a portion thereof is configured to execute instructions to initiate, execute, or control a reference Figures 1 to 4 In one particular aspect, the computing device 510 corresponds to Figure 1 device 102, AR headset 108, vehicle 104, Figure 3 AR headset 308 or a combination thereof.

[0070] The computing device 510 includes a processor 134. The processor 134 is configured to communicate with a system memory 530, one or more storage devices 540, one or more input / output interfaces 550, a transceiver 522, one or more communication interfaces 560, or a combination thereof. The system memory 530 includes volatile memory devices (e.g., random access memory (RAM) devices), non-volatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory), or both. The system memory 530 stores an operating system 532, which may include a basic input / output system for booting the computing device 510 and a complete operating system that enables the computing device 510 to interact with users, other programs, and other devices. The system memory 530 stores system (program) data 536. In one particular aspect, Figure 1 The memory 132 includes system memory 530, one or more storage devices 540, or a combination thereof. In one particular aspect, system (program) data 536 includes Figure 1 Data 103.

[0071] System memory 530 includes one or more application programs 534 that can be executed by processor 134. As an example, one or more application programs 534 include programs that can be executed by processor 134 to initiate, control, or execute reference programs. Figures 1 to 4 For illustration, one or more applications 534 include instructions executable by processor 134 to initiate, control, or perform one or more operations described with reference to user movement estimator 140, gaze target estimator 142, scene estimator 152, visualization data generator 154, or a combination thereof.

[0072] The processor 134 is configured to communicate with one or more storage devices 540. For example, one or more storage devices 540 include non-volatile storage devices, such as magnetic disks, optical disks, or flash memory devices. In a specific example, the storage devices 540 include both removable and non-removable memory devices. The storage devices 540 are configured to store operating systems, images of operating systems, applications, and program data. In a specific aspect, the system memory 530, the storage devices 540, or both include tangible computer-readable media. In a specific aspect, one or more of the storage devices 540 are external to the computing device 510.

[0073] The processor 134 is configured to communicate with one or more input / output interfaces 550, which enable the computing device 510 to communicate with one or more input / output devices 570 to facilitate user interaction. The processor 134 is configured to detect interaction events based on user input received via the input / output interface 550. The processor 134 is configured to communicate with a device or controller 580 via one or more communication interfaces 560. For example, the one or more communication interfaces 560 include Figure 1 The interface 130 of the embodiment of the present invention, and the device or controller 580 includes the AR headset 108, the headset sensor 131, the vehicle 104, the vehicle sensor 162, or a combination thereof. In one illustrative example, a non-transitory computer-readable storage medium (e.g., the system memory 530) includes instructions that, when executed by a processor (e.g., the processor 134), cause the processor to initiate, perform, or control an operation. The operation includes reference to Figures 1 to 4 Describes one or more operations.

[0074] although Figure 1-Figure 5 One or more of the embodiments illustrate systems, devices, and / or methods according to the teachings of the present disclosure, but the present disclosure is not limited to these illustrated systems, devices, and / or methods. Figure 1-Figure 5 Any one or more features or components of Figure 1-Figure 5 One or more other parts of another. For example, Figure 4 One or more elements of method 400 may be performed in combination with other operations described herein. Therefore, any single implementation described herein should not be interpreted as limiting, and the implementations of the present disclosure may be appropriately combined without departing from the teachings of the present disclosure. As an example, refer to Figures 1 to 5 One or more of the operations described may be optional, may be performed at least partially concurrently, and / or may be performed in an order different from that shown or described.

[0075] The above examples are illustrative and do not limit the present disclosure. It should be understood that many modifications and variations are possible according to the principles of the present disclosure. The description of the examples described herein is intended to provide a general understanding of the structure of various embodiments. These descriptions are not intended to be used as a complete description of all elements and features of the devices and systems utilizing the structures or methods described herein. After reviewing the present disclosure, many other embodiments may be obvious to those skilled in the art. Other embodiments may be utilized and may be derived from the present disclosure, so that structural and logical replacements and changes may be made without departing from the scope of the present disclosure. For example, the method operations may be performed in an order different from that shown in the accompanying drawings, or one or more method operations may be omitted. Therefore, the present disclosure and the accompanying drawings should be considered illustrative rather than restrictive.

[0076] In addition, although specific examples have been shown and described herein, it should be understood that any subsequent arrangement designed to achieve the same or similar results may replace the specific embodiments shown. This disclosure is intended to cover any and all subsequent modifications or changes of the various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon reviewing the specification.

[0077] Furthermore, the present disclosure includes embodiments according to the following clauses:

[0078] Clause 1. A device (102, 308) for augmented reality visualization, the device (102, 308) comprising:

[0079] An interface (130) configured to:

[0080] receiving vehicle sensor data (159) from one or more vehicle sensors (162) coupled to the vehicle (104); and

[0081] Receiving headset sensor data (141) from one or more headset sensors (131) coupled to an augmented reality headset (108); and one or more processors (134) configured to:

[0082] determining movement (105) of the vehicle (104) based on the vehicle sensor data (159);

[0083] Determining movement (109) of the augmented reality headset (108) based on the headset sensor data (141);

[0084] estimating a user portion (111) of the movement (109) of the augmented reality headset (108) based on a comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108), the user portion (111) being caused by movement of a head of a user (101) of the augmented reality headset (108) rather than by the movement (105) of the vehicle (104);

[0085] determining an orientation (115) and a position (113) of the augmented reality headset (108) relative to the vehicle (104) based on the user portion (111) of the movement (109) of the augmented reality headset (108);

[0086] estimating a gaze target (127) of the user (101) based on the headset sensor data (141) and the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104);

[0087] generating visualization data (155) based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) includes a first visual depiction (203) of a first point of interest (164) outside the vehicle (104), and wherein, in response to determining that the gaze target (127) is outside the vehicle, the visualization data (155) includes a second visual depiction (205) of a second point of interest (166) inside the vehicle (104); and

[0088] The visualization data (155) is sent to a display of the augmented reality headset (108).

[0089] Clause 2. The device (102, 308) of clause 1, wherein the one or more headset sensors (131) comprise an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit, an image sensor, a beacon, a global positioning system (GPS) receiver, or a combination thereof.

[0090] Clause 3. A device (102, 308) according to any preceding clause, wherein the vehicle sensor data (159) is associated with a first time range, wherein the headset sensor data (141) is associated with a second time range, and wherein the one or more processors (134) are further configured to perform the comparison of the movement (105) of the vehicle (104) indicated by the vehicle sensor data (159) and the movement (109) of the augmented reality headset (108) indicated by the headset sensor data (141) in response to determining that the first time range matches the second time range.

[0091] Clause 4. The device (102, 308) of any preceding clause, wherein the one or more processors (134) are further configured to:

[0092] determining whether the user (101) is in the vehicle (104) based on the headset sensor data (141); and

[0093] In response to determining that the user (101) is in the vehicle (104), the user portion (111) of the movement (109) of the augmented reality headset (108) is estimated based on the comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108).

[0094] Clause 5. A device (102, 308) according to any preceding clause, wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) also includes a first visual indicator (221) of the location of the second point of interest (166) inside the vehicle (104).

[0095] Clause 6. A device (102, 308) according to clause 5, wherein the first visual indicator (221) and the second visual depiction (205) together include at least one visual element, and wherein the visual element includes color, image, video, animation, symbol, text or a combination thereof.

[0096] Clause 7. A device (102, 308) according to any preceding clause, wherein, in response to determining that the gaze target (127) is outside the vehicle (104), the visualization data (155) also includes a second visual indicator (223) of the location of the first point of interest (164) outside the vehicle (104).

[0097] Clause 8. The device (102, 308) of clause 7, wherein the second visual indicator (223) and the first visual depiction (203) together comprise at least one visual element, and wherein the visual element comprises a color, an image, a video, an animation, a symbol, text, or a combination thereof.

[0098] Clause 9. The apparatus (102, 308) of any preceding clause, wherein the one or more processors are configured to:

[0099] estimating a gaze target area based on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104); and

[0100] The gaze target (127) is estimated by performing image recognition on the headset sensor data (141) and one or more images associated with the gaze target area.

[0101] Clause 10. A method of augmented reality visualization, the method comprising:

[0102] At a device (102, 308), receiving (402) vehicle sensor data (159) from one or more vehicle sensors (162) coupled to a vehicle (104);

[0103] At the device (102, 308), receiving (404) headset sensor data (141) from one or more headset sensors (131) coupled to an augmented reality headset (108);

[0104] Determining (406) an orientation (115) and a position (113) of the augmented reality headset (108) relative to the vehicle (104) based on the vehicle sensor data (159) and the headset sensor data (141);

[0105] estimating (408) a gaze target (127) of a user (101) of the augmented reality headset (108) based at least in part on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104);

[0106] generating (410) at the device (102, 308) visualization data (155) based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) includes a first visual depiction (203) of a first point of interest outside the vehicle (104), and wherein, in response to determining that the gaze target (127) is outside the vehicle, the visualization data (155) includes a second visual depiction (205) of a second point of interest (166) inside the vehicle (104); and

[0107] The visualization data (155) is sent (412) from the device (102, 308) to a display of the augmented reality headset (108).

[0108] Clause 11. The method according to clause 10, further comprising:

[0109] determining movement (105) of the vehicle (104) based on the vehicle sensor data (159);

[0110] determining movement (109) of the augmented reality headset (108) based on the headset sensor data (141); and

[0111] estimating a user portion (111) of the movement (109) of the augmented reality headset (108) based on a comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108), the user portion (111) being caused by movement of a head of a user (101) of the augmented reality headset (108) rather than by the movement (105) of the vehicle (104),

[0112] Wherein the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104) are determined based on the user portion (111) of the movement (109) of the augmented reality headset (108).

[0113] Clause 12. The method according to any one of clauses 10-11 further includes: determining whether the user (101) is in the vehicle (104) based on the headset sensor data (141), wherein, in response to determining that the user (101) is in the vehicle (104), estimating the gaze target (127) based on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104).

[0114] Clause 13. A method according to any of clauses 10-12, wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) also includes a first visual indicator (221) of the location of the second point of interest (166) inside the vehicle (104).

[0115] Clause 14. The method of clause 13, wherein the first visual indicator (221) and the second visual depiction (205) together comprise at least one visual element, and wherein the visual element comprises a color, an image, a video, an animation, a symbol, text, or a combination thereof.

[0116] Clause 15. A method according to any of clauses 10-14, wherein, in response to determining that the gaze target (127) is outside the vehicle (104), the visualization data (155) also includes a second visual indicator (223) of the location of the first point of interest (164) outside the vehicle (104).

[0117] Clause 16. The method of clause 15, wherein the second visual indicator (223) and the first visual depiction (203) together comprise at least one visual element, and wherein the visual element comprises a color, an image, a video, an animation, a symbol, text, or a combination thereof.

[0118] Clause 17. A computer-readable storage device (530) storing instructions that, when executed by one or more processors (134), cause the one or more processors (134) to perform operations comprising:

[0119] receiving vehicle sensor data (159) from one or more vehicle sensors (162) coupled to the vehicle (104);

[0120] receiving headset sensor data (141) from one or more headset sensors (131) coupled to the augmented reality headset (108);

[0121] estimating a gaze target (127) of a user (101) of the augmented reality headset (108) based on the vehicle sensor data (159) and the headset sensor data (141);

[0122] generating visualization data (155) based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) includes a first visual depiction (203) of a first point of interest outside the vehicle (104), and wherein, in response to determining that the gaze target (127) is outside the vehicle, the visualization data (155) includes a second visual depiction (205) of a second point of interest (166) inside the vehicle (104); and

[0123] The visualization data (155) is sent to a display of the augmented reality headset (108).

[0124] Clause 18. A computer-readable storage device (530) according to clause 17, wherein the operation further comprises: determining an orientation (115) and a position (113) of the augmented reality headset (108) relative to the vehicle (104) based on the vehicle sensor data (159) and the headset sensor data (141), wherein the gaze target (127) is based on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104).

[0125] Clause 19. The computer-readable storage device (530) of any of clauses 17-18, wherein the operations further comprise:

[0126] determining movement (105) of the vehicle (104) based on the vehicle sensor data (159);

[0127] determining movement (109) of the augmented reality headset (108) based on the headset sensor data (141); and

[0128] estimating a user portion (111) of the movement (109) of the augmented reality headset (108) based on a comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108), the user portion (111) being caused by movement of a head of a user (101) of the augmented reality headset (108) rather than by the movement (105) of the vehicle (104),

[0129] wherein the gaze target (127) of the user (101) is based on the user portion (111) of the movement (109) of the augmented reality headset (108).

[0130] Clause 20. The computer-readable storage device (530) of any one of clauses 17-19, wherein the operations further comprise:

[0131] estimating a gaze target area based on the headset sensor data (141) and the vehicle sensor data (159); and

[0132] The gaze target (127) is estimated by performing image recognition on the headset sensor data (141) and one or more images associated with the gaze target area.

[0133] The abstract of the present disclosure submitted should be understood that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above specific embodiments, various features can be combined together or described in a single embodiment for the purpose of simplifying the present disclosure. The examples described above illustrate but do not limit the present disclosure. It should also be understood that many modifications and variations are possible according to the principles of the present disclosure. As reflected in the attached claims, the subject matter claimed for protection may be directed to less than all features of any one of the disclosed examples. Therefore, the scope of the present disclosure is limited by the attached claims and their equivalents.

Claims

1. A device (102, 308) for augmented reality visualization, the device (102, 308) comprising: An interface (130) configured to: receiving vehicle sensor data (159) from one or more vehicle sensors (162) coupled to the vehicle (104); and receiving headset sensor data (141) from one or more headset sensors (131) coupled to an augmented reality headset (108); and One or more processors (134) configured to: determining movement (105) of the vehicle (104) based on the vehicle sensor data (159); Determining movement (109) of the augmented reality headset (108) based on the headset sensor data (141); estimating a user portion (111) of the movement (109) of the augmented reality headset (108) based on a comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108), the user portion (111) being caused by movement of a head of a user (101) of the augmented reality headset (108) rather than by the movement (105) of the vehicle (104); determining an orientation (115) and a position (113) of the augmented reality headset (108) relative to the vehicle (104) based on the user portion (111) of the movement (109) of the augmented reality headset (108); estimating a gaze target (127) of the user (101) based on the headset sensor data (141) and the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104); generating visualization data (155) based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) includes a first visual depiction (203) of a first point of interest (164) outside the vehicle (104), and wherein, in response to determining that the gaze target (127) is outside the vehicle, the visualization data (155) includes a second visual depiction (205) of a second point of interest (166) inside the vehicle (104); and The visualization data (155) is sent to a display of the augmented reality headset (108).

2. The device (102, 308) of claim 1, wherein the one or more headset sensors (131) include an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit, an image sensor, a beacon, a global positioning system receiver (GPS) receiver, or a combination thereof.

3. A device (102, 308) according to any preceding claim, wherein the vehicle sensor data (159) is associated with a first time range, wherein the headset sensor data (141) is associated with a second time range, and wherein the one or more processors (134) are further configured to perform the comparison of the movement (105) of the vehicle (104) indicated by the vehicle sensor data (159) and the movement (109) of the augmented reality headset (108) indicated by the headset sensor data (141) in response to determining that the first time range matches the second time range.

4. The device (102, 308) of any one of claims 1-2, wherein the one or more processors (134) are further configured to: determining whether the user (101) is in the vehicle (104) based on the headset sensor data (141); and In response to determining that the user (101) is in the vehicle (104), the user portion (111) of the movement (109) of the augmented reality headset (108) is estimated based on the comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108).

5. The device (102, 308) according to any one of claims 1-2, wherein: In response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) also includes a first visual indicator (221) of the location of the second point of interest (166) inside the vehicle (104).

6. The device (102, 308) of claim 5, wherein the first visual indicator (221) and the second visual depiction (205) together comprise at least one visual element, and wherein the visual element comprises a color, an image, a video, an animation, a symbol, text, or a combination thereof.

7. The device (102, 308) according to any one of claims 1-2, wherein: In response to determining that the gaze target (127) is outside the vehicle (104), the visualization data (155) also includes a second visual indicator (223) of the location of the first point of interest (164) outside the vehicle (104).

8. The device (102, 308) of claim 7, wherein the second visual indicator (223) and the first visual depiction (203) together comprise at least one visual element, and wherein the visual element comprises a color, an image, a video, an animation, a symbol, text, or a combination thereof.

9. The device (102, 308) of any one of claims 1-2, wherein the one or more processors are configured to: estimating a gaze target area based on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104); and The gaze target (127) is estimated by performing image recognition on the headset sensor data (141) and one or more images associated with the gaze target area.

10. A method for augmented reality visualization, the method comprising: At a device (102, 308), receiving (402) vehicle sensor data (159) from one or more vehicle sensors (162) coupled to a vehicle (104); At the device (102, 308), receiving (404) headset sensor data (141) from one or more headset sensors (131) coupled to an augmented reality headset (108); Determining (406) an orientation (115) and a position (113) of the augmented reality headset (108) relative to the vehicle (104) based on the vehicle sensor data (159) and the headset sensor data (141); estimating (408) a gaze target (127) of a user (101) of the augmented reality headset (108) based at least in part on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104); generating (410) at the device (102, 308) visualization data (155) based on the gaze target (127), wherein, in response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) includes a first visual depiction (203) of a first point of interest outside the vehicle (104), and wherein, in response to determining that the gaze target (127) is outside the vehicle, the visualization data (155) includes a second visual depiction (205) of a second point of interest (166) inside the vehicle (104); and The visualization data (155) is sent (412) from the device (102, 308) to a display of the augmented reality headset (108).

11. The method according to claim 10, further comprising: determining movement (105) of the vehicle (104) based on the vehicle sensor data (159); determining movement (109) of the augmented reality headset (108) based on the headset sensor data (141); and estimating a user portion (111) of the movement (109) of the augmented reality headset (108) based on a comparison of the movement (105) of the vehicle (104) and the movement (109) of the augmented reality headset (108), the user portion (111) being caused by movement of a head of a user (101) of the augmented reality headset (108) rather than by the movement (105) of the vehicle (104), Wherein the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104) are determined based on the user portion (111) of the movement (109) of the augmented reality headset (108).

12. The method according to any one of claims 10-11, further comprising: Based on the headset sensor data (141), it is determined whether the user (101) is in the vehicle (104), wherein, in response to determining that the user (101) is in the vehicle (104), the gaze target (127) is estimated based on the orientation (115) and the position (113) of the augmented reality headset (108) relative to the vehicle (104).

13. The method according to any one of claims 10 to 11, wherein: In response to determining that the gaze target (127) is inside the vehicle (104), the visualization data (155) also includes a first visual indicator (221) of the location of the second point of interest (166) inside the vehicle (104).

14. The method of claim 13, wherein the first visual indicator (221) and the second visual depiction (205) together comprise at least one visual element, and wherein the visual element comprises a color, an image, a video, an animation, a symbol, text, or a combination thereof.

15. The method according to any one of claims 10 to 11, wherein: In response to determining that the gaze target (127) is outside the vehicle (104), the visualization data (155) also includes a second visual indicator (223) of the location of the first point of interest (164) outside the vehicle (104), or optionally, wherein the second visual indicator (223) and the first visual depiction (203) together include at least one visual element, and wherein the visual element includes color, image, video, animation, symbol, text, or a combination thereof.

Images