Eliminating user interface element jitter in an augmented reality user interface

Abstract

In one embodiment, a computer-implemented method includes generating an augmented reality user interface, determining a position of a real-world object relative to a vehicle, generating an augmented reality element positioned in the augmented reality user interface based on the position of the real-world object, wherein a portion of the augmented reality element is animated, and projecting, using an augmented reality system, the augmented reality user interface onto a windshield of the vehicle.

Description

P240414WO (HRMN0513PC)ELIMINATING USER INTERFACE ELEMENT JITTER IN AN AUGMENTED REALITY USER INTERFACECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority benefit to U.S. provisional application titled “DECREASING VISUAL ELEMENT JITTER BY ANIMATING DISPLAY CONTENT,” filed on September 26, 2024 and having serial number 63 / 699,664. This related application is also hereby incorporated by reference in its entirety.TECHNICAL FIELD

[0001] The present disclosure relates generally to augmented reality systems and, more specifically, to eliminating user interface element jitter in an augmented reality user interface.BACKGROUND

[0002] A vehicle can be equipped with a heads-up display (HUD) system for presenting information to an occupant (e.g., a driver, an operator, a passenger) of the vehicle. The HUD system presents information in a way that allows the occupant to continue looking forward, toward the environment in front of the vehicle, without needing to look down toward an instrument panel, dashboard, or the like. Vehicles of different types can implement a heads-up display to maintain the attention of an occupant on the environment in front of the vehicle.

[0003] In a HUD system, content is typically projected onto a transparent or semi-transparent object (e.g., a windshield of the vehicle or a transparent display positioned between the occupant and the windshield), and the content reflects from the object toward the user. Some content in a HUD system is projected in a fixed position, meaning the position of the projected content is fixed in the display area although the content itself might periodically change. Some examples of fixed content include vehicle speed, speed limits, turn signals, or vehicle warnings such as engine temperature, low fuel, and low charge. In other examples, a HUD system integrates augmented reality in the projected display and overlays detailed information, such as navigation instructions, lane lines, object detection indications and warnings, onto real -world objects viewed through the display. Augmented reality content must be appropriately aligned with real world objects in the display, and as the position of real -world objects change relative to the vehicle, the position of projected augmented reality content must also change to maintain alignment.P240414WO (HRMN0513PC)

[0004] A HUD system utilizes various sensor signals as inputs to determine vehicle location and to identify the presence and position of real -world objects. The various sensor signals are processed to render projected augmented reality content. A drawback of a HUD system is that there is latency between detection of the object in the real world and when the capture, transmission and processing of the data associated with the object allows augmented reality content to be displayed to a user. The latency causes jitter that is characterized by the displayed augmented reality content failing to closely align with real -world objects as seen from the vehicle. If the delay is significant enough, and rapid events such as deceleration or acceleration occur, a user perceives the misalignment. The effects of jitter and misalignment can result in occupant discomfort, and in some cases, motion sickness.

[0005] What is needed is an improved technique for displaying augmented reality content overlayed on real-life objects while reducing the perceivable effects of latency, misalignment and jitter that are experienced by a vehicle occupant.SUMMARY

[0006] Consistent with some embodiments, a computer-implemented method includes generating an augmented reality user interface, determining a position of a real -world object relative to a vehicle, generating an augmented reality element positioned in the augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated, and projecting, using an augmented reality system, the augmented reality user interface onto a windshield of the vehicle.

[0007] Other embodiments include, without limitation, one or more non-transitory computer-readable media storing a plurality of computer-readable instructions, which when executed by a computing device, are adapted to cause the one or more processors to perform any of the methods disclosed herein.

[0008] At least one technical advantage of the disclosed techniques relative to the prior art is that an occupant of a vehicle that is moving can experience augmented reality content such as graphical display elements overlaid onto real world objects with a reduction of the perceivable effects of latency, misalignment and jitter of the graphical display elements relative to the real- world objects. By reducing the perceivable effects of latency, misalignment and jitter of the elements that are displayed in an augmented reality user interface, reduced discomfort of the vehicle occupant is achieved as well as an improved user experience in the use of a heads-up display system.P240414WO (HRMN0513PC)

[0009] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] So that the manner in which the above recited features of the various embodiments can be understood in detail, a more particular description of the inventive concepts, briefly summarized above, can be had by reference to various embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of scope in any way, and that there are other equally effective embodiments.

[0011] FIG. 1 is a block diagram of an augmented reality system configured to implement one or more aspects of the present disclosure.

[0012] FIG. 2 is a simplified diagram of an augmented reality user interface, according to some embodiments of the present disclosure.

[0013] FIGS. 3 A-3C are simplified diagrams of an augmented reality user interface showing a progression of graphical navigation instructions.

[0014] FIGS. 4A-4D are simplified diagrams of an example of a progression of animation in a navigation direction element which reduces the perception of jitter.

[0015] FIGS. 5A-5C are simplified diagrams of an alternate example of a progression of animation in a navigation direction element which reduces the perception of jitter.

[0016] FIG. 6 is a flowchart illustrating a method for displaying an animated augmented reality object aligned with a real -world object.DETAILED DESCRIPTION

[0017] This description and the accompanying drawings that illustrate inventive aspects, embodiments, or modules should not be taken as limiting — the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or describedP240414WO (HRMN0513PC) in detail in order not to obscure the invention. Like numbers in two or more figures represent the same or similar elements.

[0018] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0019] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.System Overview

[0020] Figure 1 is a schematic diagram of an augmented reality system 100, according to some embodiments. As shown in FIG. 1, augmented reality system 100 includes, without limitation, a light source 102, a display panel 104, a computing device 110, a windshield 106, a power source 130, and a light source driver 134. Computing device 110 includes, without limitation, a processing unit 112 and a memory 114. Memory 114 includes, without limitation, a HUD controller 122. In some embodiments, augmented reality system 100 also includes one or more sensors 124 such as light detection and ranging sensors, radio detection and ranging sensors, global positioning system (GPS) sensors, inertial measurement sensors, geomagnetic field sensors, cameras, and so forth.

[0021] For explanatory purposes, multiple instances of like objects are denoted with reference numbers identifying the object and additional numbers in parentheses identifying the instance where needed. Further, the augmented reality system 100 can include multiple instances of elements, even when not shown. For example, augmented reality system 100 can include multiple light sources (e.g., 102(1), 102(2), 102(3), etc.) and still be within the scope of the disclosed embodiments.P240414WO (HRMN0513PC)

[0022] In operation, HUD controller 122 controls power source 130 to provide power to display panel 104 and light source driver 134. In turn, light source driver 134 provides power to light source 102. HUD controller 122 provides image data to display panel 104, and display panel 104 displays a resulting image. Light source 102 provides a backlight that combines with the image data to generate a light pattern 152. Light pattern 152 reflects off of windshield 106 and reaches the eye of the user as a reflected light pattern 154.

[0023] Computing device 110 includes processing unit 112 and memory 114. In various embodiments, computing device 110 is a device that includes one or more processing units 112, such as a system-on-a-chip (SoC). In some embodiments, computing device 110 is a mobile computing device, such as a tablet computer, mobile phone, media player, and so forth that wirelessly connects to other devices in the vehicle. In some embodiments, computing device 110 can be a head unit or part of a head unit included in a vehicle system. In some embodiments, computing device 110 can be split among multiple physical devices in one or more locations. For example, one or more remote devices (e.g„ cloud servers, remote services, etc.) can perform one or more aspects of the disclosed techniques, such as eye tracking, media generation, and so forth. Additionally or alternatively, in some examples, computing device 110 is a detachable device that is mounted in a portion of a vehicle as part of an individual console. In some embodiments, computing device 110 is configured to coordinate the overall operation of augmented reality system 100. The embodiments disclosed herein can utilize any technically- feasible system configured to implement the functionality of augmented reality system 100 using computing device 110. The functionality and techniques of augmented reality system 100 are also applicable to other types of vehicles, including consumer vehicles, commercial trucks, airplanes, helicopters, spaceships, boats, submarines, and so forth.

[0024] Processing unit 112 includes one or more central processing units (CPUs), digital signal processing units (DSPs), microprocessors, application-specific integrated circuits (ASICs), neural processing units (NPUs), graphics processing units (GPUs), field- programmable gate arrays (FPGAs), and so forth. In some embodiments, processing unit 112 includes a programmable processor that executes program instructions to manipulate input data and generate outputs. In some examples, processing unit 112 includes any number of processing cores, memories, and other modules for facilitating program execution. Processing unit 112 receives inputs, such as sensor data from sensors 124 in communication with the augmented reality system, and generates pixels for display on an output device (e.g., display panel 104). In some embodiments, processing unit 112 is configured to execute HUD controller 122 such thatP240414WO (HRMN0513PC) processing unit 112 detects whether display panel 104 is malfunctioning. In some embodiments, processing unit 112 controls the operation of display panel 104 and / or light source 102.

[0025] Memory 114 includes a memory module or collection of memory modules. In some embodiments, memory 114 includes storage devices such as random-access memory (RAM) chips that store application programs and data for use by processing unit 112. In some embodiments, memory 114 includes non-volatile memory, such as optical drives, magnetic drives, flash drives, or other storage. In some embodiments, separate data stores, connected via a network (“cloud storage”) connect to HUD controller 122. HUD controller 122 within memory 114 is executed by processing unit 112 in order to implement the overall functionality of computing device 110 and, thus, coordinate the operation of augmented reality system 100 as a whole.

[0026] In some embodiments, light source 102 is a component in a display assembly and acts as a backlight unit (BLU) for display panel 104. For example, light source 102 can be a lightemitting diode (LED) array that provides backlight for display panel 104. Alternatively, in some embodiments, light source 102 is a laser source that provides one or more light rays to display panel 104. In various embodiments, light source 102 provides backlight as a baseline light level and / or a baseline color (e.g., a white light). In such instances, display panel 104 combines image pixels with backlight to generate the light pattern 152 that reflects off of windshield 106 to produce content viewed by the vehicle occupant using augmented reality system 100.

[0027] HUD controller 122 controls the operation of light source 102 by transmitting commands to light source driver 134. In various embodiments, light source driver 134 is a power circuit that controls power delivered to light source 102. In some embodiments, light source driver 134 controls power provided to light source 102 by power source 130. Alternatively, light source driver 134 includes and / or is connected to a separate power source (not shown).

[0028] HUD controller 122 transmits image data and / or video data to display panel 104, and display panel 104 displays the images provided by HUD controller 122, creating an augmented reality user interface. In some embodiments, display panel 104 is a thin-film transistor (TFT) liquid crystal display (LCD) panel. In some embodiments, display panel 104 includes a set of regions that selectively prevent portions of backlight from passing through. In some examples, display panel 104 draws current provided by the power source to selectively stimulate materials in various regions of display panel 104 based on the image received from HUD controller 122, causing stimulated regions of display panel 104 to become translucent and / or transparent.P240414WO (HRMN0513PC)Translucent and / or transparent regions of display panel 104 allow portions of the backlight to pass through display panel 104 and generate light pattern 152. HUD controller 122 controls display panel 104 to display an image by providing specific image data to display panel 104.

[0029] Power source 130 provides power to display panel 104. In some embodiments, power source 130 is a vehicle battery that provides electrical power to various components of the vehicle. In some embodiments, power source 130 is a dedicated power source. For example, power source 130 can be a dedicated battery that provides power to display panel 104. In some embodiments, power source 130 is connected to the display panel 104 via an electronic control unit (not shown) that controls the power provided to the display panel 104. In such instances, the electronic control unit can respond to a command provided by HUD controller 122 by connecting or disconnecting display panel 104 from power source 130.

[0030] Computing device 110 is in communication with sensor(s) 124. Data from the sensor(s) 124 is associated with vehicle speed, direction, acceleration, location, nearby objects such as lane lines, other vehicles, pedestrians and so forth. A mapping algorithm executed by computing device 110 or another device determines navigational instructions using mapping data, GPS sensor data, traffic information, and the like. Based on sensor data and navigational instructions, HUD controller 122 generates content for an augmented reality user interface for the vehicle occupant that displays one or more of the navigational instructions. The augmented reality user interface is often projected onto the windshield of a vehicle. In some embodiments, HUD controller 122 generates fixed content that includes vehicle information such as speed, certain navigation instructions, safety alerts, vehicle status data, and so forth. In some embodiments, portions of the content projected onto windshield 106 include augmented reality elements that are dynamically aligned to real -world objects. When navigation instructions are projected onto the windshield 106, as navigation progresses, the position and / or shape of dynamically aligned augmented reality elements changes to correspond to changes in vehicle position, changes in real -world objects, and updated navigation instructions. Examples of augmented reality elements aligned to real -world objects are described in more detail below. Augmented reality system 100 presents fixed and dynamically aligned information reflected off of windshield 106 to the vehicle occupant, allowing the vehicle occupant to view driving information and augmented reality elements while simultaneously maintaining attention on real- world objects visible through windshield 106 such as roads, vehicles, lane lines, street signs, pedestrians, and other objects on or near the road.P240414WO (HRMN0513PC)

[0031] Because there is latency between detection of an object in the real world by sensor(s) 124, and when the retrieval of the data associated with the object allows HUD controller 122 to display resulting information to a vehicle occupant in the augmented reality user interface, the alignment between dynamically aligned augmented reality elements and real -world objects is sometimes imperfect. In some situations, a vehicle occupant can observe jitter in the placement of the dynamically aligned augmented reality elements relative to real -world objects. Accordingly, HUD controller 122 mitigates the effects of this latency by generating animated patterns and / or animated textures within dynamically aligned augmented reality elements. In some embodiments, the dynamically aligned augmented reality elements are animated without changing the position of the augmented reality elements. In this scenario, the augmented reality elements are repositioned as the relative position of real -world objects to the vehicle changes, necessitating a change to the position of the augmented reality element in the augmented reality user interface. However, even though an augmented reality element is perceivably repositioned, the HUD controller 122 animates the components of the augmented reality element.

[0032] Animated augmented reality elements can include any graphical objects that are adaptively or dynamically positioned relative to one or more one real -world objects and which include animation of and / or within the graphical object. In some embodiments, animated augmented reality elements include an outline that is filled using one or more shapes which move relative to the outline of the augmented reality elements. In some embodiments, the outline of an animated augmented reality element is filled with a semitransparent texture which moves within the outline. In some embodiments, the outline of an animated augmented reality element is filled with a solid that pulsates in opacity or blinks within the outline. In some embodiments, animated augmented reality elements include chevrons or other shapes which appear to rotate or spin about directional axes of the shapes. The animated reality elements are overlaid onto real -world objects to which the animated reality elements correspond, from the perspective of a vehicle operator or vehicle occupant. Examples of animation of dynamically aligned augmented reality elements are described in more detail below. The motion of the animated patterns and / or animated textures included in augmented reality elements draws attention away from the effects of latency, misalignment and / or jitter, thus reducing or eliminating the perceivable effects of these artifacts experienced by the vehicle occupant.

[0033] FIG. 2 is a simplified diagram of an augmented reality user interface 200, according to some embodiments of the present disclosure. Figure 2 shows an example augmented reality user interface 200 implemented in a vehicle by the augmented reality system 100 of FIG. 1.P240414WO (HRMN0513PC)Augmented reality user interface 200 includes without limitation, a speed indicator 212, and a navigation direction element 214. The navigation direction element 214 has a directional graphical element referred to as an animated navigation path 216 which extends to and visually connects to a drone 218. Drone 218 demonstrates a virtual representation of a navigation action, and the animated navigation path 216 is an augmented reality element comprising a path between a starting position and drone 218. Drone 218 can be considered a portion of the navigation direction element 214 and / or a separate animated element. In FIG. 2, navigation direction element 214 is shown pointed in a forward direction relative to the vehicle associated in augmented reality user interface 200. However, in some embodiments, navigation direction element 214 can include one or more curves along the length of the animated navigation path 216. As shown in FIG. 2, and without limitation, a vehicle in which augmented reality user interface 200 is implemented includes the windshield 106 through which the vehicle occupant views a real -world environment 202 as well as a content area 210 on which images are generated and displayed by augmented reality system 100. Real -world environment 202 includes, without limitation, real -world objects such as one or more vehicles 204, one or more lane lines, road lines, street signs, trees, or other objects or obstacles.

[0034] As noted above, HUD controller 122 causes light source 102 and display panel 104 to project a light pattern 152 onto windshield 106. The vehicle occupant sees the reflected light pattern 154 that corresponds to specific information in the content area 210 that is overlay ed onto the view of real-world environment 202 visible through windshield 106. In some embodiments, the light pattern 152 displays the content area 210 as an overlay displaying information over at least a portion of the windshield 106 and provides information associated with the vehicle and / or the real -world environment 202. For example, the content area 210 can include a specific interface configuration that includes a group of icons, indicators, message panels, and / or other types of overlays that occupy portions of a total display area.

[0035] Content area 210 includes various textual and / or graphical elements that provide information about the vehicle and / or other information relevant to the vehicle occupant. In some embodiments, content area 210 includes fixed elements. Although the content of the fixed elements may change, the fixed elements are positioned in fixed locations of content area 210 and are not aligned to features of real-world environment 202. Examples of fixed elements include a speed indicator 212, a speed limit indicator (not shown), a turn signal indicator (not shown), a fuel or charge indicator (not shown), and so forth.P240414WO (HRMN0513PC)

[0036] In some embodiments, content area 210 includes objects which are dynamically aligned with features of real-world environment 202 to provide driving and navigational assistance to a vehicle operator. For example, content area 210 can include objects such as navigation direction element 214, which can be dynamically aligned with lane lines in real- world environment 202 to depict graphical navigational instructions for a vehicle occupant. As a vehicle progresses in time and / or distance, HUD controller 122 adapts the position and / or the shape of navigation direction element 214 to reflect changes in the position of the vehicle relative to real-world elements, updates to navigational instructions, or changes in the viewpoint of the vehicle occupant. Other examples of objects which are dynamically aligned with real- world features can include turn indicator chevrons, lane departure warnings, highlighting of pedestrians and / or other vehicles, or collision warnings.

[0037] FIGS. 3 A-3C are simplified diagrams of an augmented reality user interface showing an example of a progression of graphical navigation instructions. FIGS. 3 A-3C show simplified diagrams of an example augmented reality user interface 300, which guides a vehicle operator through a series of graphical navigation instructions to execute a lane change and an exit from a highway. Augmented reality user interface 300 includes fixed elements that are positioned in fixed locations of the HUD content area such as speed indicator 212, speed limit indicator 302, navigation instruction text 304, navigation map 306, and other vehicle data. In some embodiments, the fixed elements in the augmented reality user interface 300 are not animated. In some embodiments, the fixed elements are unassociated with objects in the real world. Augmented reality user interface 300 also includes an example of an augmented reality navigation direction element 214, which includes animated navigation path 216 and drone 218. Animated navigation path 216 includes opaque rectangles or other polygons or polygonal shapes interlaced with transparent spaces between the rectangles that are animated. In one example, the animated navigation path 216 is animated with a conveyor effect such that the elements of the animated navigation path 216 appear to move from a source location to a destination location. In other words, the animation links the source location to the destination location. In one embodiment, the source location corresponds the vehicle, such as the front end of the vehicle, and the destination location corresponds to another vehicle, a lane, an exit, or a destination of the vehicle.

[0038] The conveyor effect used for the animated navigation path 216 moves towards drone 218, as is discussed in more detail with reference to FIGS. 4A-4D. Animated navigation path 216 is displayed in a perspective mode, with graphical elements represented to be closer to theP240414WO (HRMN0513PC) vehicle shown as larger in size than elements represented to be farther away from the vehicle. Drone 218 is also shown in a perspective mode and represents the point of navigation direction element 214 farthest away from the vehicle. Navigation direction element 214 is an augmented reality element or virtual element which indicates a path for the vehicle to follow to execute a navigation instruction. In FIG. 3A, the portion of navigation direction element 214 represented as closest to the vehicle is centered between a lane line 308 and a lane line 322, both of which are real -world objects. Drone 218 is centered between lane line 322 and a left edge of the road 326. HUD controller 122 renders animated navigation path 216 to bend and extend from the graphical representation closest to the vehicle up to the drone. Navigation direction element 214 indicates to the vehicle occupant a navigation instruction to follow the path of animated navigation path 216, moving the vehicle from the center lane of the road to the left lane of the road.

[0039] FIG. 3B shows an example of the progression of augmented reality user interface 300 as the position of the vehicle relative to real world environment 202 has changed from the position in FIG. 3A. HUD controller 122 generates a modified navigation direction element 214 such that the modified navigation direction element 214 corresponds to an update in the position of the vehicle relative to real -world environment 202 and relative to a navigational goal. In some embodiments, HUD controller 122 extends, retracts, bends, repositions, aligns, and / or creates other changes in the position and / or shape of navigation direction element 214. In FIG. 3B, the drone of navigation direction element 214 continues to be aligned to be centered between lane line 322 and left edge of the road 326. The portion of navigation direction element 214 represented as closest to the vehicle has changed relative to that of FIG. 3 A, representing a change in the position of the vehicle relative to real-world environment 202.

[0040] In FIG. 3C, an example of further progression of augmented reality user interface 300 is shown. Again, HUD controller 122 generates a modified navigation direction element 214 that corresponds to an update in the position of the vehicle relative to real-world environment 202 and an update in navigation instructions. Navigation direction element 214 is shown as aligned to the center of an exit ramp 328 and indicates a navigation instruction at the moment of FIG. 3C of forward motion.

[0041] In the examples shown in FIGs 3A-3C, various components of navigation direction element 214 are augmented reality elements or virtual elements which are aligned to one or more real-world objects such as lane line 322, lane line 324, road 326, and exit ramp 328. Navigation direction element 214 is continuously updated by HUD controller 122 based on dataP240414WO (HRMN0513PC) related to the real-world environment 202 from sensor(s) 124. To draw the attention of a vehicle occupant away from jitter and potential misalignment of navigation direction element 214 or other user interface elements relative to real-world environment 202, rather than providing a static or solid fill area within the bounds of navigation direction element 214, HUD controller 122 includes rectangles, polygons or other shapes as well as transparent sections that are continuously animated within an area defined by navigation direction element 214. The animation of the rectangles within the area of navigation direction element 214 is an additional animation to one or more animations related to navigation, such as extension, retraction, bending, repositioning, aligning, and other changes in position and / or shape of navigation direction element 214.

[0042] FIGS. 4A-4D are simplified diagrams of an example of a progression of animation in a navigation direction element 214 according to various embodiments. FIGS. 4A-4D show an embodiment represented by animation 400. In one embodiment, animation 400 includes two or more shapes or polygons, such as rectangles, and one or more complementary transparent shapes that move within the area of the augmented reality element to produce a conveyor effect such that the animation appears like a conveyor moving from a source location on the windshield to a destination location on the windshield. In some examples, the source location is the front end of the vehicle, and the destination location is another vehicle, a lane, an exit, or another real- world object. The destination location can be marked with an indicator, such as a chevron or drone.

[0043] Animation 400 includes navigation direction element 214, which as shown in FIGS. 3A-3C, is an augmented reality object that can be aligned to objects in real-world environment 202. Navigation direction element 214 includes animated navigation path 216, which includes rectangles 402 interlaced with transparent spaces between the various rectangles 402. Rectangles 402 and the transparent spaces segment the appearance of a longitudinal edge along the length of navigation direction element 214. Navigation direction element 214 also includes drone 218. Animation 400 is generated by HUD controller 122 so that rectangles 402 appear to be moving from the bottom of animated navigation path 216 toward drone 218. The motion of rectangles 402 draws attention away from alignment jitter of navigation direction element 214 relative to real-world environment 202.

[0044] FIG. 4A shows rectangles 402(1) through 402 (5) placed relative to drone 218, with rectangle 402(5) truncated at the location of animated navigation path 216 represented closest to the vehicle. Referring to FIG. 4B, the animation 400 has progressed relative to theP240414WO (HRMN0513PC) representation shown in FIG. 4A. FIG. 4B shows animation 400 at a subsequent moment in time after the animation 400. The position of rectangles 402 in the animation 400 of FIG. 4B are closer to drone 218 than the position of the rectangles 402 in FIG. 4 A to create a conveyor effect. Rectangle 402(5) is extended further into animated navigation path 216. Rectangle 402(1) has been truncated at the end of animated navigation path 216 closest to drone 218. The effect of the change in position, extension, and truncation of rectangles 402 is an animation which makes the rectangles 402 in animated navigation path 216 appear to move toward drone 218. FIG. 4C shows animation 400 at an additional time step after animation 400 in FIG. 4B. Rectangles 402 show an additional change in position toward drone 218, rectangle 402(5) shows a further extension, and rectangle 402(1) shows a further truncation, creating a continuing animation which makes rectangles 402 in animated navigation path 216 appear to move toward drone 218. FIG. 4D shows animation 400 at an additional time step after animation 400 in FIG. 4C. The positions of rectangles 402(2) to 402 (5) are closer yet to drone 218. Rectangle 402(1) is no longer showing, and a new rectangle 402(6) is displayed at the bottom of animated navigation path 216. FIGS. 4A-4D show steps in an animation in which rectangles 402 appear to continually be moving toward drone 218 away from the vehicle occupant and toward a destination or target location. The perceived motion of rectangles 402 draw the vehicle occupant’s attention away from jitter caused by misalignment of navigation direction element 214 to real -world environment 202, reducing discomfort in the vehicle operator.

[0045] The example shown in FIGS 4A-4D include an animated navigation path with a straight navigation direction element 214, however, animation 400 can also be generated by HUD controller 122 with curved navigation direction elements 214. The example shown in FIGS 4A-4D is one embodiment of an animation which can be used to reduce the perception of jitter of navigation direction element 214. In some embodiments, the perceived motion of rectangles 402 can be generated with a conveyor effect that moves away from drone 218. In some embodiments the patterns and elements in navigation direction element 214 can vary and utilize different designs of rectangles, animated navigation paths, animated chevrons and / or animated drones.

[0046] FIGS. 5A-5C are simplified diagrams of an alternate example of a progression of animation in a navigation direction element 214 which reduces the perception of jitter. FIGS. 5A-5C show an embodiment of the present invention represented by animation 500. Animation 500 includes navigation direction element 214, which as shown in FIGS. 3A-3C, is an augmented reality object that can be aligned to objects in real-world environment 202.P240414WO (HRMN0513PC)Navigation direction element 214 includes animated navigation path 216, which is composed of a set of rectangles 502 interlaced with transparent spaces between the various rectangles 502. HUD controller 122 generates a textured fill within rectangles 502 in which a portion of the fill is opaque, and the opacity of the fill surrounding the opaque portion decreases in accordance with a gradient. Navigation direction element 214 also includes drone 218. Animation 500 is generated by HUD controller 122 such that the textured opacity filling rectangles 502 appears to be moving from the bottom of rectangles 502 to the top of rectangles 502 to produce a conveyor effect. The motion of textured opacity within rectangles 502 draws attention away from alignment jitter of navigation direction element 214 relative to real -world environment 202, reducing vehicle operator discomfort.

[0047] FIG. 5A shows rectangles 502(1) through 502 (4) placed relative to drone 218, with rectangle 502(4) at the location of animated navigation path 216 represented to be closest to the vehicle. In FIG. 5A, the opaque portions of the fill of rectangles 502 is near the bottom of rectangles 502. FIG. 5B shows animation 500 at a subsequent time after FIG. 5 A. The positions of the opaque portions of the fill of rectangles 502 in FIG. 5B are closer to drone 218 than the positions of the opaque fill of rectangles 502 in FIG. 5A. The effect of the change in positions of the opaque portions of the fill of rectangles 502 is an animation which makes the texture of the rectangles 502 in animated navigation path 216 appear to move toward drone 218. FIG. 5C shows animation 500 at an additional time after animation 500 in FIG. 5B. The opaque portions of rectangles 502 show an additional change in position toward drone 218, creating a continuing animation which makes the texture of the rectangles 502 in animated navigation path 216 appear to move toward drone 218. FIGS. 5A-5C show steps in an animation in which the textures of the rectangles 502 appear to continually be moving toward drone 218, away from the vehicle occupant and toward a destination or target location. The example shown in FIGS. 5A-5C includes an animated navigation path with a straight navigation direction element 214, however, an animation such as animation 500 can also be applied to curved navigation direction elements 214 such as the navigation direction elements 214 shown in FIGS. 3A-3C. The example shown in FIGS 5A-5C is just one embodiment of animation which can be used to reduce the perception of jitter of navigation direction element 214. In some embodiments, the perceived motion of the texture of rectangles 502 may be in a direction away from drone 218 and toward the vehicle occupant. In some embodiments the patterns and elements in navigation direction element 214 can include varying opacity, rectangles, animated navigation paths, animated chevrons, and / or animated drones.P240414WO (HRMN0513PC)

[0048] In some embodiments, augmented reality elements are hidden until a navigation action is identified. In some examples, animated augmented reality elements are removed from the augmented reality user interface in response to identifying that a navigation action is completed.

[0049] FIG. 6 is a flowchart illustrating a method for displaying an animated augmented reality object aligned with a real -world object. Although the method steps are described in conjunction with the embodiments of FIGs. 1-5, persons skilled in the art will understand that any system configured to perform the method steps, in any order, falls within the scope of the present disclosure.

[0050] A method 600 begins at step 610, where HUD controller 122 generates an augmented reality user interface by transmitting image data and / or video data to display panel 104. Display panel displays the images provided by HUD controller 122. HUD controller 122 controls display panel 104 to display an image by providing specific image data to display panel 104.

[0051] At step 620, HUD controller 122 determines the position of one or more real world objects based on data from sensor(s) 124. Data from the sensor(s) 124 is associated with vehicle speed, direction, acceleration, location, nearby objects such as lane lines, other vehicles, pedestrians and so forth.

[0052] At step 630, HUD controller 122 generates one or more animated augmented reality elements which are positioned relative to one or more real -world objects identified in step 620. In some embodiments, the animated augmented reality elements are generated in response to navigational instructions determined by a mapping algorithm utilizing GPS sensor data, traffic information, and the like. The animated augmented reality elements can include any graphical objects that are adaptively or dynamically positioned and which include animation graphical objects associated with the animated augmented reality elements.

[0053] At step 640, the one or more animated augmented reality elements generated in step 630 are projected onto a windshield of a vehicle. Display panel 104 combines image pixels with backlight to generate a light pattern 152 that reflects off of windshield 106 to produce content viewed by the vehicle occupant using augmented reality system 100. As a vehicle moves or the location of real -world objects changes, the position and / or shape of dynamically aligned augmented reality elements changes to correspond to changes in vehicle position, changes in real -world objects, and updated navigation instructions. The motion of animated patterns and / or animated textures included in the augmented reality elements draws user attention away fromP240414WO (HRMN0513PC) the effects of latency, misalignment and / or jitter, thereby reducing or eliminating the perceivable effects of these artifacts experienced by the user.

[0054] In sum, techniques are disclosed for animating display content in a heads-up display to reduce or eliminate the perceivable effects of latency, misalignment, and jitter that occur as the position of augmented reality display elements are updated to be aligned with real world objects. The techniques include use of a computing unit which generates dynamic graphical display elements such as navigation instructions, and a projection system which displays the graphical display elements. The graphical display elements are reflected off a semitransparent object such as the windshield of a vehicle and presented to a vehicle occupant as overlaid onto real world objects that are visible through the semitransparent object. The computing unit generates patterns and / or textures within the graphical display elements that are animated. For example, a graphical display element indicating the path of a lane change comprised of a navigational drone with an animated navigation path includes rectangles that are animated to move along the path of the animated navigation path. The motion of the animated patterns and / or animated textures included in graphical display elements draws attention away from effects of latency, misalignment and jitter, thus reducing the perceivable effects of these artifacts that are experienced by the vehicle occupant.

[0055] At least one technical advantage of the disclosed techniques relative to the prior art is that a vehicle occupant can view augmented reality content such as graphical display elements overlaid onto real world objects with a reduction of the perceivable effects of latency, misalignment and jitter of the graphical display elements relative to the real -world objects. This results in reduced discomfort of the vehicle occupant and an improved experience in the use of a heads-up display system.

[0056] 1. In some embodiments, a computer-implemented method comprises generating an augmented reality user interface, determining a position of a real -world object relative to a vehicle, generating an augmented reality element positioned in the augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated, and projecting, using an augmented reality system, the augmented reality user interface onto a windshield of the vehicle.

[0057] 2. The computer-implemented method of clause 1, wherein the position of the real- world object is based on sensor data detected using one or more sensors in communication with the augmented reality system.P240414WO (HRMN0513PC)

[0058] 3. The computer-implemented method of clauses 1 or 2, wherein the augmented reality system comprises a heads-up display (HUD) system included in a vehicle.

[0059] 4. The computer-implemented method of any of clauses 1-3, wherein the augmented reality element is generated in response to identifying a navigation instruction, and the augmented reality element graphically depicts the navigation instruction.

[0060] 5. The computer-implemented method of any of clauses 1-4, wherein generating the augmented reality element further comprises generating an animation linking a source location to a destination location on the windshield of the vehicle.

[0061] 6. The computer-implemented method of any of clauses 1-5, wherein the source location corresponds to the vehicle and the destination location corresponds to the position of the real -world object as viewed through the windshield.

[0062] 7. The computer-implemented method of any of clauses 1-6, wherein the animation comprises one or more polygons that are animated to produce a conveyor effect towards a destination location on the windshield of the vehicle.

[0063] 8. The computer-implemented method of any of clauses 1-7, wherein the one or more polygons are animated to rotate.

[0064] 9. The computer-implemented method of any of clauses 1-8, wherein the one or more polygons are animate to pulsate.

[0065] 10. The computer-implemented method of any of clauses 1-9, wherein generating the augmented reality element further comprises animating an indicator associated with a destination location.

[0066] 11. The computer-implemented method of any of clauses 1-10, wherein the indicator comprises an animated chevron or animated drone.

[0067] 12. The computer-implemented method of any of clauses 1-11, wherein the generating the augmented reality element further comprises animating a rotating object that is overlaid onto the real -world object on the windshield from a perspective of a vehicle operator or occupant.

[0068] 13. The computer-implemented method of any of clauses 1-12, further comprising updating, using the augmented reality system, the position of the real -world object based onP240414WO (HRMN0513PC) sensor data from one or more sensors associated with the vehicle, and updating, using the augmented reality system, the position of the augmented reality element based on the position of the real -world object.

[0069] 14. In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of generating an augmented reality user interface, determining a position of a real -world object relative to a vehicle, generating an augmented reality element positioned in the augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated, and projecting, using an augmented reality system, the augmented reality user interface onto a windshield of the vehicle.

[0070] 15. The one or more non-transitory computer-readable media of clause 14, wherein the augmented reality user interface further comprises at least one fixed element and the at least one fixed element is not animated.

[0071] 16. The one or more non-transitory computer-readable media of clauses 14 or 15, wherein the at least one fixed element is unassociated with the position of the real -world object.

[0072] 17. The one or more non-transitory computer-readable media of any of clauses 14-16, wherein the augmented reality element is animated by generating an animated texture that includes two or more shapes and one or more complementary transparent shapes that move within the area of the augmented reality element.

[0073] 18. The one or more non-transitory computer-readable media of any of clauses 14-17, wherein the augmented reality system comprises a heads-up display (HUD) system included in a vehicle.

[0074] 19. The one or more non-transitory computer-readable media of any of clauses 14-18, wherein the augmented reality element is generated in response to identifying a navigation instruction, and the augmented reality element graphically depicts the navigation instruction.

[0075] 20. In some embodiments, an augmented reality system comprises a heads-up display (HUD) system associated with a vehicle, one or more sensors associated with the vehicle, a memory storing instructions, and one or more processors, that when executing the instructions, are configured to perform the steps of generating an augmented reality user interface, determining a position of a real -world object relative to the vehicle based on sensor data from the one or more sensors, generating an augmented reality element positioned in theP240414WO (HRMN0513PC) augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated, and projecting, using the HUD system, the augmented reality user interface onto a windshield of the vehicle.

[0076] Any and all combinations of any of the claim elements recited in any of the claims and / or any elements described in this application, in any fashion, fall within the contemplated scope of the present disclosure and protection.

[0077] The descriptions of the various embodiments have been presented for purposes of illustration but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0078] Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module” or “system.” Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0079] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non- exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable readonly memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0080] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer programP240414WO (HRMN0513PC) products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / acts specified in the flowchart and / or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0081] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0082] Terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e^, rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the”P240414WO (HRMN0513PC) are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0083] Elements described in detail with reference to one embodiment or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, embodiments, or application may be incorporated into other embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.

[0084] While the preceding is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

1. P240414WO (HRMN0513PC)WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: generating an augmented reality user interface; determining a position of a real -world object relative to a vehicle; generating an augmented reality element positioned in the augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated; and projecting, using an augmented reality system, the augmented reality user interface onto a windshield of the vehicle.

2. The computer-implemented method of claim 1, wherein the position of the real -world object is based on sensor data detected using one or more sensors in communication with the augmented reality system.

3. The computer-implemented method of claim 1, wherein the augmented reality system comprises a heads-up display (HUD) system included in a vehicle.

4. The computer-implemented method of claim 1, wherein the augmented reality element is generated in response to identifying a navigation instruction, and the augmented reality element graphically depicts the navigation instruction.

5. The computer-implemented method of claim 1, wherein generating the augmented reality element further comprises generating an animation linking a source location to a destination location on the windshield of the vehicle.

6. The computer-implemented method of claim 5, wherein the source location corresponds to the vehicle and the destination location corresponds to the position of the real- world object as viewed through the windshield.

7. The computer-implemented method of claim 1, wherein the animation comprises one or more polygons that are animated to produce a conveyor effect towards a destination location on the windshield of the vehicle.P240414WO (HRMN0513PC)8. The computer-implemented method of claim 7, wherein the one or more polygons are animated to rotate.

9. The computer-implemented method of claim 7, wherein the one or more polygons are animate to pulsate.

10. The computer-implemented method of claim 1, wherein generating the augmented reality element further comprises animating an indicator associated with a destination location.

11. The computer-implemented method of claim 8, wherein the indicator comprises an animated chevron or animated drone.

12. The computer-implemented method of claim 1, wherein the generating the augmented reality element further comprises animating a rotating object that is overlaid onto the real- world object on the windshield from a perspective of a vehicle operator or occupant.

13. The computer-implemented method of claim 1, further comprising: updating, using the augmented reality system, the position of the real -world object based on sensor data from one or more sensors associated with the vehicle; and updating, using the augmented reality system, the position of the augmented reality element based on the position of the real -world object.

14. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: generating an augmented reality user interface; determining a position of a real -world object relative to a vehicle; generating an augmented reality element positioned in the augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated; and projecting, using an augmented reality system, the augmented reality user interface onto a windshield of the vehicle.P240414WO (HRMN0513PC)15. The one or more non-transitory computer-readable media of claim 14, wherein the augmented reality user interface further comprises at least one fixed element and the at least one fixed element is not animated.

16. The one or more non-transitory computer-readable media of claim 13, wherein the at least one fixed element is unassociated with the position of the real -world object.

17. The one or more non-transitory computer-readable media of claim 14, wherein the augmented reality element is animated by generating an animated texture that includes two or more shapes and one or more complementary transparent shapes that move within the area of the augmented reality element.

18. The one or more non-transitory computer-readable media of claim 14, wherein the augmented reality system comprises a heads-up display (HUD) system included in a vehicle.

19. The one or more non-transitory computer-readable media of claim 14, wherein the augmented reality element is generated in response to identifying a navigation instruction, and the augmented reality element graphically depicts the navigation instruction.

20. An augmented reality system comprising: a heads-up display (HUD) system associated with a vehicle; one or more sensors associated with the vehicle; a memory storing instructions; and one or more processors, that when executing the instructions, are configured to perform the steps of: generating an augmented reality user interface; determining a position of a real -world object relative to the vehicle based on sensor data from the one or more sensors; generating an augmented reality element positioned in the augmented reality user interface based on the position of the real -world object, wherein a portion of the augmented reality element is animated; and projecting, using the HUD system, the augmented reality user interface onto a windshield of the vehicle.

Images