Low latency hand tracking in augmented reality systems

Abstract

A method for reducing motion-to-photon delay for hand tracking is described. In one aspect, a method includes acquiring a first frame from a camera of an augmented reality (AR) device, tracking a first image of a hand in the first frame, rendering virtual content based on the tracking of the first image of the hand in the first frame, acquiring a second frame from the camera prior to completion of rendering of the virtual content, the second frame immediately after the first frame, and rendering the virtual content based on the second frame. In one embodiment, a method includes receiving virtual content from a hand, tracking a second image of the hand in a second frame using a computer vision engine of an AR device, generating an annotation based on tracking the second image of the hand in the second frame, forming annotated virtual content based on the annotation and the virtual content, and displaying the annotated virtual content in a display of the AR device.

Description

Technical Field

[0001] The subject matter disclosed herein relates generally to display systems. In particular, the present disclosure relates to systems and methods for reducing annotation delays in hand tracking for augmented reality (AR) devices. Background Art

[0002] Augmented reality (AR) systems present virtual content to enhance a user's real-world environment. For example, virtual content superimposed on physical objects can be used to create the illusion that the physical objects are moving, animating, etc. The augmented reality device worn by the user continuously updates the presentation of the virtual content based on the user's movements to create the illusion that the virtual content physically exists in the user's real-world environment. For example, when the user moves their head, the augmented reality device updates the presentation of the virtual content to create the illusion that the virtual content remains in the same geographic location within the user's real-world environment. Thus, the user can move around the virtual objects presented by the augmented reality device in the same way that the user moves around physical objects.

[0003] In order to convincingly create the illusion that virtual objects are in the user's real-world environment, augmented reality devices must update the rendering of virtual objects almost instantaneously as the device moves. However, virtual content may take longer to update because the AR display device must process environmental data, render virtual content, and then project the virtual content. This delay can also be referred to as "motion-to-photon delay." Any perceptible motion-to-photon delay will degrade the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0005] Figure 1 is a block diagram illustrating a network environment for operating an AR display device according to an example embodiment.

[0006] Figure 2 is a block diagram illustrating an AR display device according to one example embodiment.

[0007] Figure 3 is a block diagram illustrating a tracking system according to one example implementation.

[0008] Figure 4 is a block diagram illustrating an example process for low latency annotation according to one example implementation.

[0009] Figure 5 An example of annotation according to one example implementation is shown.

[0010] Figure 6is a flow chart illustrating a method for reducing latency in an AR display device according to an example embodiment.

[0011] Figure 7 is a flow chart illustrating a method for reducing latency in an AR display device according to an example embodiment.

[0012] Figure 8 is a flowchart illustrating a method for reducing delay in an AR display device according to another example embodiment.

[0013] Fig. 9 is a block diagram illustrating example operational stages according to the prior art.

[0014] Fig.10 is a block diagram illustrating example operational stages according to one example implementation.

[0015] Fig.11 is a block diagram illustrating example operational stages according to another example implementation.

[0016] Fig.12 is a block diagram illustrating a software architecture within which the present disclosure may be implemented according to an example implementation.

[0017] Fig.13 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein according to an example embodiment. DETAILED DESCRIPTION

[0018] The following description describes the system, method, technology, instruction sequence and computer program product of the example implementation of this theme. In the following description, for the purpose of illustration, many specific details are set forth to provide an understanding of the various implementations of this theme. However, it will be apparent to those skilled in the art that the implementation of this theme can be put into practice without some or other details in these specific details. Examples only represent possible variations. Unless otherwise clearly stated, structures (e.g., structural components, such as modules) are optional and can be combined or subdivided, and operations (e.g., in processes, algorithms or other functions) can be sequentially varied or combined or subdivided.

[0019] Augmented reality (AR) applications allow users to experience information, such as in the form of virtual objects rendered in a display of an AR display device (also referred to as a display device). The rendering of virtual objects can be based on the position of the display device relative to physical objects or relative to a reference frame (external to the display device) so that the virtual objects are correctly presented in the display. Virtual objects are presented to be aligned with physical objects perceived by the user of the AR display device. Graphics (e.g., graphical elements containing instructions and guidance) are presented to be attached to the physical object of interest. To do this, the AR display device detects the physical object and tracks the posture of the AR display device relative to the position of the physical object. The posture identifies the position and orientation of the object relative to a reference frame or relative to another object.

[0020] One problem with implementing AR is the delay associated with presenting virtual content. When a user moves an AR display device, the user's view of the real-world environment changes instantly. Because the AR display device must process environmental data with inertial measurement unit (IMU) data, render virtual content, and project the virtual content in front of the user's field of view, it takes longer for the virtual content to change. This delay is defined herein and referred to as "motion-to-photon delay" (e.g., the duration between a user moving an AR display device (or a tracked object, such as a hand) and the presentation of its virtual content adapting to that particular motion). Motion-to-photon delay causes jitter or lag in the presentation of virtual content and reduces the user's augmented reality experience.

[0021] This application describes a method for significantly reducing the latency of annotations for visual-only trackers (e.g., hand tracking) in augmented reality use cases. Annotations are simple visual representations. For example, in the case of hand tracking, Figure 5 As shown in , the hand is often visualized using simple connected dots.

[0022] Existing solutions do not address the problem of motion-to-photon latency for hand tracking. Instead, rendering the visualization of the tracked hand is treated as a conventional object tracking and visualization problem. Such conventional tracking and visualization results in a very long pipeline with many stages, resulting in a long total duration. For example, conventional hand tracking solutions in AR devices have motion-to-photon latency in the range of about 100ms.

[0023] The present application describes systems and methods for reducing motion-to-photon latency for annotations (e.g., finger tracking annotations) for hand tracking in AR. For example, the present application utilizes certain simple rendering operations that are short and predictable in their duration and can therefore be scheduled in advance. In one example embodiment, the system uses the tracking results of the next camera frame to draw "simple annotations" for the current frame before rendering (based on the current frame) is completed. Therefore, these annotations have a latency of one frame less. The term "simple annotation" refers to annotations with minimal components (e.g., lines and points). Examples of non-simple annotations include user interface elements or hand grids.

[0024] In one example implementation, the present application describes a method for reducing motion-to-photon latency for hand tracking in an AR device. In one aspect, the method includes acquiring a first frame from a camera of an augmented reality (AR) device, tracking a first image of a hand in the first frame using a computer vision engine of the AR device, rendering virtual content based on tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device, acquiring a second frame from the camera before completing rendering of the virtual content, the second frame immediately following the first frame, tracking a second image of the hand in the second frame using the computer vision engine of the AR device, generating annotations based on tracking of the second image of the hand in the second frame, forming annotated virtual content based on the annotations and the virtual content, and displaying the annotated virtual content in a display of the AR device.

[0025] Thus, one or more methods described herein help solve the technical problem of motion-to-photon latency in AR devices. The currently described methods provide improvements to the functional operation of a computer by reducing latency in hand tracking annotations in an AR device. Thus, one or more methods described herein may avoid the need for certain work or computing resources. Examples of such computing resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, network bandwidth, and cooling capacity.

[0026] Figure 1 1 is a network diagram illustrating a network environment 100 suitable for operating an AR display device 108 according to some example embodiments. The network environment 100 includes an AR display device 108 and a server 110, which are communicatively coupled to each other via a network 104. The AR display device 108 and the server 110 may be implemented in whole or in part in a computer system, as described below with respect to Fig.13 The server 110 may be part of a network-based system. For example, the network-based system may be or include a cloud-based server system that provides additional information such as virtual content (e.g., a three-dimensional model of a virtual object) to the AR display device 108.

[0027] The user 106 operates the AR display device 108. The user 106 may be a human user (e.g., a human), a machine user (e.g., a computer configured by a software program to interact with the AR display device 108), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user 106 is not part of the network environment 100, but is associated with the AR display device 108.

[0028] The AR display device 108 may be a computing device with a display, such as a smartphone, tablet, or wearable computing device (e.g., a watch or glasses). The computing device may be handheld, or may be removably mounted to the head of the user 106. In one example, the display may be a screen that displays content captured with a camera of the AR display device 108. In another example, the display of the device may be transparent, such as in the lenses of wearable computing glasses. In another example implementation, the display may be opaque and may be worn by the user to cover the user's field of view.

[0029] The user 106 operates an application of the AR display device 108. The application may include an AR application that is configured to provide the user 106 with an experience triggered by tracking a physical object (e.g., hand 112). For example, the user 106 may point to the camera of the AR display device 108 to capture an image of his hand 112. The hand 112 is tracked and recognized locally in the AR display device 108 using the local background recognition dataset module of the AR application of the AR display device 108. The local background recognition dataset module may include a library of virtual objects associated with real-world physical objects or references. In one example, the AR application generates additional information corresponding to the image (e.g., a three-dimensional model), and in response to recognizing the hand 112, the additional information is presented in the display of the AR display device 108. If the captured image is not recognized locally at the AR display device 108, the AR display device 108 downloads the additional information (e.g., a three-dimensional model) corresponding to the captured image from the database of the server 110 via the network 104.

[0030] The AR display device 108 includes a tracking system (not shown). The tracking system tracks the pose (e.g., position and orientation) of the AR display device 108 relative to the real-world environment 102 using optical sensors (e.g., depth-enabled 3D cameras, image cameras), inertial sensors (e.g., gyroscopes, accelerometers), wireless sensors (Bluetooth, Wi-Fi), GPS sensors, and audio sensors to determine the position of the AR display device 108 within the real-world environment 102. In another example implementation, the tracking system tracks the pose of the hand 112 in a video frame captured by a camera of the AR display device 108.

[0031] In one example implementation, the server 110 may be configured to detect and recognize the hand 112 (or track a physical object) based on sensor data (e.g., image and depth data) from the AR display device 108, determine the pose of the AR display device 108 and the pose of the hand 112 based on the sensor data. The server 110 may also generate a virtual object based on the AR display device 108 and the pose of the hand 112. The server 110 transmits the virtual object to the AR display device 108. Object recognition, tracking, and AR rendering may be performed on the AR display device 108, the server 110, or a combination between the AR display device 108 and the server 110.

[0032] Figure 1 Any of the machines, databases, or devices shown in the description may be implemented in a general-purpose computer that is modified (e.g., configured or programmed) by software to be a special-purpose computer to perform one or more of the functions described herein for the machine, database, or device. Figures 6 to 8 A computer system capable of implementing any one or more of the methods described herein is discussed. As used herein, a "database" is a data storage resource and may store data structured as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database), a triple store, a hierarchical data store, or any suitable combination thereof. In addition, Figure 1 Any two or more of the machines, databases, or devices shown in the may be combined into a single machine, and the functionality described herein for any single machine, database, or device may be divided among multiple machines, databases, or devices.

[0033] The network 104 may be any network capable of communicating between machines (e.g., server 110), databases, and devices (e.g., AR display device 108). Thus, the network 104 may be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network 104 may include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0034] Figure 2 2 is a block diagram illustrating modules (e.g., components) of an AR display device 108 according to some example implementations. The AR display device 108 includes a sensor 202, a display 204, a processor 208, a rendering system 224, and a storage device 206. Examples of the AR display device 108 include a head-mounted device, a wearable computing device, a desktop computer, a vehicle-mounted computer, a tablet computer, a navigation device, a portable media device, or a smart phone.

[0035] Sensors 202 include, for example, optical sensors 214 (e.g., cameras such as color cameras, thermal cameras, depth sensors, and one or more grayscale global shutter tracking cameras) and inertial sensors 216 (e.g., gyroscopes, accelerometers). Other examples of sensors 202 include proximity or position sensors (e.g., near field communication, GPS, Bluetooth, Wi-Fi), audio sensors (e.g., microphones), or any suitable combination thereof. Note that sensors 202 are described herein for illustrative purposes, and thus sensors 202 are not limited to the sensors described above.

[0036] The display 204 includes a screen or monitor configured to display images generated by the processor 208. In one example implementation, the display 204 may be transparent or semi-transparent so that the user 106 can see through the display 204 (in an AR use case). In another example, the display 204 (e.g., an LCOS display) presents each frame of the virtual content in multiple renderings.

[0037] The processor 208 includes an AR application 210 and a tracking system 212. The AR application 210 uses computer vision to detect and track the hand 112. The AR application 210 retrieves a virtual object (e.g., a 3D object model) based on the image of the tracked hand 112. The AR application 210 renders the virtual object in the display 204. In the AR scene, the AR application 210 generates annotations / virtual content that are superimposed (e.g., covered on or otherwise displayed simultaneously with) on the image of the hand 112 captured by the optical sensor 214. The annotations / virtual content can be manipulated by changing the posture of the hand 112 relative to the optical sensor 214 (e.g., its physical position, orientation, or both). Similarly, the visualization of the annotations / virtual content can be manipulated by adjusting the posture of the AR display device 108 relative to the hand 112. For the VR scene, the AR application 210 displays the annotations / virtual content in the display 204 at a position (in the display 204) determined based on the posture of the hand 112.

[0038] In another example implementation, the AR application 210 includes a background local image recognition module (not shown) configured to determine whether the captured image matches an image in a local database (e.g., storage device 206) of images and corresponding additional information (e.g., virtual models and interactive features) stored locally on the AR display device 108. In one example, the background local image recognition module retrieves the primary content dataset from the server 110 and generates and updates the background content dataset based on the image captured using the AR display device 108.

[0039] The tracking system 212 estimates the pose of the AR display device 108 and / or the pose of the hand 112. In one example, the tracking system 212 uses image data and corresponding inertial data from the optical sensor 214 and the inertial sensor 216 to track the position and pose of the AR display device 108 relative to a reference frame (e.g., the real-world environment 102). In one example, the tracking system 212 uses the sensor data to determine a three-dimensional pose of the AR display device 108. The three-dimensional pose is the determined orientation and position of the AR display device 108 relative to the user's real-world environment 102. For example, the AR display device 108 can use an image of the user's real-world environment 102 and other sensor data to identify the relative position and orientation of the AR display device 108 and physical objects in the real-world environment 102 around the AR display device 108. The tracking system 212 continuously collects and uses updated sensor data describing the movement of the AR display device 108 to determine an updated three-dimensional pose of the AR display device 108, which indicates a change in the relative position and orientation of the AR display device 108 relative to the physical objects in the real-world environment 102. The tracking system 212 provides the three-dimensional pose of the AR display device 108 to the rendering system 224 .

[0040] In another example, the tracking system 212 receives an image of the hand 112 from the optical sensor 214. The tracking system 212 then uses computer vision to track the pose of the hand 112 in the image. The pose of the hand 112 can be identified relative to the AR display device 108.

[0041] The rendering system 224 includes a graphics processing unit 218 and a display controller 220. The graphics processing unit 218 includes a rendering engine (not shown) configured to render a frame of a 3D model of a virtual object based on the virtual content provided by the AR application 210 and the pose of the AR display device 108 (or the pose of the hand 112). In other words, the graphics processing unit 218 uses the three-dimensional pose of the AR display device 108 to generate a frame of virtual content to be presented on the display 204. For example, the graphics processing unit 218 uses the three-dimensional pose to render the frame of the virtual content so that the virtual content is presented in the display 204 with an appropriate orientation and position to appropriately enhance the reality of the user. As an example, the graphics processing unit 218 can use the three-dimensional pose data to render the frame of the virtual content so that when presented on the display 204, the virtual content is superimposed on the hand 112 in the user's real world environment 102. The graphics processing unit 218 generates updated frames of virtual content based on the updated three-dimensional pose of the AR display device 108 , which reflects the changes in the position and orientation of the user's hand 112 relative to the user's real-world environment 102 .

[0042] The graphics processing unit 218 transmits the rendered frame to the display controller 220. The display controller 220 is positioned as an intermediary between the graphics processing unit 218 and the display 204, receiving image data (e.g., annotated rendered frames) from the graphics processing unit 218 and providing the annotated rendered frames to the display 204.

[0043] In one example implementation, both the optical sensor 214 and the graphics processing unit 218 operate at the same rate / frequency (e.g., 60 Hz). The timing controller 226 controls the timing of the optical sensor 214 and the rendering of the graphics processing unit 218 so that the optical sensor 214 provides a new frame and the tracking system 212 generates a new track based on the new frame before the display controller 220 sends the original rendered frame to the display 204. Figure 4 The operation of the timing controller 226 is described in more detail.

[0044] The storage device 206 stores virtual object content 222. The virtual object content 222 includes, for example, a database of visual references (eg, images, QR codes) and corresponding virtual content (eg, three-dimensional models of virtual objects).

[0045] Any one or more modules described herein can be implemented using hardware (e.g., a processor of a machine) or a combination of hardware and software. For example, any module described herein can configure a processor to perform the operations described herein for the module. In addition, any two or more of these modules can be combined into a single module, and the functions described herein for a single module can be subdivided between multiple modules. In addition, according to various example embodiments, the modules described herein as implemented in a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0046] Figure 3 A tracking system 212 is shown according to an example implementation. The tracking system 212 includes a visual tracking system 308 and a content tracking system 310, for example.

[0047] The visual tracking system 308 includes an inertial sensor module 302, an optical sensor module 304, and a pose estimation module 306. The inertial sensor module 302 acquires inertial sensor data from the inertial sensor 216. The optical sensor module 304 acquires optical sensor data from the optical sensor 214.

[0048] The pose estimation module 306 determines the pose (e.g., location, position, orientation) of the AR display device 108 relative to a reference frame (e.g., the real-world environment 102). In one example implementation, the pose estimation module 306 estimates the pose of the AR display device 108 based on images captured by the optical sensor 214 (via the optical sensor module 304) and a 3D map of feature points from inertial sensor data captured by the inertial sensor 216 (via the inertial sensor module 302).

[0049] In one example, the pose estimation module 306 includes an algorithm that combines inertial information from the inertial sensor 216 and image information from the optical sensor 214 coupled to a rigid platform (e.g., the AR display device 108) or a vehicle (rig). The vehicle may include multiple cameras (with non-overlapping (distributed aperture) or overlapping (stereo or more) fields of view) mounted on the rigid platform and an IMU (e.g., the vehicle may therefore have at least one IMU and at least one camera). In another example embodiment, the motion-to-photon latency optimization described herein may operate with a simpler tracking module (e.g., a tracking module that only tracks rotational data from an IMU), and therefore does not require image data from the optical sensor 214.

[0050] The content tracking system 310 includes a hand tracking system 312. The hand tracking system 312 uses computer vision to detect and track the position of a hand (within an image). In one example, the hand tracking system 312 detects and tracks the hand 112 of the user 106 or a hand from another person in an image captured using the optical sensor 214. The hand tracking system 312 tracks the pose of the image of the hand 112 in the image frame captured from the optical sensor module 304.

[0051] Figure 4 2 is a block diagram illustrating an example process according to one example implementation. Tracking system 212 receives sensor data (e.g., image A) from sensor 202 to determine a pose of AR display device 108 and / or a pose of hand 112. Tracking system 212 provides the tracked pose of hand 112 to graphics processing unit 218.

[0052] The graphics processing unit 218 includes a rendering engine 402 and an annotation engine 404. The rendering engine 402 renders a frame (e.g., frame A) of virtual content (provided by the AR application 210) at a location (in the display 204) based on the pose of the hand 112 (and / or the pose of the AR display device 108) determined by the tracking system 212. The rendering engine 402 provides the rendered frame (e.g., frame A) to the annotation engine 404. The tracking system 212 identifies the latest pose of the hand 112 to the annotation engine 404 based on the latest image frame (e.g., image B) from the optical sensor 214. The tracking system 212 sends the latest tracking pose of the hand 112 to the annotation engine 404. The annotation engine 404 generates or draws a "simple annotation" (e.g., hand tracking lines and points) on frame A from the rendering engine 402.

[0053] The graphics processing unit 218 provides the annotated rendered frame (eg, annotated frame A) to the display controller 220. The display controller 220 transmits the annotated frame A to the display 204 for display.

[0054] In one example implementation, the timing controller 226 controls the timing of the optical sensor 214 and the timing of the rendering system 224. Both the optical sensor 214 and the rendering system 224 may operate at the same frequency / rate (e.g., 60 Hz). The timing controller 226 selects the timing of the optical sensor 214 and the timing of the rendering system 224 so that tracking of the second image (e.g., image B) is sufficiently completed before the display controller 220 sends the annotated frame A to the display 204.

[0055] In another example implementation, the tracking system 212 completes tracking of the second image (e.g., image B) "immediately" before the rendering engine 402 completes the rendered virtual content (e.g., frame A), so that the annotation engine 404 can still draw annotations on the rendered virtual content before it is processed by the display controller 220. The term "immediately" can refer to a time period of minimum duration that enables the annotation engine 404 to draw annotations before sending the rendered frame to the display controller 220.

[0056] In one example, the optical sensor 214 and the rendering system 224 each have their own subsystems with their own timing. Therefore, the timing controller 226 controls and aligns the timing of the optical sensor 214 and the rendering system 224.

[0057] Figure 5 An example of a "simple" annotation is shown according to an example implementation. Annotation 502 is drawn on an image of a hand tracking model 504 .

[0058] Figure 6is a flowchart illustrating a method for reducing latency in an AR display device according to an example embodiment. The operations in routine 600 may be performed by the AR display device 108 using the above-described Figure 2 and Figure 4 Therefore, the routine 600 is described by way of example with reference to the rendering system 224. However, it should be understood that at least some operations of the routine 600 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0059] In block 602, the tracking system 212 acquires a first image of the hand 112 and tracks a first pose of the hand 112 based on the first image. In block 604, the rendering engine 402 renders virtual content based on the first pose of the hand 112. In block 606, the tracking system 212 acquires a second image of the hand 112 and tracks a second pose of the hand 112. In block 608, the annotation engine 404 generates an annotation about the virtual content based on the second pose of the hand 112. In block 610, the display 204 displays the annotation and the rendered virtual content.

[0060] It should be noted that other embodiments can use different sorting, additional or less operations and different naming or terminology to complete similar functions. In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein are selected to illustrate some operating principles in a simplified form.

[0061] Figure 7 is a flowchart illustrating a method for reducing latency in an AR display device according to an example embodiment. The operations in routine 700 may be performed by the AR display device 108 using the above-described Figure 2 and Figure 4 Therefore, routine 700 is described by way of example with reference to timing controller 226. However, it should be understood that at least some operations of routine 700 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0062] In block 702, the timing controller 226 sets or verifies that the camera rate and the rendering system rate are set to the same rate (e.g., 60 Hz). In block 704, the timing controller 226 controls / adjusts the timing of the optical sensor 214 relative to the timing of the rendering system 224. In block 706, the timing controller 226 aligns the timing of the optical sensor 214 with the timing of the rendering system 224 so that tracking of the second image is completed immediately before the rendered frame is sent to the display 204. For example, if the optical sensor 214 is running at 60 Hz (16.6 ms), then after a 10 ms exposure, there will be a 6.6 ms period of time that the camera waits. The timing controller 226 aligns / times the operation of the optical sensor 214 with the operation of the rendering system 224 so that tracking (e.g., Fig.10 The tracking stage 1112) is just in the rendering (e.g., Fig.10 The tracking is completed before the rendering phase 1008 of the image processing unit (e.g., within 1 ms for 60 Hz). In one example, the tracking is completed just when the rendering is completed. In another example, the tracking is completed no earlier than the rendering is completed (e.g., greater than 1 ms for 60 Hz).

[0063] It should be noted that other embodiments can use different ordering, additional or less operation and different nomenclature or terminology to complete similar functions.In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner.The operation described herein is selected to illustrate some operating principles in a simplified form.

[0064] Figure 8 800 is a flowchart illustrating a method for reducing latency in an AR display device according to another example embodiment. The operations in routine 800 may be performed by the AR display device 108 using the above-described Figure 2 and Figure 4 Therefore, the routine 800 is described by way of example with reference to the rendering system 224. However, it should be understood that at least some operations of the routine 800 can be deployed on various other hardware configurations or performed by similar components residing elsewhere.

[0065] In block 802, the tracking system 212 acquires an image of the hand 112 and tracks a gesture of the hand 112 based on the image. In block 804, the rendering engine 402 acquires rendered virtual content based on a previous image of the hand 112. In block 806, the annotation engine 404 generates annotations on the virtual content based on the gesture of the hand 112. In block 808, the display 204 displays the annotations and the rendered virtual content.

[0066] It should be noted that other embodiments can use different sorting, additional or less operations and different naming or terminology to implement similar functions. In some embodiments, various operations can be performed in parallel with other operations in a synchronous or asynchronous manner. The operations described herein are selected to illustrate some operating principles in a simplified form.

[0067] Fig. 9 9 is a block diagram illustrating example operation stages according to the prior art. The motion-to-photon delay 914 is approximately 44 ms and includes the following stages: a capture stage 902, a delivery stage 904, a tracking stage 906, a rendering stage 908, a display send stage 910, and a display stage 912. The capture stage 902 represents how long it takes for the optical sensor 214 to capture an image (e.g., approximately 8 ms). The delivery stage 904 represents the time when the frame read out from the image is delivered to the computer vision algorithm (e.g., approximately 3 ms). The tracking stage 906 represents how long it takes for the computer vision algorithm to process the image and track the image of the hand 112 (e.g., approximately 9 ms). The rendering stage 908 represents how long it takes for the rendering engine 402 to render virtual content based on the tracked image of the hand 112 (e.g., approximately 16 ms). The display send stage 910 represents how long it takes to deliver the rendered frame at the display controller 220 (e.g., approximately 8 ms). The display stage 912 represents how long it takes for the display 204 to display the rendered frame (e.g., approximately 2 ms).

[0068] Fig.10 is a block diagram illustrating example operational stages according to one example implementation. The motion-to-photon delay 1022 is approximately 28 ms, rather than Fig. 9 The phases include: capture phase 1002, delivery phase 1004, tracking phase 1006, rendering phase 1008, display send phase 1010, presentation phase 1012, capture phase 1014, delivery phase 1016, and tracking phase 1018. Capture phase 1002 is similar to Fig. 9 The capture phase 902 of FIG. The delivery phase 1004 is similar to Fig. 9 The delivery phase 904. The tracking phase 1006 is similar to Fig. 9 The tracking stage 906. The rendering stage 1008 is similar to Fig. 9 Rendering stage 908.

[0069] The tracking system 212 captures the next camera frame in a capture phase 1014 and delivers the next camera frame to the hand tracking system 312 in a delivery phase 1016. The hand tracking system 312 uses computer vision to identify the latest pose of the hand 112 based on the next camera frame in a tracking phase 1018. The annotation engine 404 generates simple annotations in an annotation phase 1020. The annotations from the annotation phase 1020 are combined with the rendered content from the rendering phase 1008 to generate an annotated frame. The annotated frame is sent to the display 204 in a display sending phase 1010. The display 204 renders the annotated frame in a rendering phase 1012.

[0070] Fig.11 is a block diagram illustrating example operation stages according to another example implementation. The motion-to-photon delay 1022 is approximately 20 ms, rather than Fig.10 The phases include: capture phase 1102, delivery phase 1104, tracking phase 1106, capture phase 1108, delivery phase 1110, tracking phase 1112, rendering phase 1114, annotation phase 1116, display sending phase 1118, display phase 1120, and rendering phase 1122. Capture phase 1102 is similar to Fig. 9 The capture phase 902 of FIG. The delivery phase 1104 is similar to Fig. 9 The delivery phase 904. The tracking phase 1106 is similar to Fig. 9 The tracking stage 906. The rendering stage 1114 is similar to Fig. 9 Rendering stage 908.

[0071] The tracking system 212 captures the next camera frame in the capture phase 1108 and delivers the next camera frame to the hand tracking system 312 in the delivery phase 1110. The hand tracking system 312 uses computer vision to identify the latest pose of the hand 112 based on the next camera frame in the tracking phase 1112. The annotation engine 404 generates simple annotations in the annotation phase 1116. The annotations from the annotation phase 1116 are combined with the rendered content from the rendering phase 1114 to generate an annotated frame (before the content is rendered based on the image from the capture phase 1108). The annotated frame is sent to the display 204 in the display sending phase 1118. The display 204 presents the annotated frame in the display phase 1120.

[0072] Fig.121200 is a block diagram illustrating a software architecture 1204 that may be installed on any one or more of the devices described herein. The software architecture 1204 is supported by hardware such as a machine 1202, which includes a processor 1220, a memory 1226, and an I / O component 1238. In this example, the software architecture 1204 may be conceptualized as a stack of layers in which each layer provides specific functionality. The software architecture 1204 includes layers such as an operating system 1212, a library 1210, a framework 1208, and an application 1206. In operation, the application 1206 invokes an API call 1250 through the software stack and receives a message 1252 in response to the API call 1250.

[0073] The operating system 1212 manages hardware resources and provides public services. The operating system 1212 includes, for example, a kernel 1214, a service 1216, and a driver 1222. The kernel 1214 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 1214 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. The service 1216 can provide other public services for other software layers. The driver 1222 is responsible for controlling the underlying hardware or interfacing with the underlying hardware. For example, the driver 1222 may include a display driver, a camera driver, or Low-power drivers, Flash drivers, Serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0074] The library 1210 provides a low-level common infrastructure used by the application 1206. The library 1210 may include a system library 1218 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1210 may include an API library 1224, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a library for a database (e.g., SQLite that provides various relational database functions), a web library (e.g., WebKit that provides web browsing functions), etc. The library 1210 may also include various other libraries 1228 to provide many other APIs to the application 1206 .

[0075] The framework 1208 provides a high-level common infrastructure used by the applications 1206. For example, the framework 1208 provides various graphical user interface (GUI) functions, advanced resource management, and advanced positioning services. The framework 1208 can provide a wide range of other APIs that can be used by the applications 1206, some of which can be specific to a specific operating system or platform.

[0076] In an example implementation, applications 1206 may include home applications 1236, contact applications 1230, browser applications 1232, book reader applications 1234, location applications 1242, media applications 1244, messaging applications 1246, game applications 1248, and a broad category of other applications such as third-party applications 1240. Applications 1206 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 1206 structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly languages). In a specific example, third-party applications 1240 (e.g., applications created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Applications developed using a software development kit (SDK) can be mobile operating systems such as IOS TM ANDROID TM , Mobile software running on a mobile phone or Linux OS or other mobile operating system. In this example, third-party applications 1240 can activate API calls 1250 provided by the operating system 1212 to facilitate the functions described herein.

[0077] Fig.131300 within which instructions 1308 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 1300 to perform any one or more of the methods discussed herein. For example, the instructions 1308 may cause the machine 1300 to perform any one or more of the methods described herein. The instructions 1308 convert a general purpose, unprogrammed machine 1300 into a specific machine 1300 that is programmed to perform the functions described and illustrated in the manner described. The machine 1300 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1300 may operate as a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1300 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1308 that specify actions to be taken by the machine 1300, either sequentially or otherwise. In addition, while only a single machine 1300 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1308 to perform any one or more of the methodologies discussed herein.

[0078] The machine 1300 may include a processor 1302, a memory 1304, and an I / O component 1342, which may be configured to communicate with each other via a bus 1344. In an example embodiment, the processor 1302 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), other processors, or any suitable combination thereof) may include, for example, a processor 1306 that executes instructions 1308 and a processor 1310. The term "processor" is intended to include a multi-core processor, which may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Fig.13 Multiple processors 1302 are shown, but the machine 1300 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0079] The memory 1304 includes a main memory 1312, a static memory 1314, and a storage unit 1316, which are all accessible by the processor 1302 via the bus 1344. The main memory 1304, the static memory 1314, and the storage unit 1316 store instructions 1308 that embody any one or more of the methodologies or functions described herein. The instructions 1308 may also reside, in whole or in part, within the main memory 1312, within the static memory 1314, within the machine-readable medium 1018 within the storage unit 1316, within at least one of the processors 1302 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 1300.

[0080] I / O components 1342 may include a variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O components 1342 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O components 1342 may include Fig.13 1342 may include an output component 1328 and an input component 1330. In various example embodiments, the I / O component 1342 may include an output component 1328 and an input component 1330. The output component 1328 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistive mechanism), other signal generators, etc. The input component 1330 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the location and / or force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0081] In other example embodiments, the I / O component 1342 may include a biometric component 1332, a motion component 1334, an environmental component 1336, or a positioning component 1338, as well as various other components. For example, the biometric component 1332 includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1334 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental components 1336 include, for example, lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), auditory sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors that detect hazardous gas concentrations for safety or measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning components 1338 include positioning sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc.

[0082] A variety of technologies may be used to achieve communication. I / O components 1342 also include communication components 1340, which are operable to couple machine 1300 to network 1320 or device 1322 via coupling 1324 and coupling 1326, respectively. For example, communication components 1340 may include a network interface component or another suitable device that interfaces with network 1320. In other examples, communication components 1340 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Parts (e.g. Low energy consumption), Device 1322 may be any peripheral device (eg, a peripheral device coupled via USB) from another machine or from a variety of peripheral devices.

[0083] In addition, the communication component 1340 can detect the identifier or include a component operable to detect the identifier. For example, the communication component 1340 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data symbol (Dataglyph), a maximum code (MaxiCode), PDF417, UltraCode, UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1340, such as a location obtained via Internet Protocol (IP) geolocation, ... Location derived from signal triangulation, location derived via detection of NFC beacon signals that can indicate a specific location, and so on.

[0084] Various memories (e.g., memory 1304, main memory 1312, static memory 1314, and / or memory of processor 1302) and / or storage unit 1316 may store one or more sets of instructions and data structures (e.g., software) that implement or are used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 1308) when executed by processor 1302 cause various operations to implement the disclosed embodiments.

[0085] Instructions 1308 may be sent or received via a network interface device (e.g., a network interface component included in communication component 1340) using a transmission medium and using any of a number of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over network 1320. Similarly, instructions 1308 may be sent or received to device 1322 via coupling 1326 (e.g., a peer-to-peer coupling) using a transmission medium.

[0086] Although the embodiments have been described with reference to specific example embodiments, it is obvious that various modifications and changes may be made to these embodiments without departing from the broader scope of the present disclosure. Therefore, the description and the drawings should be considered in an illustrative sense, rather than a restrictive sense. The drawings constituting a part of the present disclosure show, by way of illustration and not limitation, specific embodiments in which the present subject matter can be practiced. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, so that structural and logical replacements and changes may be made without departing from the scope of the present disclosure. Therefore, the specific embodiments should not be understood as restrictive, and the scope of the various embodiments is limited only by the full scope of the equivalent schemes assigned to the appended claims and these claims.

[0087] An abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. The abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing detailed description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. The method of the disclosure should not be interpreted as reflecting the intention that the claimed embodiment requires more features than those explicitly recited in each claim. On the contrary, as reflected in the appended claims, the subject matter of the present invention lies in less than all the features of a single disclosed embodiment. Therefore, the appended claims are incorporated into the detailed description herein, and each claim exists independently as a separate embodiment.

[0088] Example

[0089] Example 1 is a method comprising: acquiring a first frame from a camera of an augmented reality (AR) device; tracking a first image of a hand in the first frame using a computer vision engine of the AR device; rendering virtual content based on tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device; acquiring a second frame from the camera before completing rendering of the virtual content, the second frame immediately following the first frame; tracking a second image of the hand in the second frame using the computer vision engine of the AR device; generating annotations based on tracking of the second image of the hand in the second frame; forming annotated virtual content based on the annotations and the virtual content; and displaying the annotated virtual content in a display of the AR device.

[0090] Example 2 includes the method of Example 1, wherein the camera device and the rendering engine operate at the same frequency.

[0091] Example 3 includes the method of Example 2, wherein tracking of the second image of the hand in the second frame is completed immediately before the rendering engine completes rendering of the virtual content.

[0092] Example 4 includes the method of Example 3, further comprising: aligning the timing of the camera device with the timing of the rendering engine using a timing controller based on the computer vision engine completing tracking of the second image of the hand in the second frame immediately before the rendering engine completes rendering of the virtual content.

[0093] Example 5 includes the method of Example 1, wherein acquiring the second frame is during rendering of the virtual content.

[0094] Example 6 includes the method of Example 1, wherein tracking of the second image of the opponent in the second frame is completed immediately before completing rendering of the virtual content.

[0095] Example 7 includes the method of Example 1, wherein forming the annotated virtual content comprises drawing a finger tracking annotation on the virtual content.

[0096] Example 8 is a computing device comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the device to: acquire a first frame from a camera of an augmented reality (AR) device; track a first image of a hand in the first frame using a computer vision engine of the AR device; render virtual content based on tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device; acquire a second frame from the camera before completing the rendering of the virtual content, the second frame immediately following the first frame; track a second image of the hand in the second frame using the computer vision engine of the AR device; generate annotations based on tracking of the second image of the hand in the second frame; form annotated virtual content based on the annotations and the virtual content; and display the annotated virtual content in a display of the AR device.

[0097] Example 9 includes the computing device of Example 8, wherein the camera device and the rendering engine operate at the same frequency.

[0098] Example 10 includes the computing device of Example 9, wherein the tracking of the second image of the opponent in the second frame is completed immediately before the rendering engine completes rendering of the virtual content.

[0099] Example 11 includes the computing device of Example 10, wherein the instructions further configure the device to: align the timing of the camera device with the timing of the rendering engine using a timing controller based on the computer vision engine completing tracking of the second image of the hand in the second frame immediately before the rendering engine completes rendering of the virtual content.

[0100] Example 12 includes the computing device of Example 8, wherein acquiring the second frame is during rendering of the virtual content.

[0101] Example 13 includes the computing device of Example 8, wherein the tracking of the second image of the opponent in the second frame is completed immediately before the rendering of the virtual content is completed.

[0102] Example 14 includes the computing device of Example 8, wherein forming the annotated virtual content comprises drawing a finger tracking annotation on the virtual content.

[0103] Example 15 is a non-transitory computer-readable storage medium, which includes instructions that, when executed by a computer, cause the computer to: acquire a first frame from a camera of an augmented reality (AR) device; track a first image of a hand in the first frame using a computer vision engine of the AR device; render virtual content based on tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device; acquire a second frame from the camera before completing rendering of the virtual content, the second frame immediately following the first frame; track a second image of the hand in the second frame using the computer vision engine of the AR device; generate annotations based on tracking of the second image of the hand in the second frame; form annotated virtual content based on the annotations and the virtual content; and display the annotated virtual content in a display of the AR device.

[0104] Example 16 includes the computer-readable storage medium of Example 15, wherein the camera device and the rendering engine operate at the same frequency.

[0105] Example 17 includes the computer-readable storage medium of Example 16, wherein the tracking of the second image of the opponent in the second frame is completed immediately before the rendering engine completes rendering of the virtual content.

[0106] Example 18 includes the computer-readable storage medium of Example 17, wherein the instructions further configure the computer to: based on the computer vision engine completing tracking of the second image of the hand in the second frame immediately before the rendering engine completes rendering of the virtual content, align the timing of the camera device with the timing of the rendering engine using a timing controller.

[0107] Example 19 includes the computer-readable storage medium of Example 15, wherein acquiring the second frame is during rendering of the virtual content.

[0108] Example 20 includes the computer-readable storage medium of Example 15, wherein the tracking of the second image of the opponent in the second frame is completed immediately before the rendering of the virtual content is completed.

Claims

1. A method comprising: Acquire a first frame from a camera of an augmented reality (AR) device; tracking a first image of a hand in the first frame using a computer vision engine of the AR device; rendering virtual content based on the tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device; Acquire a second frame from the camera device before completing rendering of the virtual content, the second frame being immediately after the first frame; tracking, using the computer vision engine of the AR device, a second image of the hand in the second frame; generating annotations based on tracking of a second image of the hand in the second frame; forming annotated virtual content based on the annotation and the virtual content; as well as The annotated virtual content is displayed on a display of the AR device.

2. The method according to claim 1, wherein: The camera device and the rendering engine operate at the same frequency.

3. The method according to claim 2, wherein: The tracking of the second image of the hand in the second frame is completed immediately before the rendering engine completes rendering of the virtual content.

4. The method according to claim 3, further comprising: Based on completing the computer vision engine's tracking of the second image of the hand in the second frame immediately before the rendering engine completes rendering of the virtual content, aligning the timing of the camera device with the timing of the rendering engine using a timing controller.

5. The method according to claim 1, wherein: Acquiring the second frame is during rendering of the virtual content.

6. The method according to claim 1, wherein: Tracking of the second image of the hand in the second frame is completed immediately before rendering of the virtual content is completed.

7. The method according to claim 1, wherein: The annotated virtual content is formed by: A finger tracking annotation is drawn on the virtual content.

8. A computing device, comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to: Acquire a first frame from a camera of an augmented reality (AR) device; tracking a first image of a hand in the first frame using a computer vision engine of the AR device; rendering virtual content based on the tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device; Acquire a second frame from the camera device before completing rendering of the virtual content, the second frame being immediately after the first frame; tracking, using the computer vision engine of the AR device, a second image of the hand in the second frame; generating annotations based on tracking of a second image of the hand in the second frame; forming annotated virtual content based on the annotation and the virtual content; as well as The annotated virtual content is displayed on a display of the AR device.

9. The computing device according to claim 8, wherein: The camera device and the rendering engine operate at the same frequency.

10. The computing device according to claim 9, wherein: The tracking of the second image of the hand in the second frame is completed immediately before the rendering engine completes rendering of the virtual content.

11. The computing device according to claim 10, wherein: The instructions further configure the apparatus to: Based on completing the computer vision engine's tracking of the second image of the hand in the second frame immediately before the rendering engine completes rendering of the virtual content, aligning the timing of the camera device with the timing of the rendering engine using a timing controller.

12. The computing device according to claim 8, wherein: Acquiring the second frame is during rendering of the virtual content.

13. The computing device according to claim 8, wherein: Tracking of the second image of the hand in the second frame is completed immediately before rendering of the virtual content is completed.

14. The computing device of claim 8, wherein: The annotated virtual content is formed by: A finger tracking annotation is drawn on the virtual content.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: Acquire a first frame from a camera of an augmented reality (AR) device; tracking a first image of a hand in the first frame using a computer vision engine of the AR device; rendering virtual content based on the tracking of the first image of the hand in the first frame using a rendering engine of a graphics processing unit (GPU) of the AR device; Acquire a second frame from the camera device before completing rendering of the virtual content, the second frame being immediately after the first frame; tracking, using the computer vision engine of the AR device, a second image of the hand in the second frame; generating annotations based on tracking of a second image of the hand in the second frame; forming annotated virtual content based on the annotation and the virtual content; as well as The annotated virtual content is displayed on a display of the AR device.

16. The computer-readable storage medium of claim 15, wherein: The camera device and the rendering engine operate at the same frequency.

17. The computer-readable storage medium of claim 16, wherein: The tracking of the second image of the hand in the second frame is completed immediately before the rendering engine completes rendering of the virtual content.

18. The computer-readable storage medium of claim 17, wherein: The instructions further configure the computer to: Based on completing the computer vision engine's tracking of the second image of the hand in the second frame immediately before the rendering engine completes rendering of the virtual content, aligning the timing of the camera device with the timing of the rendering engine using a timing controller.

19. The computer-readable storage medium of claim 15, wherein: Acquiring the second frame is during rendering of the virtual content.

20. The computer-readable storage medium of claim 15, wherein: Tracking of the second image of the hand in the second frame is completed immediately before rendering of the virtual content is completed.

Images