Post-deformation of dynamic anchor content

Through the pose estimation technology combined with image sensor and inertial measurement unit, the rendering of virtual content is updated in real time, solving the problem of inaccurate pose prediction of dynamic objects and improving the anchoring effect and user experience of virtual content in an extended real environment.

CN119068093APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410692950.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to accurately predict and update the posture of dynamic objects, resulting in insufficient registration and display of virtual content in extended real environments, affecting the anchoring effect of virtual content and real-world objects.

Method used

The image of the object in the physical environment is captured by the image sensor, the pose estimation module is used to determine the current and predicted poses of the object, combined with the inertial measurement unit data, the rendering and deformation of the virtual content is updated in real time to match the dynamic changes of the object.

Benefits of technology

It realizes more accurate virtual content registration and display in an extended real environment, improves the anchoring effect of virtual content and dynamic objects, and enhances the stability and authenticity of user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068093A_ABST
    Figure CN119068093A_ABST
Patent Text Reader

Abstract

The invention relates to late deformation of dynamic anchor content. In one embodiment, a method of displaying an image is performed by a device including an image sensor, a display, one or more processors, and a non-transitory memory. The method includes capturing an image of an object in a physical environment using an image sensor. The method includes obtaining a first predicted object pose of an object in the physical environment at a display time based on the image. The method includes rendering virtual content based on the first predicted object pose. The method includes obtaining a second predicted object pose of the object in the physical environment at the display time. The method includes deforming the virtual content based on the second predicted object pose. The method includes displaying the deformed virtual content on a display at a display time.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 470,662, filed on June 2, 2023, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems, methods, and devices for rendering virtual content anchored to real-world objects. Background Art

[0004] In various specific implementations, an extended reality (XR) environment presented by an electronic device including a display includes world-locked content (which moves on the display when the posture of the electronic device changes) and display-locked content (which maintains its position on the display when the posture of the electronic device changes.) In addition, some world-locked content is anchored to static objects, such as a virtual cylinder displayed on a real table, and some world-locked content is anchored to dynamic objects, such as a virtual mask displayed on a person's face. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] So that the present disclosure may be understood by those of ordinary skill in the art, a more particular description may be had with reference to aspects of some illustrative embodiments, some of which are shown in the accompanying drawings.

[0006] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0007] FIG. 2A to FIG. 2E An XR environment is shown during various time periods according to some implementations.

[0008] Figure 3 An electronic device according to some implementations is shown.

[0009] Figure 4 is a flowchart representation of a method of displaying an image according to some specific implementations.

[0010] Figure 5 is a flowchart representation of another method of displaying an image according to some specific implementations.

[0011] Figure 6 is a flowchart representation of a method of inferring an image according to some specific implementations.

[0012] Figure 7 is a flowchart representation of another method of inferring an image according to some specific implementations.

[0013] Figure 8 is a block diagram of an example controller according to some implementations.

[0014] Fig. 9 is a block diagram of an exemplary electronic device according to some implementations.

[0015] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0016] Various embodiments disclosed herein include devices, systems, and methods for displaying images. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of an object in a physical environment using an image sensor. The method includes obtaining a first predicted object pose of the object in the physical environment at a display time based on the image. The method includes rendering virtual content based on the first predicted object pose. The method includes obtaining a second predicted object pose of the object in the physical environment at a display time. The method includes deforming the virtual content based on the second predicted object pose. The method includes displaying the deformed virtual content on the display at the display time.

[0017] Various specific implementations disclosed herein include devices, systems, and methods for displaying images. In various specific implementations, the method is performed by a device including a display, one or more processors, and a non-volatile memory. The method includes obtaining a first predicted device posture of a device at a display time and a first predicted object posture of an object at a display time. The method includes rendering virtual content based on the first predicted device posture and the first predicted object posture. The method includes obtaining a second predicted device posture of the device at a display time and a second predicted object posture of the object at a display time. The method includes deforming the virtual content based on the second predicted device posture and the second predicted object posture. The method includes displaying the deformed virtual content on the display at a display time.

[0018] Various embodiments disclosed herein include devices, systems, and methods for inferring images. In various embodiments, the method is performed by a device including an image sensor, a display, one or more processors, and a non-volatile memory. The method includes capturing an image of an object in a physical environment using an image sensor. The method includes obtaining a first predicted object pose of the object at a first display time based on the image. The method includes rendering virtual content based on the first predicted object pose. The method includes displaying the virtual content at the first display time. The method includes obtaining a second predicted object pose of the object at a second display time after the first display time. The method includes deforming the virtual content based on the second predicted object pose. The method includes displaying the deformed virtual content at the second display time.

[0019] Various specific implementations disclosed herein include devices, systems, and methods for inferring images. In various specific implementations, the method is performed by a device including a display, one or more processors, and a non-volatile memory. The method includes obtaining a first predicted device posture of a device at a first display time and a first predicted object posture of an object at the first display time. The method includes rendering virtual content based on the first predicted device posture and the first predicted object posture. The method includes displaying the virtual content on the display at the first display time. The method includes obtaining a second predicted device posture of the device at a second display time and a second predicted object posture of the object at the second display time. The method includes deforming the virtual content based on the second predicted device posture and the second predicted object posture. The method includes displaying the deformed virtual content on the display at the second display time.

[0020] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein.

[0021] describe

[0022] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0023] As described above, in various implementations, some world-locked content is anchored to dynamic objects, such as a virtual mask displayed on a person's face. Determining where on the display to display the dynamically anchored world-locked content is based on the pose of the dynamic object. Accurately predicting this pose is important for registration of virtual content. After rendering begins, a more accurate pose may be available. Therefore, in various implementations, the rendered content is shifted (or deformed) based on the updated pose.

[0024] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120.

[0025] In some implementations, the controller 110 is configured to manage and coordinate the user's XR experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Figure 8 Controller 110 is described in more detail. In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. For example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). As another example, controller 110 is included in a housing of electronic device 120. In some implementations, the functionality of controller 110 is provided by electronic device 120 and / or is combined with the electronic device.

[0026] In some implementations, the electronic device 120 is configured to provide an XR experience to the user. In some implementations, the electronic device 120 includes a suitable combination of software, firmware, and / or hardware. According to some implementations, the electronic device 120 presents XR content to the user via a display 122 while the user is physically present within a physical environment 105, which includes a table 107 within a field of view 111 of the electronic device 120. In some implementations, the user holds the electronic device 120 in one or both of his / her hands. In some implementations, when providing XR content, the electronic device 120 is configured to display an XR object (e.g., an XR cylinder 109) and implement video pass-through of the physical environment 105 (e.g., including a representation 117 of the table 107) on the display 122. Reference is made below. Fig. 9 The electronic device 120 is described in more detail.

[0027] According to some implementations, the electronic device 120 provides an XR experience to the user while the user is virtually and / or physically present within the physical environment 105 .

[0028] In some implementations, the user wears the electronic device 120 on his / her head. For example, in some implementations, the electronic device includes a head-mounted system (HMS), a head-mounted device (HMD), or a head-mounted housing (HME). Therefore, the electronic device 120 includes one or more XR displays configured to display XR content. For example, in various implementations, the electronic device 120 surrounds the user's field of view. In some implementations, the electronic device 120 is a handheld device (such as a smart phone or tablet) configured to present XR content, and the user no longer wears the electronic device 120 but holds the device with the display facing the user's field of view and the camera facing the physical environment 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, the electronic device 120 is replaced with an XR cabin, housing, or room configured to present XR content, in which the user no longer wears or holds the electronic device 120.

[0029] FIG. 2A to FIG. 2EAn XR environment 200 is shown, which is at the perspective of a user of an electronic device and is displayed at least in part by a display of the electronic device. In various embodiments, the user's perspective is from the position of an image sensor of the electronic device. For example, in various embodiments, the electronic device is a handheld electronic device, and the user's perspective is from the position of the image sensor of the handheld electronic device facing the physical environment. In various embodiments, the user's perspective is from the position of the user of the electronic device. For example, in various embodiments, the electronic device is a head-mounted electronic device, and the user's perspective is from the position of the user facing the physical environment, which, in the absence of a head-mounted electronic device, is generally approximated to the user's field of view. In various embodiments, the user's perspective is from the position of the user's avatar. For example, in various embodiments, the XR environment 200 is a virtual environment, and the user's perspective is from the position of an avatar or other representation of the user facing the virtual environment.

[0030] In one specific implementation, the electronic device is a head-mounted device with a transparent display. Thus, the XR environment includes a real environment (viewed by the user through the transparent display) with virtual objects (displayed by the transparent display) superimposed on the real environment. In addition, because it is head-mounted, the posture of the electronic device (e.g., its position and / or orientation) changes as the user changes the posture of the user's head.

[0031] FIG. 2A to FIG. 2E The XR environment 200 is shown during a series of time periods. In various implementations, each time period is an instant, a fraction of a second, a few seconds, a few hours, a few days, or any length of time.

[0032] The XR environment 200 includes multiple objects, including one or more real objects (e.g., a user's hand 211, a table 212, a person other than the user 213, and a smart watch 214 worn by the user) and one or more virtual objects (e.g., a virtual ring 221, a virtual flower 222, virtual glasses 223, and a virtual clock 224). In various specific implementations, certain objects (such as the virtual ring 221, the virtual flower 222, and the virtual glasses 223) are displayed at locations in the XR environment 200, such as at locations defined by three coordinates in a three-dimensional (3D) XR coordinate system. Therefore, when the electronic device moves (e.g., changes position and / or orientation) in the XR environment 200, the object moves on the display of the electronic device but maintains its (possibly time-dependent) position in the XR environment 200. Such virtual objects that move on the display but maintain their positions in the XR environment 200 in response to the movement of the electronic device are referred to as world-locked objects. In various implementations, certain virtual objects, such as the virtual clock 224, are displayed at a position on the display such that the object is stationary on the display on the electronic device as the electronic device moves in the XR environment 200. Such virtual objects that maintain their position on the display in response to the movement of the electronic device are referred to as head-locked objects or display-locked objects.

[0033] Some world-locked objects are anchored to stationary objects, and may be referred to as statically-anchored world-locked objects. For example, virtual flower 222 is displayed as being located on table 212. However, some world-locked objects are anchored to dynamic (or moving) objects, and may be referred to as dynamically-anchored world-locked objects. For example, virtual ring 221 is displayed as being worn on user's hand 211. Thus, as the user's hand 211 moves in XR environment 200, virtual ring 221 moves on the display. As another example, virtual glasses 223 are displayed as being worn on the face of person 213. Thus, as person 213 moves in the XR environment, virtual glasses 223 move on the display.

[0034] Figure 2AThe XR environment 200 is shown during a first time period. During the first time period, the electronic device displays a virtual ring 221 at a first ring position on the display, which corresponds to a first ring position in the XR environment 200 for a first posture of the electronic device and a first posture of the hand 211, such as a position on the hand 211 when the hand 211 has the first posture. During the first time period, the electronic device displays a virtual flower 222 at a first flower position on the display, which corresponds to a flower position in the XR environment 200 for the first posture of the electronic device, such as a position on the table 212. During the first time period, the electronic device displays virtual glasses 223 at a first glasses position on the display, which corresponds to a first glasses position in the XR environment 200 for the first posture of the electronic device and a first posture of the person 213, such as a position on the person 213 when the person 213 has the first posture. During the first time period, the electronic device displays a virtual clock 224 and a fixed clock position on the display.

[0035] Figure 2B The XR environment 200 is shown during a second time period after the first time period. Compared to the first time period, during the second time period, the posture of the electronic device has changed from the first posture to the second posture. In particular, the electronic device has moved to the left. Compared to the first time period, during the second time period, the posture of the hand 211 and the posture of the person 213 have not changed.

[0036] During the second time period, the electronic device displays a virtual ring 221 at a second ring position on the display, which corresponds to the first hand position in the XR environment 200 for the second posture of the electronic device and the first posture of the hand 211. During the second time period, the electronic device displays a virtual flower 222 at a second flower position on the display, which corresponds to the flower position in the XR environment 200 for the second posture of the electronic device. During the second time period, the electronic device displays virtual glasses 223 at a second glasses position on the display, which corresponds to the first person position in the XR environment 200 for the second posture of the electronic device and the first posture of the person 213. During the second time period, the electronic device displays a virtual clock 224 at a fixed clock position on the display.

[0037] Thus, during the second time period, as compared to the first time period, due to the electronic device moving to the left in the XR environment 200, the world-locked objects (e.g., the virtual ring 221, the virtual flower 222, and the virtual glasses 223) have moved to the right on the display.

[0038] Figure 2CXR environment 200 is shown during a third time period after the second time period. During the third time period, the posture of the electronic device and the posture of hand 211 have not changed compared to the second time period. During the third time period, the posture of person 213 has changed from the first posture to the second posture compared to the second time period. In particular, person 213 has moved closer to the electronic device.

[0039] During the third time period, the electronic device displays a virtual ring 221 at a second ring position on the display, which corresponds to the first hand position in the XR environment 200 for the second posture of the electronic device and the first posture of the hand 211. During the third time period, the electronic device displays a virtual flower 222 at a second flower position on the display, which corresponds to the flower position in the XR environment 200 for the second posture of the electronic device. During the third time period, the electronic device displays virtual glasses 223 at a third glasses position on the display, which corresponds to the second person position in the XR environment 200 for the second posture of the electronic device and the second posture of the person 213. During the third time period, the electronic device displays a virtual clock 224 at a fixed clock position on the display.

[0040] Thus, during the third time period, as person 213 moves closer to the electronic device in XR environment 200 compared to the second time period, virtual glasses 223 have zoomed in and moved downward on the display.

[0041] Figure 2D XR environment 200 is shown during a fourth time period after the third time period. During the fourth time period, the posture of the electronic device and the posture of person 213 have not changed compared to the third time period. During the fourth time period, the posture of hand 211 has changed from the first posture to the second posture compared to the third time period. In particular, hand 211 has moved to the left in XR environment 200.

[0042] During a fourth time period, the electronic device displays a virtual ring 221 at a third ring position on the display, the third ring position corresponding to the second hand position in the XR environment 200 for the second posture of the electronic device and the second posture of the hand 211. During the fourth time period, the electronic device displays a virtual flower 222 at a second flower position on the display, the second flower position corresponding to the flower position in the XR environment 200 for the second posture of the electronic device. During the fourth time period, the electronic device displays virtual glasses 223 at a third glasses position on the display, the third glasses position corresponding to the second person position in the XR environment 200 for the second posture of the electronic device and the second posture of the person 213. During the fourth time period, the electronic device displays a virtual clock 224 at a fixed clock position on the display.

[0043] Thus, during the fourth time period, as compared to the third time period, the virtual ring 221 has moved to the left on the display due to the hand 211 moving to the left in the XR environment 200 .

[0044] Figure 2E The XR environment 200 is shown during a fifth time period after the fourth time period. During the fifth time period, the posture of the person 213 has not changed compared to the fourth time period. During the fifth time period, the posture of the hand 211 has changed from the second posture to the third posture compared to the fourth time period. In particular, the hand 211 has moved to the right in the XR environment 200. In addition, during the fifth time period, the posture of the electronic device has changed from the second posture to the third posture compared to the fourth time period. In particular, the electronic device has moved to the right in the XR environment 200.

[0045] During the fifth time period, the electronic device displays a virtual ring 221 at a fourth ring position on the display, which corresponds to a third hand position in the XR environment 200 for the third posture of the electronic device and the third posture of the hand 211. During the fifth time period, the electronic device displays a virtual flower 222 at a third flower position on the display, which corresponds to a flower position in the XR environment 200 for the third posture of the electronic device. During the fifth time period, the electronic device displays virtual glasses 223 at a fourth glasses position on the display, which corresponds to a second person position in the XR environment 200 for the third posture of the electronic device and the second posture of the person 213. During the fifth time period, the electronic device displays a virtual clock 224 at a fixed clock position on the display.

[0046] Thus, compared to the fourth time period, during the fifth time period, the world-locked objects (e.g., virtual ring 221, virtual flower 222, and virtual glasses 223) have moved to the left on the display due to the electronic device moving to the right in the XR environment 200. However, since the hand 211 also moves to the right in the XR environment 200, the virtual ring 221 does not move to the left like the other world-locked objects.

[0047] Figure 3An electronic device 300 according to some specific implementations is shown. The electronic device 300 exists in a physical environment. The electronic device 300 includes a posture estimation module 301, which determines the current posture of the electronic device in the physical environment, and determines the predicted posture of the electronic device at a future time. In addition, the posture estimation module 201 determines the current posture of one or more dynamic objects in the physical environment, and determines the predicted posture of one or more dynamic objects at a future time. For example, in various specific implementations, the posture estimation module 301 determines the current posture and predicted posture of the user's hand. In various specific implementations, the posture estimation module 301 determines the current posture and predicted posture of the face of a person (except the user). In various specific implementations, the posture estimation module 301 determines the current posture of the electronic device 300 based on camera-based posture tracking and / or IMU (inertial measurement unit) tracking. In various specific implementations, the posture estimation module 301 determines the predicted posture of the electronic device 300 by inferring the previous movement of the electronic device 300, such as the current speed and / or acceleration of the electronic device 300.

[0048] In various specific implementations, the posture estimation module 301 determines the current posture of one or more objects based on the image of the physical environment. For example, in various specific implementations, the posture estimation module 301 determines the current posture of the user's hand based on a hand tracking algorithm applied to the image of the physical environment. In various specific implementations, the posture estimation module 301 determines the current posture of the person's face based on a face detection algorithm applied to the image of the physical environment. In various specific implementations, the posture estimation module 301 determines the current posture of one or more objects based on an IMU attached to the object. For example, in various specific implementations, the posture estimation module 301 determines the current posture of the user's hand based on IMU data received from a watch worn by the user. As another example, in various specific implementations, the posture estimation module 301 determines the current posture of the user's hand based on IMU data received from a stylus held by the user. As another example, in various specific implementations, the posture estimation module 301 determines the current posture of the person's face based on IMU data received from a head-mounted device worn by the person. In various specific implementations, the posture estimation module 301 determines the predicted posture of one or more objects by inferring the previous movement of one or more objects, such as the current speed and / or acceleration of one or more objects.

[0049] The electronic device 300 includes a content module 310 that generates an image at a display frame rate based on the posture information from the posture estimation module 301, and further includes a display module 320 that displays the image at the display frame rate.

[0050] The content module 310 includes a rendering module 311 that generates a rendered image at a rendering frame rate based on the posture information from the posture estimation module 301. In various specific implementations, the rendering frame rate is approximately equal to the display frame rate. In various specific implementations, the rendering frame rate is less than the display frame rate. For example, in various specific implementations, the rendering frame rate is half of the display frame rate.

[0051] In various specific implementations, the rendered images each include a display-locked content layer and one or more world-locked content layers, the display-locked content layer including display-locked content rendered independently of the pose information from the pose estimation module 301, and the world-locked content layer including world-locked content rendered based on the pose information from the pose estimation module 301. In various specific implementations, the world-locked content layer includes a statically anchored world-locked content layer rendered based on the device pose information of the electronic device 300 and independently of any other pose information, and a layer for each dynamic object tracked by the pose prediction module 301.

[0052] As an example, the rendering module 311 generates a first rendered image in a first time period before a first display time period of the first display image based on a first predicted posture of the electronic device 300 during the first display time period and a first predicted posture of one or more objects from the posture estimation module 301. The first rendered image includes a display-locked content layer, a statically-anchored world-locked content layer, and one or more dynamically-anchored world-locked content layers.

[0053] The content module 310 includes a reprojection module 312 that generates a composite image at a display frame rate based on the updated pose information from the pose estimation module 301. In various implementations, the reprojection module 312 generates a composite image by transforming the world-locked content layers of the rendered image based on the pose information from the pose estimation module 301 and flattening the rendered image into a single layer. In various implementations, the reprojection module 312 transforms each world-locked content layer by applying a homography transformation to the world-locked content layer. In various implementations, the reprojection module 312 transforms each world-locked content layer using one or more other perspective transformations.

[0054] As an example, the reprojection module 312 generates a first composite image in a second time period after the first time period and before the first display time period of the first display image based on the second predicted posture of the electronic device 300 during the first display time period and the second predicted posture of one or more objects from the posture estimation module 301. In various specific implementations, the reprojection module 312 generates the first composite image by transforming the static anchored world locked content layer of the first rendered image based on the second predicted posture of the electronic device 300 and transforming the one or more dynamic anchored world locked content layers of the first rendered image based on the second predicted posture of the electronic device and the second predicted posture of the one or more objects. The posture estimation module 301 generates the second predicted posture after generating the first predicted posture. Compared with the first predicted posture, the second predicted posture generated closer in time to the first display time period is more accurate than the first predicted posture.

[0055] The reprojection module 312 also generates a template for each composite image that indicates how portions of the composite image will be deformed by a post-warping (LSW) module 321, as described further below.

[0056] In various specific implementations, the template includes a matrix of the same resolution as the composite image, and each element of the matrix corresponds to a pixel of the composite image. In various specific implementations, if the corresponding pixel of the composite image should not be deformed by the LSW module 321, each element of the matrix has a value of "0"; if the corresponding pixel of the composite image will be deformed by the LSW module 321 based only on the posture of the electronic device 300, each element of the matrix has a value of "1"; if the corresponding pixel of the composite image will be deformed by the LSW module 321 based on the posture of the electronic device 300 and the posture of the first dynamic object, each element of the matrix has a value of "2"; if the corresponding pixel of the composite image will be deformed by the LSW module 321 based on the posture of the electronic device 300 and the posture of the second dynamic object, each element of the matrix has a value of "3", etc.

[0057] For example, pixels in the composite image corresponding to display locked content have a value of "0", pixels in the composite image corresponding to static anchored world locked content have a value of "1", pixels in the composite image corresponding to dynamic anchored world locked content anchored to, for example, a user's hand have a value of "2", and pixels in the composite image corresponding to dynamic anchored world locked content anchored to, for example, a person's face have a value of "3".

[0058] The template may be defined in a manner other than a matrix. For example, in various implementations, the template includes pixel coordinates that indicate the area to be displaced by the LSW module 321 , such as the corners of a rectangle surrounding the area to be displaced by the LSW module 321 .

[0059] The display module 320 includes an LSW module 321 that generates a display image at a display frame rate based on the updated pose information from the pose estimation module 301. In various implementations, the LSW module 321 performs a one-dimensional or two-dimensional pixel shift of a portion of the synthesized image indicated by the corresponding template based on the updated pose information from the pose estimation module 301.

[0060] As an example, the LSW module 321 generates a first display image based on a third predicted gesture of the electronic device 300 during the first display time period and a third predicted gesture of one or more objects from the gesture estimation module 301 in a third time period after the second time period and before the first display time period of the first display image. In various specific implementations, the LSW module 321 generates the first display image by shifting a portion of the first synthetic image indicated by the template as corresponding to the static anchor world locked content based on the third predicted gesture of the electronic device 300 and shifting a portion of the first synthetic image indicated by the template as corresponding to the dynamic anchor world locked content based on the third predicted gesture of the electronic device 300 and the third predicted gesture of one or more objects. The gesture estimation module 301 generates the third predicted gesture after generating the second predicted gesture. Compared with the second predicted gesture, the third predicted gesture generated closer in time to the first display time period is more accurate than the second predicted gesture.

[0061] The display 322 displays the display image at a display frame rate. As an example, the display 322 displays the first display image in the first display period.

[0062] As described above, in various implementations, the display frame rate is greater than the rendering frame rate. To achieve this, in various implementations, the reprojection module 312 performs frame rate inference.

[0063] As an example, the reprojection module 312 generates a second composite image based on the first rendered image, the first predicted posture of the electronic device 300 during the second display time period from the posture estimation module 301, and the first predicted posture of one or more objects from the posture estimation module 301 in a first time period before the second display time period of the second display image. In various specific implementations, the reprojection module 312 generates the second composite image by transforming the world-locked content layer of the first rendered image based on the first predicted posture of the electronic device and the first predicted posture of the one or more objects and synthesizing the transformed world-locked content layer and the display-locked content layer into a single layer. The LSW module 321 generates a second display image based on the second predicted posture of the electronic device 300 during the second display time period and the second predicted posture of one or more objects from the posture estimation module 301 in a second time period after the first time period and before the second display time period of the second display image. In various specific implementations, the LSW module 321 generates a second display image by displacing a portion of the second composite image indicated by the template as corresponding to the static anchor world locked content based on the second predicted posture of the electronic device 300 and displacing a portion of the second composite image indicated by the template as corresponding to the dynamic anchor world locked content based on the third predicted posture of the electronic device 300 and the third predicted posture of the one or more objects. The display 322 displays the second display image in a second display time period.

[0064] In various implementations, the rendering frame rate is substantially equal to the display frame rate. Therefore, in various implementations, the electronic device 300 does not include the re-projection module 312, and the rendering module 311 outputs a composite image.

[0065] As an example, see Figure 2A ,exist Figure 2A At a first time before the first time period of the first gesture of the electronic device 300, the rendering module 311 renders a display-locked content layer including the virtual clock 224. The rendering module 311 presents a statically anchored world-locked content layer including the virtual flower 222 based on the first prediction of the first gesture of the electronic device 300. The rendering module 311 renders a first dynamically anchored world-locked content layer including the virtual ring 221 based on the first prediction of the first gesture of the electronic device 300 and the first prediction of the first gesture of the hand 211. The rendering module 311 renders a second dynamically anchored world-locked content layer including the virtual glasses 223 based on the first prediction of the first gesture of the electronic device 300 and the first prediction of the first gesture of the person.

[0066] After the first time but Figure 2AAt a second time before the first time period of the electronic device 300, the reprojection module 312 transforms the static anchored world locked content layer based on the second prediction of the first posture of the electronic device 300. The reprojection module 312 transforms the first dynamic anchored world locked content layer based on the second prediction of the first posture of the electronic device 300 and the second prediction of the first posture of the hand 211. The reprojection module 312 transforms the second dynamic anchored world locked content layer based on the second prediction of the first posture of the electronic device 300 and the second prediction of the posture of the person 213.

[0067] The reprojection module 312 flattens the transformed world-locked content layer and the display-locked content layer into a composite image. In addition, the reprojection module 312 generates a template including a value "0" at the pixel position of the virtual clock 224, a value "1" at the pixel position of the virtual flower 222, a value "2" at the pixel position of the virtual ring 221, and a value "3" at the pixel position of the virtual glasses 223.

[0068] After the second time but Figure 2A At a third time before the first time period of the electronic device 300, the LSW module 321 deforms the pixels of the composite image with the corresponding template value "1" based on the third prediction of the first posture of the electronic device 300. The LSW module 321 deforms the pixels of the composite image with the corresponding template value "2" based on the third prediction of the first posture of the electronic device 300 and the third prediction of the first posture of the hand 211. The LSW module 321 deforms the pixels of the composite image with the corresponding template value "3" based on the third prediction of the first posture of the electronic device 300 and the third prediction of the first posture of the person 213.

[0069] exist Figure 2A During the first time period, the display 322 displays the deformed composite image, for example, displays the image.

[0070] As an example of a use that does not include the reprojection module 312, refer to Figure 2A ,exist Figure 2A At a first time before the first time period of the first gesture of the electronic device 300, the rendering module 311 renders a display-locked content layer including the virtual clock 224. The rendering module 311 presents a statically anchored world-locked content layer including the virtual flower 222 based on the first prediction of the first gesture of the electronic device 300. The rendering module 311 renders a first dynamically anchored world-locked content layer including the virtual ring 221 based on the first prediction of the first gesture of the electronic device 300 and the first prediction of the first gesture of the hand 211. The rendering module 311 renders a second dynamically anchored world-locked content layer including the virtual glasses 223 based on the first prediction of the first gesture of the electronic device 300 and the first prediction of the first gesture of the person.

[0071] The rendering module 311 flattens the world-locked content layer and the display-locked content layer into a composite image, and generates a template including a value “0” at the pixel position of the virtual clock 224 , a value “1” at the pixel position of the virtual flower 222 , a value “2” at the pixel position of the virtual ring 221 , and a value “3” at the pixel position of the virtual glasses 223 .

[0072] After the first time but Figure 2A At a second time before the first time period, the LSW module 321 deforms the pixels of the composite image with the corresponding template value "1" based on the second prediction of the first posture of the electronic device 300. The LSW module 321 deforms the pixels of the composite image with the corresponding template value "2" based on the second prediction of the first posture of the electronic device 300 and the second prediction of the first posture of the hand 211. The LSW module 321 deforms the pixels of the composite image with the corresponding template value "3" based on the second prediction of the first posture of the electronic device 300 and the second prediction of the first posture of the person 213.

[0073] exist Figure 2A During the first time period, the display 322 displays the deformed composite image, for example, displays the image.

[0074] Figure 4 4 is a flowchart representation of a method 400 for displaying an image according to some specific implementations. In various specific implementations, the method 400 is performed by an electronic device, such as Figure 1 Electronic equipment 120 or Figure 3 The electronic device 300 of FIG. 4 is a device having an image sensor, a display, one or more processors, and a non-transitory memory. In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0075] Method 400 begins in block 410, where a device captures an image of an object in a physical environment using an image sensor. In various implementations, the object is a dynamic object in the physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is a face of a person (other than the user of the device).

[0076] Method 400 continues in block 420, where the device obtains a first predicted object pose of an object in a physical environment at a display time based on the image. In various specific implementations, obtaining the first predicted object pose includes performing a hand tracking algorithm on the image. In various specific implementations, obtaining the first predicted object pose includes performing a face detection algorithm on the image. In various specific implementations, obtaining the first predicted object pose is based on the image and a previously obtained image. Therefore, in various specific implementations, obtaining the first predicted object pose includes determining a current object pose at each time based on a previously obtained image and the image, and inferring the current object pose to predict a first predicted object pose at the display time.

[0077] The method 400 continues in block 430, where the device renders virtual content based on the first predicted object pose. In various implementations, the method 400 further includes obtaining a first predicted device pose of a device in the physical environment at a first display time. In various implementations, rendering the virtual content is also based on the first predicted device pose.

[0078] Method 400 continues in block 440 where the device obtains a second predicted object pose of an object in the physical environment at display time. In various implementations, obtaining the second predicted object pose is performed after rendering the virtual content or at least after the virtual content rendering has begun.

[0079] In various specific implementations, obtaining the second predicted object pose is based on the additional image. For example, in various specific implementations, obtaining the second predicted object pose includes determining the current object pose at each time based on the previously obtained image, the image, and the additional image, and inferring the current object pose to predict the second predicted object pose at the display time.

[0080] In various implementations, an image is captured at a first capture rate, and an additional image is captured at a second capture rate that is higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

[0081] In various implementations, obtaining the second predicted object pose is based on data received from an inertial measurement unit (IMU). For example, in various implementations, data is received from an IMU of a watch worn by a user. As another example, in various implementations, data is received from an IMU of a stylus held by a user. As another example, in various implementations, data is received from an IMU of a head-mounted device worn by a person (other than the user). In various implementations, images are captured at a capture rate, and data is received at a reception rate higher than the capture rate.

[0082] Method 400 continues in box 450, where the device deforms the virtual content based on the second predicted object pose. In various specific implementations, method 400 includes obtaining a second predicted device pose of the device at display time, and deforming the virtual content is also based on the second predicted device pose. In various specific implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. Therefore, in various specific implementations, determining the second predicted device pose is based on data received from the IMU of the device, and determining the second predicted object pose is based on data received from different IMUs of different devices.

[0083] In various implementations, deforming the virtual content is based on a difference between the first predicted object pose and the second predicted object pose. In various implementations, deforming the virtual content is further based on a difference between the first predicted device pose and the second predicted device object pose.

[0084] In various implementations, deforming the virtual content includes performing a homography transformation of the virtual content. In various implementations, deforming the virtual content includes shifting the virtual content. In various implementations, deforming the virtual content includes scaling the virtual content.

[0085] In various implementations, method 400 further includes generating a template indicating the portion of the virtual content to be deformed based on the second predicted object pose, and deforming the virtual content is also based on the template. In various implementations, the template indicates the portion of the virtual content to be deformed based on the second predicted device pose and without regard to the second predicted object pose. In various implementations, the template also indicates the portion of the virtual content that should not be deformed.

[0086] The method 400 continues at block 460 where the device displays the transformed virtual content on the display at a display time.

[0087] Figure 5 is a flowchart representation of another method 500 for displaying content according to some specific implementations. In various specific implementations, the method 500 is performed by an electronic device, such as Figure 1 Electronic equipment 120 or Figure 3 The electronic device 300 of FIG. 500. In various specific implementations, the method 500 is performed by a device having an image sensor, a display, one or more processors, and a non-transitory memory. In some specific implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 500 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0088] Method 500 begins in block 510, where a device obtains a first predicted device pose of a device at a display time and a first predicted object pose of an object at a display time. In various implementations, the object is a dynamic object in a physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is a face of a person (other than a user of the device).

[0089] In various specific implementations, obtaining a first predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various specific implementations, obtaining a first predicted object pose is based on an image of the object. In various specific implementations, obtaining a first predicted object pose includes performing a hand tracking algorithm on the image. In various specific implementations, obtaining a first predicted object pose includes performing a face detection algorithm on the image. In various specific implementations, obtaining a first predicted object pose is based on the image and a previously obtained image. Therefore, in various specific implementations, obtaining a first predicted object pose includes determining a current object pose at various times based on a previously obtained image and the image, and inferring the current object pose to predict a first predicted object pose at a display time.

[0090] Method 500 continues at block 520 where the device renders virtual content based on the first predicted device pose and the first predicted object pose.

[0091] The method 500 continues in block 530 where the device obtains a second predicted device pose of the device at display time and a second predicted object pose of the object at display time. In various implementations, obtaining the second predicted device pose and the second predicted object pose is performed after rendering the virtual content or at least after the virtual content rendering has begun.

[0092] In various specific implementations, obtaining the second predicted object pose is based on the additional image. For example, in various specific implementations, obtaining the second predicted object pose includes determining the current object pose at each time based on the previously obtained image, the image, and the additional image, and inferring the current object pose to predict the second predicted object pose at the display time.

[0093] In various implementations, an image is captured at a first capture rate, and an additional image is captured at a second capture rate that is higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

[0094] In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various implementations, obtaining the second predicted object pose is based on data received from different inertial measurement units (IMUs) of different devices. For example, in various implementations, data is received from an IMU of a watch worn by a user. As another example, in various implementations, data is received from an IMU of a stylus held by a user. As another example, in various implementations, data is received from an IMU of a head-mounted device worn by a person other than the user. In various implementations, images are captured at a capture rate, and data is received at a reception rate higher than the capture rate.

[0095] Method 500 continues in block 540, where the device deforms the virtual content based on the second predicted device pose and the second predicted object pose. In various implementations, deforming the virtual content is based on a difference between the first predicted device pose and the second device object pose and a difference between the first predicted object pose and the second predicted object pose.

[0096] In various implementations, deforming the virtual content includes performing a homography transformation of the virtual content. In various implementations, deforming the virtual content includes shifting the virtual content. In various implementations, deforming the virtual content includes scaling the virtual content.

[0097] In various implementations, method 500 further includes generating a template indicating the portion of the virtual content to be deformed based on the second predicted device pose and the second predicted object pose, and deforming the virtual content is also based on the template. In various implementations, the template indicates the portion of the virtual content to be deformed based on the second predicted device pose and without regard to the second predicted object pose. In various implementations, the template also indicates the portion of the virtual content that should not be deformed.

[0098] Thus, in various implementations, deforming the virtual content includes deforming a first portion of the virtual content based on the second predicted device pose and the second predicted object pose and deforming a second portion of the virtual content based on the second predicted device pose and independent of the second predicted object pose.

[0099] The method 500 continues at block 550 where the device displays the transformed virtual content on the display at a display time.

[0100] Figure 6 6 is a flowchart representation of a method 600 for inferring an image according to some specific implementations. In various specific implementations, the method 600 is performed by an electronic device, such as Figure 1 Electronic equipment 120 or Figure 3The electronic device 300 of FIG. 600 is a device having an image sensor, a display, one or more processors, and a non-transitory memory. In some implementations, the method 600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0101] Method 600 begins in block 610, where a device captures an image of an object in a physical environment using an image sensor. In various implementations, the object is a dynamic object in the physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is a face of a person (other than the user of the device).

[0102] Method 600 continues in block 620, where the device obtains a first predicted object pose of an object in a physical environment at a first display time based on the image. In various specific implementations, obtaining the first predicted object pose includes performing a hand tracking algorithm on the image. In various specific implementations, obtaining the first predicted object pose includes performing a face detection algorithm on the image. In various specific implementations, obtaining the first predicted object pose is based on the image and previously obtained images. Therefore, in various specific implementations, obtaining the first predicted object pose includes determining a current object pose at each time based on previously obtained images and the image, and inferring the current object pose to predict a first predicted object pose at the display time.

[0103] The method 600 continues in block 630, where the device renders virtual content based on the first predicted object pose. In various implementations, the method 600 further includes obtaining a first predicted device pose of the device in the physical environment at the first display time. In various implementations, rendering the virtual content is also based on the first predicted device pose.

[0104] Method 600 continues at block 640, where the device displays the virtual content on the display at the first display time. Figure 4 and Figure 5 The virtual content is deformed based on an updated predicted object pose of the object at a first display time and / or an updated predicted device pose of the device at a first display time.

[0105] Method 600 continues at block 650 where the device obtains a second predicted object pose of an object in the physical environment at a second display time. In various implementations, obtaining the second predicted object pose is performed after rendering the virtual content or at least after the virtual content rendering has begun.

[0106] In various specific implementations, obtaining the second predicted object pose is based on the additional image. For example, in various specific implementations, obtaining the second predicted object pose includes determining the current object pose at each time based on the previously obtained image, the image, and the additional image, and inferring the current object pose to predict the second predicted object pose at the display time.

[0107] In various implementations, an image is captured at a first capture rate, and an additional image is captured at a second capture rate that is higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

[0108] In various implementations, obtaining the second predicted object pose is based on data received from an inertial measurement unit (IMU). For example, in various implementations, data is received from an IMU of a watch worn by a user. As another example, in various implementations, data is received from an IMU of a stylus held by a user. As another example, in various implementations, data is received from an IMU of a head-mounted device worn by a person (other than the user). In various implementations, images are captured at a capture rate, and data is received at a reception rate higher than the capture rate.

[0109] The method 600 continues in block 660, where the device deforms the virtual content based on the second predicted object pose. In various implementations, the method 600 includes obtaining a second predicted device pose of the device at a second display time, and the virtual content is also based on the second predicted device pose. In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. Therefore, in various implementations, determining the second predicted device pose is based on data received from the IMU of the device, and determining the second predicted object pose is based on data received from different IMUs of different devices.

[0110] In various implementations, deforming the virtual content is based on a difference between the first predicted object pose and the second predicted object pose. In various implementations, deforming the virtual content is further based on a difference between the first predicted device pose and the second predicted device object pose.

[0111] In various implementations, deforming the virtual content includes performing a homography transformation of the virtual content. In various implementations, deforming the virtual content includes shifting the virtual content. In various implementations, deforming the virtual content includes scaling the virtual content.

[0112] In various implementations, method 600 further includes generating a template indicating the portion of the virtual content to be deformed based on the second predicted object pose, and deforming the virtual content is also based on the template. In various implementations, the template indicates the portion of the virtual content to be deformed based on the second predicted device pose and without regard to the second predicted object pose. In various implementations, the template also indicates the portion of the virtual content that should not be deformed.

[0113] The method 600 continues at block 670 where the device displays the transformed virtual content on the display at a second display time.

[0114] Figure 7 700 is a flowchart representation of another method 700 for inferring an image according to some specific implementations. In various specific implementations, the method 700 is performed by an electronic device, such as Figure 1 Electronic equipment 120 or Figure 3 The electronic device 300 of FIG. 700. In various specific implementations, the method 700 is performed by a device having an image sensor, a display, one or more processors, and a non-transitory memory. In some specific implementations, the method 700 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 700 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0115] Method 700 begins in block 710, where a device obtains a first predicted device pose of a device at a first display time and a first predicted object pose of an object at the first display time. In various implementations, the object is a dynamic object in a physical environment. In various implementations, the object is a hand of a user of the device. In various implementations, the object is a face of a person (other than the user of the device).

[0116] In various specific implementations, obtaining a first predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various specific implementations, obtaining a first predicted object pose is based on an image of the object. In various specific implementations, obtaining a first predicted object pose includes performing a hand tracking algorithm on the image. In various specific implementations, obtaining a first predicted object pose includes performing a face detection algorithm on the image. In various specific implementations, obtaining a first predicted object pose is based on the image and a previously obtained image. Therefore, in various specific implementations, obtaining a first predicted object pose includes determining a current object pose at various times based on a previously obtained image and the image, and inferring the current object pose to predict a first predicted object pose at a display time.

[0117] Method 700 continues at block 720 where the device renders virtual content based on the first predicted device pose and the first predicted object pose.

[0118] Method 700 continues at block 730, where the device displays the virtual content on the display at the first display time. Figure 4 and Figure 5 The virtual content is deformed based on an updated predicted object pose of the object at a first display time and / or an updated predicted device pose of the device at a first display time.

[0119] The method 700 continues in block 740, where the device obtains a second predicted device pose of the device at a second display time and a second predicted object pose of the object at the second display time. In various specific implementations, obtaining the second predicted device pose and the second predicted object pose is performed after rendering the virtual content or at least after the virtual content rendering has begun.

[0120] In various specific implementations, obtaining the second predicted object pose is based on the additional image. For example, in various specific implementations, obtaining the second predicted object pose includes determining the current object pose at each time based on the previously obtained image, the image, and the additional image, and inferring the current object pose to predict the second predicted object pose at the display time.

[0121] In various implementations, an image is captured at a first capture rate, and an additional image is captured at a second capture rate that is higher than the first capture rate. In various implementations, the additional image has a lower resolution than the image.

[0122] In various implementations, obtaining the second predicted device pose is based on data received from an inertial measurement unit (IMU) of the device. In various implementations, obtaining the second predicted object pose is based on data received from different inertial measurement units (IMUs) of different devices. For example, in various implementations, data is received from an IMU of a watch worn by a user. As another example, in various implementations, data is received from an IMU of a stylus held by a user. As another example, in various implementations, data is received from an IMU of a head-mounted device worn by a person other than the user. In various implementations, images are captured at a capture rate, and data is received at a reception rate higher than the capture rate.

[0123] The method 700 continues in block 750 where the device deforms the virtual content based on the second predicted device pose and the second predicted object pose. In various implementations, deforming the virtual content is based on a difference between the first predicted device pose and the second device object pose and a difference between the first predicted object pose and the second predicted object pose.

[0124] In various implementations, deforming the virtual content includes performing a homography transformation of the virtual content. In various implementations, deforming the virtual content includes shifting the virtual content. In various implementations, deforming the virtual content includes scaling the virtual content.

[0125] In various implementations, method 700 further includes generating a template indicating the portion of the virtual content to be deformed based on the second predicted device pose and the second predicted object pose, and deforming the virtual content is also based on the template. In various implementations, the template indicates the portion of the virtual content to be deformed based on the second predicted device pose and without regard to the second predicted object pose. In various implementations, the template also indicates the portion of the virtual content that should not be deformed.

[0126] Thus, in various implementations, deforming the virtual content includes deforming a first portion of the virtual content based on the second predicted device pose and the second predicted object pose and deforming a second portion of the virtual content based on the second predicted device pose and independent of the second predicted object pose.

[0127] The method 700 continues at block 760 where the device displays the transformed virtual content on the display at a second display time.

[0128] Figure 8 is a block diagram of one example of a controller 110 according to some implementations. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 802 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 806, one or more communication interfaces 808 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 810, a memory 820, and one or more communication buses 804 for interconnecting these components and various other components.

[0129] In some implementations, the one or more communication buses 804 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices 806 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0130] The memory 820 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 820 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 820 optionally includes one or more storage devices remotely located from the one or more processing units 802. The memory 820 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 820 or the non-transitory computer-readable storage medium of the memory 820 stores the following programs, modules, and data structures or a subset thereof including an optional operating system 830 and an XR experience module 840.

[0131] The operating system 830 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the XR experience module 840 is configured to manage and coordinate one or more XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various specific implementations, the XR experience module 840 includes a data acquisition unit 842, a tracking unit 844, a coordination unit 846, and a data transmission unit 848.

[0132] In some specific implementations, the data acquisition unit 842 is configured to at least Figure 1 The electronic device 120 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.). To this end, in various specific implementations, the data acquisition unit 842 includes instructions and / or logic components for instructions and heuristics and metadata for heuristics.

[0133] In some implementations, the tracking unit 844 is configured to map the physical environment 105 and to track at least the electronic device 120 relative to the physical environment 105. Figure 1 The location / position of the physical environment 105. To this end, in various specific implementations, the tracking unit 844 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0134] In some implementations, the coordination unit 846 is configured to manage and coordinate the XR experience presented to the user by the electronic device 120. To this end, in various implementations, the coordination unit 846 includes instructions and / or logic for instructions and heuristics and metadata for heuristics.

[0135] In some implementations, the data transfer unit 848 is configured to transfer data (e.g., presentation data, location data, etc.) to at least the electronic device 120. To this end, in various implementations, the data transfer unit 848 includes instructions and / or logic components for instructions as well as heuristics and metadata for the heuristics.

[0136] Although the data acquisition unit 842, the tracking unit 844, the coordination unit 846, and the data transmission unit 848 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other specific implementations, any combination of the data acquisition unit 842, the tracking unit 844, the coordination unit 846, and the data transmission unit 848 may be located in separate computing devices.

[0137] also, Figure 8 It serves more as a functional description of various features that may be present in a particular implementation, rather than as a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 8 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0138] Fig. 9is a block diagram of one example of an electronic device 120 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the electronic device 120 includes one or more processing units 902 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 906, one or more communication interfaces 908 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 910, one or more XR displays 912, one or more optional internal-facing and / or external-facing image sensors 914, memory 920, and one or more communication buses 904 for interconnecting these and various other components.

[0139] In some implementations, the one or more communication buses 904 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices and sensors 906 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0140] In some implementations, one or more XR displays 912 are configured to provide an XR experience to the user. In some implementations, one or more displays 912 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more XR displays 912 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the electronic device 120 includes a single XR display. As another example, the electronic device includes an XR display for each eye of the user. In some implementations, one or more XR displays 912 can present MR and VR content.

[0141] In some implementations, the one or more image sensors 914 are configured to obtain image data corresponding to at least a portion of the user's face, including the user's eyes (any of which may be referred to as an eye-tracking camera). In some implementations, the one or more image sensors 914 are configured to face forward so as to obtain image data corresponding to the physical environment that the user would see when the electronic device 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 914 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and the like.

[0142] The memory 920 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 920 includes a non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 920 optionally includes one or more storage devices remotely located from one or more processing units 902. The memory 920 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 920 or the non-transitory computer-readable storage medium of the memory 920 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 930 and an XR rendering module 940.

[0143] The operating system 930 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the XR rendering module 940 is configured to present XR content to a user via one or more XR displays 912. To this end, in various implementations, the XR rendering module 940 includes a data acquisition unit 942, a deformation unit 944, an XR rendering unit 946, and a data transmission unit 948.

[0144] In some specific implementations, the data acquisition unit 942 is configured to at least Figure 1 The controller 110 obtains data (eg, presentation data, interaction data, sensor data, location data, etc.). To this end, in various implementations, the data acquisition unit 942 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0145] In some implementations, the deformation unit 944 is configured to deform the content based on the updated predicted pose of the object. To this end, in various implementations, the deformation unit 944 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0146] In some implementations, the XR rendering unit 946 is configured to display the warped image via one or more XR displays 912. To this end, in various implementations, the XR rendering unit 946 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0147] In some implementations, the data transmission unit 948 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. In some implementations, the data transmission unit 948 is configured to transmit authentication credentials to the electronic device. To this end, in various implementations, the data transmission unit 948 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0148] Although the data acquisition unit 942, the deformation unit 944, the XR rendering unit 946, and the data transmission unit 948 are shown as residing on a single device (e.g., the electronic device 120), it should be understood that in other specific implementations, any combination of the data acquisition unit 942, the deformation unit 944, the XR rendering unit 946, and the data transmission unit 948 may be located in separate computing devices.

[0149] also, Fig. 9 It is more of a functional description of various features that may be present in a particular implementation, as opposed to a schematic diagram of the structure of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Fig. 9 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0150] As described above, one aspect of the present technology is to collect and use data from various sources to improve content registration between virtual content and dynamic objects. The present disclosure contemplates that in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying information or personal information.

[0151] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to improve the content registration between virtual content and the face and / or hands of the person. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health, or can be used as positive feedback to individuals using technology to pursue health goals.

[0152] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and protect access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0153] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to content registration, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0154] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0155] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, the various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, virtual content may be displayed on a person's face and / or hands based on non-personal information data or an absolute minimum measure of personal information (such as content requested by a device associated with the user, other non-personal information available to the device, or publicly available information).

[0156] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0157] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0158] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0159] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device including an image sensor, a display, one or more processors, and a non-transitory memory: capturing images of objects in a physical environment using the image sensor; obtaining a first predicted object pose of the object in the physical environment at a display time based on the image; rendering virtual content based on the first predicted object posture; obtaining a second predicted object posture of the object in the physical environment at the display time; deforming the virtual content based on the second predicted object posture; as well as The deformed virtual content is displayed on the display at the display time. The method of claim 1 , wherein the object is a dynamic object in the physical environment. The method of claim 2 , wherein the object is a hand of a user of the device. The method of claim 2 , wherein the object is a human face. 5 . The method according to claim 1 , wherein obtaining the second predicted object pose is based on an additional image of the object. 6 . The method of claim 1 , wherein obtaining the second predicted object pose is based on data received from an inertial measurement unit. 7 . The method of claim 1 , wherein deforming the virtual content is based on a difference between the first predicted object pose and the second predicted object pose.

8. The method of any one of claims 1 to 7, wherein deforming the virtual content comprises performing a homographic transformation of the virtual content.

9. The method of any one of claims 1 to 8, wherein deforming the virtual content comprises displacing the virtual content.

10. The method of any one of claims 1 to 9, further comprising generating a template indicating a portion of the virtual content to be deformed based on the second predicted object pose, wherein deforming the virtual content is further based on the template. The method of claim 10 , wherein the template further indicates portions of the virtual content that should not be deformed.

12. The method according to any one of claims 1 to 11, further comprising: obtaining a first predicted device posture of the device at the display time, wherein rendering the virtual content is based on the first predicted device posture, and After rendering the virtual content, a second predicted device pose of the device at the display time is obtained, wherein deforming the virtual content is based on the second predicted device pose. The method of claim 12 , wherein obtaining the second predicted device pose is based on data received from an inertial measurement unit of the device.

14. A device comprising: Image sensor; monitor; non-transitory memory; as well as One or more processors configured to: capturing images of objects in a physical environment using the image sensor; obtaining a first predicted object pose of the object in the physical environment at a display time based on the image; rendering virtual content based on the first predicted object posture; obtaining a second predicted object posture of the object in the physical environment at the display time; deforming the virtual content based on the second predicted object posture; as well as The deformed virtual content is displayed on the display at the display time.

15. The apparatus of claim 14, wherein the object is a dynamic object in the physical environment.

16. The device of claim 15, wherein the object is a hand of a user of the device.

17. The apparatus of claim 15, wherein the object is a human face.

18. The apparatus of any one of claims 14 to 17, wherein deforming the virtual content is based on a difference between the first predicted object pose and the second predicted object pose.

19. The apparatus according to any one of claims 14 to 18, wherein the one or more processors are further configured to: obtaining a first predicted device posture of the device at the display time, wherein rendering the virtual content is based on the first predicted device posture, and After rendering the virtual content, a second predicted device pose of the device at the display time is obtained, wherein deforming the virtual content is based on the second predicted device pose.

20. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device comprising an image sensor and a display, cause the device to perform any of the methods of claims 1 to 13.