Adjust the display of images based on device position

Abstract

The present disclosure relates to adjusting the display of an image based on a device position. Various embodiments disclosed herein include devices, systems, and methods for adjusting the display of an image based on a device position. In some embodiments, a device includes an image sensor, a display, non-volatile memory, and one or more processors coupled to the image sensor, the display, and the non-volatile memory. In some embodiments, the method includes obtaining a first image of a body part of a user of the device. In some embodiments, the method includes capturing a second image of the body part via the image sensor. In some embodiments, the method includes determining a position of the body part relative to the device based on the second image. In some embodiments, the method includes generating an adjusted image by adjusting the first image based on the position of the body part relative to the device. In some embodiments, the method includes displaying the adjusted image on a display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 242,179, filed on September 9, 2021, and U.S. Patent Application No. 17 / 877,255, filed on July 29, 2022, which are incorporated by reference in their entireties. Technical Field

[0003] The present disclosure generally relates to adjusting the display of images based on device position. Background Art

[0004] Some devices include an image sensor that captures images and a display that displays the captured images. These images can depict various objects or people. These images can be presented on a mobile communication device. Summary of the Invention

[0005] Various implementations disclosed herein include devices, systems, and methods for adjusting the display of an image based on a device position. In some implementations, a device includes an image sensor, a display, non-volatile memory, and one or more processors coupled to the image sensor, the display, and the non-volatile memory. In some implementations, the method includes obtaining a first image of a body part of a user of the device. In some implementations, the method includes capturing a second image of the body part via the image sensor. In some implementations, the method includes determining a position of the body part relative to the device based on the second image. In some implementations, the method includes generating an adjusted image by adjusting the first image based on the position of the body part relative to the device. In some implementations, the method includes displaying the adjusted image on the display.

[0006] According to some specific implementations, a device includes one or more processors, non-transitory memory, and one or more programs. In some specific implementations, the one or more programs are stored in the non-transitory memory and executed by the one or more processors. In some specific implementations, the one or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by one or more processors of a device, causes the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-transitory memory, and a device for executing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] So that the present disclosure may be understood by those skilled in the art, a more detailed description may be had with reference to aspects of certain exemplary implementations, some of which are illustrated in the accompanying drawings.

[0008] Figure 1A is a side view of a user and an electronic device in an operating environment according to some implementations.

[0009] Figure 1B is a front view of a user and an electronic device according to some implementations.

[0010] Figure 1C is another side view of a user and an electronic device according to some implementations.

[0011] Figure 1D is another front view of a user and an electronic device according to some implementations.

[0012] Figure 2 is a block diagram of an image rendering engine according to some specific implementations.

[0013] Figure 3 is a flowchart representation of a method for adjusting the display of an image based on a device position according to some specific implementations.

[0014] Figure 4 is a block diagram of a device for adjusting the display of an image based on the device's position, according to some implementations.

[0015] As is common practice, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, the dimensions of various features may be arbitrarily expanded or reduced for clarity. Furthermore, some of the accompanying drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used to denote similar features throughout the specification and accompanying drawings. DETAILED DESCRIPTION

[0016] Numerous details are described to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings illustrate only some example aspects of the present disclosure and, therefore, should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example implementations described herein.

[0017] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment may include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner that complies with at least one law of physics. For example, an XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. As another example, an XR system can detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) that presents an XR environment and, in response, adjust the graphical content and sound field presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust features of the graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0018] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some embodiments, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology that projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into a physical environment, such as as a hologram or onto a physical surface.

[0019] A device may include a display that displays an image of a body part of a user of the device. Displaying the image of the body part allows another person in the device's physical environment to view a representation of the body part even when the body part is obscured by the device. For example, if the device obscures a user's hand, the display displays an image of the user's hand, allowing another person in the device's physical environment to view the image of the user's hand obscured by the device. Displaying the image of the body part provides the appearance that a person in the physical environment is directly viewing the body part. For example, displaying an image of the user's hand provides the appearance that the user, or someone interacting with the user, is directly viewing the user's hand. However, misalignment between the image of the body part and the body part can detract from this appearance. For example, if the display displays the user's hand in the center of the display, and the user is holding the device so that the left side of the device just obscures the hand, the position of the pixels corresponding to the user's hand may not match the expected position of the hand. Therefore, the device may need to adjust the display of the image based on the device's position relative to the body part. For example, if the left side of the device just obscures the hand, the device may need to shift the image of the hand toward the left.

[0020] The present disclosure provides methods, systems and / or devices for adjusting the display of an image of a body part based on the position of the body part relative to the device. When the device displays the image of the body part on the display, the device detects whether the body part is aligned with the image of the body part. If the body part and the image of the body part are not aligned, the device shifts the image to align the image with the body part. For example, if the display of the device is displaying an image of a user's hand, the device shifts the image of the user's hand to align the pixels representing the hand with the user's hand. For example, if the user is holding the device further to the right than expected, the device may shift the image to the left to align the image of the hand with the user's hand. For another example, if the user is holding the device further to the left than expected, the device may shift the image to the right to align the image of the hand with the user's hand. Shifting the image of the hand based on the position of the device results in positioning the image of the hand at the expected position on the display to provide the appearance that the user or a person in the physical environment is looking directly at the user's hand.

[0021] Figure 1A is a diagram illustrating an example physical environment 10 according to some implementations. While relevant features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the physical environment 10 includes an electronic device 20 and a user 12 of the electronic device 20. Figure 1A A side view of user 12 and electronic device 20 is shown. In some implementations, electronic device 20 comprises a handheld computing device capable of being held by user 12. For example, in some implementations, electronic device 20 comprises a smartphone, tablet computer, media player, laptop computer, etc. In some implementations, electronic device 20 comprises a wearable computing device capable of being worn by user 12. For example, in some implementations, electronic device 20 comprises a head-mountable device (HMD) or an electronic watch.

[0022] exist Figure 1AIn the example of FIG, electronic device 20 is positioned adjacent to user 12 so as to obscure body part 14 of user 12. In some implementations, body part 14 is a hand of user 12, and electronic device 20 is a tablet or smartphone that user 12 holds near the hand to capture an image of the hand. In some implementations, electronic device 20 comprises a wearable computing device that user 12 wears around body part 14. For example, body part 14 may be a wrist of user 12, and electronic device 20 may comprise a watch that user 12 wears around the wrist. As another example, body part 14 may include an eye of user 12, and electronic device 20 may comprise an HMD that user 12 wears around the head of user 12, thereby covering the eye of user 12. Consequently, a person located in physical environment 10 and interacting with user 12 may not be able to directly view body part 14 (e.g., hand, wrist, or eye) of user 12 because electronic device 20 blocks the line of sight between body part 14 and the eye of the person viewing body part 14.

[0023] In some implementations, electronic device 20 includes an image sensor 22 and a display 26. In some implementations, image sensor 22 faces a first direction, and display 26 faces a second direction opposite the first direction. For example, in some implementations, image sensor 22 is a camera that captures an image of body part 14, and display 26 is a display that displays an image 30 (hereinafter, for brevity, "image 30") of body part 14 toward a person viewing user 12. In some implementations, image 30 includes an image captured by image sensor 22. Alternatively, in some implementations, image 30 includes an image captured during a registration phase.

[0024] like Figure 1A As shown in , image 30 includes a set of one or more pixels 34 (hereinafter referred to as "pixels 34" for simplicity) corresponding to body part 14. For example, if body part 14 is a finger of user 12, pixel 34 represents the finger of user 12. Figure 1A As can be seen in FIG, pixel 34 corresponding to body part 14 does not intersect horizontal axis 40 that intersects body part 14. In other words, pixel 34 corresponding to body part 14 is not aligned with body part 14. Specifically, pixel 34 is a distance 50 away from horizontal axis 40. Figure 1A , pixel 34 does not intersect horizontal axis 40 because user 12 may have positioned electronic device 20 at a distance 50 from the intended position of electronic device 20. For example, user 12 may have positioned electronic device 20 at a different position than the intended position on his / her hand.

[0025] Figure 1A A side view of a user 12 holding or wearing an electronic device 20 is shown, and Figure 1B A front view of a user 12 holding or wearing an electronic device 20 is shown. Figure 1B As can be seen in FIG, the pixel 34 corresponding to the body part 14 is separated from the projection of the body part 14 on the plane formed by the display 26 by a distance 50. Figure 1B As shown, pixel 34 corresponding to body part 14 is not aligned with body part 14. If body part 14 includes a finger of user 12 and pixel 34 represents the finger displayed on display 26, pixel 34 may not be aligned with the finger because user 12 may have positioned electronic device 20 at a different location than expected.

[0026] like Figure 1C and Figure 1D As shown, the electronic device 20 adjusts the display of the image 30 based on the position of the electronic device 20 relative to the body part 14. In various implementations, the electronic device 20 adjusts the display of the image 30 so as to align the pixels 34 with the body part 14. Figure 1C and Figure 1D In the example of , electronic device 20 shifts image 30 downward by a distance 50 so that pixel 34 intersects horizontal axis 40 passing through body part 14. In other words, electronic device 20 shifts image 30 so that the pixel coincides with the projection of body part 14 on the plane formed by display 26. Figure 1D As can be seen in the image, pixel 34 is aligned with body part 14. For example, if body part 14 is a hand and pixel 34 represents a picture of the hand, the picture of the hand is aligned with the hand. Therefore, the picture of the hand is displayed on display 26 at a location that coincides with the projection of the hand. In other words, the picture of the hand is displayed on display 26 at the location where the hand is expected to be located.

[0027] Figure 2 A block diagram of an exemplary image rendering engine 200 is shown. In some implementations, the image rendering engine 200 resides in Figures 1A to 1D . In various specific implementations, the image rendering engine 200 includes a data acquirer 210, a position determiner 220, an image adjuster 230, and a display engine 240. In various specific implementations, the data acquirer 210 obtains an image 212 captured by an image sensor (hereinafter referred to as "captured image 212" for simplicity). For example, the data acquirer 210 obtains an image 212 captured by an image sensor. Figure 1A and Figure 1C The image sensor 22 shown in FIG. receives a captured image 212. In some implementations, the captured image 212 depicts Figures 1A to 1D For example, the captured image 212 includes pixels 34 corresponding to the body part 14 of the user 12. The data acquirer 210 provides the captured image 212 to the position determiner 220.

[0028] In various implementations, the position determiner 220 utilizes the captured image 212 to determine a position 222 of the electronic device 20 relative to the body part 14 depicted in the captured image 212. For example, in some implementations, the position determiner 220 utilizes the captured image 212 to determine the position of the electronic device 20 relative to the hand of the user 12, the finger of the user 12, the arm of the user 12, the leg of the user 12, the torso of the user 12, or the eye of the user 12. In some implementations, the position 222 indicates whether the electronic device 20 is positioned at an expected position relative to the user 12. For example, in some implementations, the position 222 indicates whether the user 12 is holding the electronic device 20 at an expected position relative to the body part 14. As another example, the position 222 indicates whether the user 12 is wearing the electronic device 20 at a position higher or lower than the expected position on his / her wrist. In some implementations, the position 222 includes an offset value (e.g., a distance) indicating the current position of the electronic device 20 and the expected position of the electronic device 20. Figures 1A to 1C The distance 50 shown in FIG.

[0029] In some implementations, the image adjuster 230 determines an adjustment 232 for the image 30 based on the position 222 of the electronic device 20 relative to the body part 14. In some implementations, the adjustment 232 is to shift the image 30 so that the pixels 34 corresponding to the body part 14 are aligned with the position 222 of the body part 14. For example, as described with respect to Figures 1A to 1D As discussed, in some implementations, adjustment 232 is to shift image 30 downward to align pixel 34 with body part 14. Figure 1C and Figure 1D An example of a one-dimensional adjustment 232 is provided (e.g., a vertical adjustment, such as a downward shift), but in some implementations, the adjustment 232 includes a two-dimensional adjustment. For example, in some implementations, the adjustment 232 includes a combination of a vertical shift and a horizontal shift.

[0030] In some implementations, adjustment 232 includes a depth adjustment, where the depth of image 30 varies based on the distance between body part 14 and electronic device 20. For example, if user 12 moves further away from electronic device 20, adjustment 232 may include displaying image 30 at a greater depth so that image 30 appears further away from the person viewing display 26.

[0031] In some implementations, adjusting 232 includes adjusting Figure 1A and Figure 1C. For example, if user 12 holds electronic device 20 with electronic device 20 tilted toward an index finger of user 12, adjustment 232 may include a rotation of image 30 so as to align pixels 34 corresponding to the finger with the finger of user 12. For another example, if electronic device 20 is tilted toward the left eye of user 12, adjustment 232 may include a rotation of image 30 so as to align pixels 34 corresponding to the eye with the eye of user 12.

[0032] In some implementations, display engine 240 obtains an indication of adjustment 232 from image adjuster 230, and display engine 240 performs adjustment 232 on image 30. In some implementations, adjustment 232 is to perform a translation operation (e.g., a shift operation), and display engine 240 translates (e.g., shifts) image 30. In some implementations, adjustment 232 is to perform a rotation operation, and display engine 240 rotates image 30. In some implementations, adjustment 232 is to perform a zoom operation, and display engine 240 zooms a portion of image 30.

[0033] Figure 3 is a flowchart representation of a method 300 for adjusting the display of an image based on a position of an electronic device relative to a body part. In some implementations, the method 300 is performed by a device (e.g., Figures 1A to 1D In some implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 300 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0034] As represented by block 310, in various implementations, method 300 includes obtaining a first image of a body part of a user of a device. In some implementations, method 300 includes retrieving the first image from a database (e.g., local non-transitory memory or a remote database). For example, Figure 1A As shown in FIG, electronic device 20 obtains image 30 depicting body part 14. In some implementations, the device blocks the body part of a user or person from the physical environment of the device. For example, the device may block the line of sight between the eyes of a user or person in the physical environment of the device and the body part. Figures 1A to 1D As shown in FIG, the electronic device 20 hides the body part 14 from view.

[0035] As represented by block 320, in some implementations, method 300 includes capturing a second image of the body part via the image sensor. Figure 1A 、 Figure 1B and Figure 2As discussed, the image sensor 22 captures an image 212 of the body part 14. For example, the device's camera may capture an image of a user's hand, fingers, arm, leg, torso, face, or eye.

[0036] As represented by block 330, in some implementations, method 300 includes determining a position of the body part relative to the device based on the second image. Figure 2 As shown in FIG, the position determiner 220 determines the position 222 of the electronic device 20 relative to the body part 14 based on the captured image 212. Figure 2 As discussed, in some implementations, position 222 indicates the position of electronic device 20 relative to a hand, finger, arm, leg, torso, face, or eye of user 12 .

[0037] In some implementations, method 300 includes determining a distance between a current location of the device and an expected location of the device. Figure 1A , electronic device 20 determines a distance 50 between a first display location of a display pixel 34 and a second display location where horizontal axis 40 intersects display 26. In some implementations, method 300 includes determining a distance between a projection of body part 14 on a plane formed by display 26 and pixel 34 corresponding to body part 14.

[0038] In some implementations, method 300 includes determining whether the body part is aligned with a pixel corresponding to the body part in the first image. Figure 1A , electronic device 20 determines whether pixel 34 corresponding to body part 14 is aligned with body part 14. In some implementations, determining whether pixel 34 is aligned with body part 14 includes determining whether pixel 34 is located on a horizontal axis 40 passing through body part 14. In some implementations, determining whether pixel 34 is aligned with body part 14 includes determining whether pixel 34 coincides with a projection of body part 14 on a plane formed by display 26.

[0039] In some implementations, method 300 includes utilizing stereophotogrammetry to determine the position of a body part relative to the device. In some implementations, method 300 includes utilizing depth sensing to determine the position of a body part relative to the device. As represented by block 330a, in some implementations, the body part includes an eye of the user, and determining the position of the body part includes utilizing triangulation to detect the eye of the user in the second image. In some implementations, the body part includes a pupil of the user, and the electronic device utilizes triangulation to detect the corresponding position of the pupil of the user.

[0040] As indicated at 330b, in some implementations, method 300 includes determining a corresponding position of a body part over a time period and averaging the corresponding positions of the body part over the time period to reduce noise associated with detecting the position. For example, the body part may include an eye of the user, and determining the position of the body part includes determining a corresponding position of a pupil over a time period and averaging the corresponding positions of the pupil over the time period. Averaging the positions of the pupil over the time period tends to reduce noise associated with detecting the pupil position.

[0041] As represented by block 340, in some implementations, method 300 includes generating an adjusted image by adjusting the first image based on the position of the body part relative to the device. Figure 1C As shown in FIG, the electronic device 20 shifts the image 30 downward by a distance 50 in order to align the pixel 34 with the body part 14. Figure 2 As depicted, image adjuster 230 determines adjustment 232 based on position 222 , and display engine 240 performs adjustment 232 on image 30 .

[0042] As represented by block 340a, in some implementations, adjusting the first image includes aligning pixels in the first image corresponding to the body part with the body part. Figure 1C , electronic device 20 shifts image 30 so as to align pixels 34 depicting body part 14 with body part 14. In some implementations, aligning pixels in the first image with the body part includes shifting the first image so that the pixels lie on an axis passing through the body part.

[0043] As represented by block 340b, in some implementations, adjusting the first image includes using a matrix to match a first coordinate space corresponding to the first image to a second coordinate space corresponding to the body part. In some implementations, method 300 includes generating the matrix by identifying a geometry of the body part and encoding the geometry of the body part as coefficients of the matrix.

[0044] As represented by block 340c, in some implementations, adjusting the first image includes rotating the first image. In some implementations, if electronic device 20 is positioned at an angle relative to a horizontal axis passing through the eyes of user 12, electronic device 20 rotates the image of the eye displayed on the display so as to align the image of the eye with the eye. In some implementations, method 300 includes determining that the device does not form a right angle with a vertical axis passing through the nose of user 12 and rotating the image of the eye displayed on the display so as to align the image of the eye with the eye. In some implementations, method 300 includes performing the rotation operation on the first image using a rotation matrix. For example, the device adjusts the first image by multiplying the first image by the rotation matrix.

[0045] In some implementations, adjusting the first image includes shifting the first image. In some implementations, shifting the first image includes shifting the first image horizontally. In some implementations, shifting the first image includes shifting the first image vertically. For example, Figure 1C As shown in , electronic device 20 vertically shifts image 30 a distance 50. In some implementations, shifting the first image includes performing a combination of horizontal shifting and vertical shifting.

[0046] In some implementations, adjusting the first image includes scaling the first image. In some implementations, the device scales the first image by applying a scaling factor that is a function of a difference between an expected position of the device relative to the body part and a current position of the device relative to the body part.

[0047] As represented by block 340d, in some implementations, adjusting the first image includes detecting a misalignment between the body part and the first image of the body part, and adjusting the first image in response to the misalignment being greater than a tolerance threshold. In some implementations, method 300 includes abandoning the adjustment in response to the misalignment being less than the tolerance threshold. For example, Figures 1A to 1D As shown in , the electronic device 20 detects a misalignment between the pixels 34 and the body part 14 , and the electronic device 20 adjusts the display of the image 30 to align the pixels 34 with the body part 14 .

[0048] As represented by block 340e, in some implementations, adjusting the first image includes determining whether the device is moving relative to the body part, adjusting the first image in response to determining that the device is not moving relative to the body part, and forgoing adjusting the first image in response to determining that the device is moving relative to the body part. For example, when the user is positioning the device, the device forgoes adjusting the image displayed on the display. Because performing adjustments utilizes computing resources and consumes battery power, forgoing adjustments when adjustments are not needed conserves limited computing resources and extends the battery life of the device.

[0049] As represented by block 340f, in some implementations, adjusting the first image includes determining whether a person is within a threshold distance of the device, adjusting the first image in response to determining that a person is within the threshold distance of the device, and forgoing adjusting the first image in response to determining that no person is within the threshold distance of the device. The device forgoes adjusting the image when no person is within a reasonable viewing distance of the display. Forgoing adjusting the image when no one is expected to view the image conserves battery power, thereby extending the battery life of the device.

[0050] As represented by block 340g, in some implementations, adjusting the first image includes determining a battery level of the device, adjusting the first image in response to the battery level meeting a threshold battery level, and abandoning adjusting the first image in response to the battery level exceeding the threshold battery level. When the battery level is below the threshold battery level, the device abandons adjusting the image. When the battery level is below the threshold battery level, abandoning adjusting the image extends the battery life of the device by reducing battery consumption.

[0051] As represented by block 350, in various implementations, method 300 includes displaying the adjusted image on a display. Figure 1C and Figure 1D As shown, electronic device 20 displays the adjusted version of image 30 .

[0052] See again Figures 1A to 1D In some implementations, the electronic device 20 includes or is attached to a head-mounted device (HMD) worn by the user 12. Depending on various implementations, the HMD presents (e.g., displays) an XR environment. In some implementations, the HMD includes an integrated display (e.g., a built-in display) that displays the XR environment. In some implementations, the HMD includes a head-mounted housing. In various implementations, the head-mounted housing includes an attachment area to which another device with a display can be attached. For example, in some implementations, the electronic device 20 can be attached to the head-mounted housing. In various implementations, the head-mounted housing is shaped to form a receiver for receiving another device including a display (e.g., the electronic device 20). For example, in some implementations, the electronic device 20 slides / snaps into the head-mounted housing or is otherwise attached to the head-mounted housing. In some implementations, the display of the device attached to the head-mounted housing presents (e.g., displays) the XR environment. In various implementations, examples of the electronic device 20 include smartphones, tablets, media players, laptops, etc.

[0053] In some implementations, if the device obscures a user's eyes, the display displays an image of the user's eyes so that another person in the device's physical environment can view the image of the user's eyes obscured by the device. Displaying the image of the user's eyes provides the appearance that the person interacting with the user is looking directly into the user's eyes. However, misalignment between the image of the eyes and the eyes can detract from this appearance. For example, if the display is displaying the user's eyes and the user is wearing the device higher on their face than expected, the position of the pixels corresponding to the user's eyes may not match the expected position of the eyes. Therefore, the device may need to adjust the display of the image based on the position of the device relative to the body part. In such implementations, the device shifts the image of the user's eyes to align the pixels representing the eyes with the user's eyes. For example, if the user is wearing the device higher on their face than expected, the device may shift the image toward a lower position to align the image of the eyes with the user's eyes. For another example, if the user is wearing the device lower on their face than expected, the device may shift the image toward a higher position to align the image of the eyes with the user's eyes. Shifting the image of the eye based on the position of the device results in positioning the image of the eye at an expected location on the display to provide the appearance that a person in the physical environment is looking directly into the user's eyes.

[0054] Figure 4 is a block diagram of a device 400 according to some implementations. In some implementations, the device 400 implements Figures 1A to 1D The electronic device 20 and / or Figure 2 . Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some implementations, device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, one or more input / output (I / O) devices 410, and one or more communication buses 405 for interconnecting these and various other components.

[0055] In some implementations, a network interface 402 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network comprising one or more compatible devices. In some implementations, one or more communication buses 405 include circuits for interconnecting and controlling communications between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices located remotely from one or more CPUs 401. Memory 404 includes non-transitory computer-readable storage media.

[0056] In some implementations, the memory 404 or the non-transitory computer-readable storage medium of the memory 404 stores the following programs, modules, and data structures, or a subset thereof, including the optional operating system 406, the data acquirer 210, the position determiner 220, the image adjuster 230, and the display engine 240. In various implementations, the device 400 executes Figure 3 The method 300 shown in FIG.

[0057] In some implementations, the data acquirer 210 includes instructions 210a and heuristics and metadata 210b for acquiring (e.g., receiving and / or capturing) Figure 2 In some implementations, the data acquirer 210 executes the Figure 3 At least some of the operations represented by block 320 in FIG.

[0058] In some implementations, the position determiner 220 includes instructions 220a and heuristics and metadata 220b for determining a position 222 of the electronic device 20 relative to the body part 14 depicted in the captured image 212. In some implementations, the position determiner 220 performs the following operations: Figure 3 At least some of the operations represented by block 330 in FIG.

[0059] In some implementations, the image adjuster 230 includes instructions 230a and heuristics and metadata 230b for determining adjustments 232 to the image 30 based on the position 222 of the electronic device 20 relative to the body part 14. In some implementations, the image adjuster 230 performs the following operations: Figure 3 At least some of the operations represented by block 340 in FIG.

[0060] In some implementations, the display engine 240 includes instructions 240a and heuristics and metadata 240b for adjusting the image 30 based on the position 222 of the electronic device 20 relative to the body part 14. In some implementations, the display engine 240 performs the following operations: Figure 3 At least some of the operations represented by block 340 in FIG.

[0061] In some implementations, the one or more I / O devices 410 include an input device for obtaining input (e.g., user input, images, and / or environmental data). In some implementations, the one or more I / O devices 410 include a touch screen, a depth sensor (e.g., a depth camera), and / or an image sensor (e.g., a camera, such as a visible light camera or an infrared camera, such as a Figures 1A to 1D In some implementations, one or more I / O devices 410 include an environmental sensor, such as an ambient light sensor. In some implementations, one or more I / O devices 410 include a display (e.g., Figures 1A to 1D 26 shown in FIG).

[0062] In various implementations, one or more I / O devices 410 include a video see-through display that displays at least a portion of the physical environment surrounding device 400 as an image captured by a scene camera. In various implementations, one or more I / O devices 410 include an optical see-through display that is at least partially transparent and passes light emitted or reflected by the physical environment.

[0063] It should be understood that Figure 4 Serves as a functional description of various features that may be present in a particular implementation, as distinct from the structural diagrams of the implementations described herein. As will be appreciated by one of ordinary skill in the art, items shown separately may be combined, and some items may be separated. For example, Figure 4 Some functional blocks shown separately in the figure may be implemented as a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of blocks and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation and, in some specific implementations, will depend in part on the specific combination of hardware, software, and / or firmware selected for a particular embodiment.

[0064] The various processes defined herein contemplate the option of obtaining and utilizing users' personal information. For example, such personal information may be utilized to provide enhanced privacy screens on electronic devices. However, to the extent such personal information is collected, it should be obtained with the user's informed consent. As described herein, users should understand and control the use of their personal information.

[0065] Personal information will be used by appropriate parties only for legitimate and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that, at a minimum, comply with applicable laws and regulations. Furthermore, such policies should be comprehensive, accessible, and recognized to meet or exceed government / industry standards. Furthermore, parties may not distribute, sell, or otherwise share such information except for any legitimate and lawful purpose.

[0066] However, users can limit the extent to which parties can access or otherwise obtain personal information. For example, settings or other preferences can be adjusted to allow users to determine whether their personal information is accessible to various entities. Furthermore, while some features defined herein are described in the context of using personal information, aspects of these features can be implemented without the use of such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that it does not identify the corresponding user.

[0067] Although various aspects of specific implementations within the scope of the appended claims have been described above, it should be apparent that the various features of the above-described specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such an apparatus and / or such a method can be practiced.

Claims

1. A method for adjusting the display of an image, comprising: At a device comprising an image sensor, a display, a non-transitory memory, and one or more processors coupled to the image sensor, the display, and the non-transitory memory: obtaining a first image of a body part of a user of the device; capturing, via the image sensor, a second image of the body part, wherein the device occludes the body part when the user wears the device; determining a position of the body part relative to the device based on the second image; generating an adjusted image by adjusting the first image based on the position of the body part relative to the device such that the first image is aligned with the body part; as well as The adjusted image is displayed on the display. 2 . The method of claim 1 , wherein adjusting the first image comprises aligning pixels in the first image corresponding to the body part with the body part. 3 . The method of claim 1 , wherein adjusting the first image comprises matching a first coordinate space corresponding to the first image with a second coordinate space corresponding to the body part using a matrix. 4 . The method of claim 3 , further comprising generating the matrix by identifying a geometry of the body part and encoding the geometry of the body part. The method of claim 1 , wherein adjusting the first image comprises rotating the first image. The method of claim 1 , wherein adjusting the first image comprises shifting the first image. The method of claim 1 , wherein adjusting the first image comprises scaling the first image.

8. The method of claim 1 , wherein adjusting the first image comprises: detecting a misalignment between the body part and the first image of the body part; and The first image is adjusted in response to the misalignment being greater than a tolerance threshold.

9. The method of claim 1 , wherein adjusting the first image comprises: determining whether the device is moving relative to the body part; adjusting the first image in response to determining that the device is not moving relative to the body part; as well as Responding to determining that the device is moving relative to the body part, adjusting the first image is abandoned.

10. The method of claim 1, wherein adjusting the first image comprises: determining whether a person is within a threshold distance of the device; adjusting the first image in response to determining that a person is within the threshold distance of the device; as well as Adjusting the first image is abandoned in response to determining that no person is within the threshold distance of the device.

11. The method of claim 1 , wherein adjusting the first image comprises: determining a battery level of the device; adjusting the first image in response to the battery level satisfying a threshold battery level; as well as Adjusting the first image is abandoned in response to the battery level breaching the threshold battery level.

12. The method of claim 1 , wherein the body part comprises an eye of the user; and Wherein determining the position of the body part comprises: Determine the relative position of the pupil over a period of time; as well as The corresponding positions of the pupil within the time period are averaged.

13. The method of claim 1, wherein the device comprises a wearable computing device.

14. An electronic device comprising: one or more processors; Image sensor; monitor; non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the electronic device to: obtaining a first image of a body part of a user of the electronic device; capturing, via the image sensor, a second image of the body part, wherein the electronic device occludes the body part when the user wears the electronic device; determining a position of the body part relative to the electronic device based on the second image; generating an adjusted image by adjusting the first image based on the position of the body part relative to the electronic device such that the first image is aligned with the body part; as well as The adjusted image is displayed on the display.

15. The electronic device of claim 14, wherein adjusting the first image comprises aligning pixels in the first image corresponding to the body part with the body part. 16 . The electronic device of claim 14 , wherein adjusting the first image comprises matching a first coordinate space corresponding to the first image with a second coordinate space corresponding to the body part using a matrix.

17. The electronic device of claim 16, wherein the one or more programs further cause the electronic device to generate the matrix by identifying a geometric shape of the body part and encoding the geometric shape of the body part.

18. The electronic device of claim 14, wherein adjusting the first image comprises rotating the first image.

19. The electronic device of claim 14, wherein adjusting the first image comprises shifting the first image.

20. A non-transitory memory storing one or more programs that, when executed by one or more processors of a device comprising a display and an image sensor, cause the device to: obtaining a first image of a body part of a user of the device; capturing, via the image sensor, a second image of the body part, wherein the device occludes the body part when the user wears the device; determining a position of the body part relative to the device based on the second image; generating an adjusted image by adjusting the first image based on the position of the body part relative to the device such that the first image is aligned with the body part; as well as The adjusted image is displayed on the display.

Images