Wearable device, method, and non-transitory computer-readable storage medium for displaying visual object corresponding to external object

Abstract

The present invention may comprise: a memory for storing instructions and including one or more storage media; one or more cameras; a display assembly including a display; and at least one processor including processing circuitry, wherein the instructions, when individually or collectively executed by the processor, cause the wearable device to: display, on the display assembly, an avatar representing a user; obtain images by using the one or more cameras while displaying the avatar; detect, using at least a portion of the images, an external object gripped by the user's hand; identify a first size of the user's hand by using the at least a portion of the images on the basis of the detection; identify a second size of a visual object corresponding to the external object according to the first size; and display, on the display assembly, the avatar gripping the visual object having the second size.

Description

Wearable device, method, and non-transitory computer-readable storage medium for displaying a visual object corresponding to an external object

[0001] The present disclosure relates to a wearable device, a method, and a non-transitory computer-readable storage medium for displaying a visual object corresponding to an external object.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] A wearable device is described. The wearable device may include a memory storing instructions and including one or more storage media, a display assembly including one or more cameras, a display, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display an avatar representing a user on the display assembly. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a first size of the hand of the user using at least a portion of the images based on the detection. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a second size of a visual object corresponding to the external object based on the first size. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, on the display assembly, the avatar gripping the visual object having the second size.

[0005] A method is described. The method may be performed in a wearable device comprising a display assembly including one or more cameras and a display. The method may include an operation of displaying an avatar representing a user on the display assembly. The method may include an operation of acquiring images using the one or more cameras while displaying the avatar. The method may include an operation of detecting an external object gripped by a hand of the user using at least a portion of the images. The method may include an operation of identifying a first size of the hand of the user using at least a portion of the images based on the detection. The method may include an operation of identifying a second size of a visual object corresponding to the external object based on the first size. The method may include an operation of displaying the avatar gripping the visual object having the second size on the display assembly.

[0006] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a display assembly including one or more cameras and a display, cause the wearable device to display an avatar representing a user on the display assembly. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify a first size of the hand of the user using at least a portion of the images based on the detection. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify a second size of a visual object corresponding to the external object based on the first size. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, on the display assembly, the avatar gripping the visual object having the second size.

[0007] A wearable device is described. The wearable device may include a memory storing instructions and including one or more storage media, a display assembly including one or more cameras, a display, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display an avatar representing a user on the display assembly. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain a visual object corresponding to the external object using at least a portion of the images based on the detection. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in a first orientation such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user, based on a first object display mode.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in a second orientation so that the portion of the visual object is visible to another user, based on a second object display mode.

[0008] A method is described. The method may be performed in a wearable device comprising a display assembly including one or more cameras and a display. The method may include displaying an avatar representing a user on the display assembly. The method may include acquiring images using the one or more cameras while displaying the avatar. The method may include detecting an external object gripped by a hand of the user using at least a portion of the images. The method may include acquiring a visual object corresponding to the external object using at least a portion of the images based on the detection. The method may include displaying the visual object gripped by the avatar in a first orientation based on a first object display mode such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user. The method may include displaying the visual object gripped by the avatar in a second orientation based on a second object display mode such that the portion of the visual object is visible to another user.

[0009] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a display assembly including one or more cameras and a display, cause the wearable device to display an avatar representing a user on the display assembly. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to obtain a visual object corresponding to the external object using at least a portion of the images based on the detection. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in a first orientation such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user, based on a first object display mode.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object drawn by the avatar in a second orientation so that the portion of the visual object is visible to another user, based on a second object display mode.

[0010] Figure 1 illustrates an example of a visual object corresponding to an external object that is not displayed.

[0011] Figure 2 is a simplified block diagram of an exemplary wearable device.

[0012] FIG. 3 is a flowchart illustrating exemplary operations of a wearable device for detecting an external object gripped by a user's hand.

[0013] Figure 4 illustrates an example of detecting an external object gripped by a user's hand.

[0014] FIG. 5 is a flowchart illustrating exemplary operations of a wearable device for displaying an avatar that draws a visual object.

[0015] Figure 6 illustrates an example of displaying an avatar that draws a visual object.

[0016] Figure 7 illustrates an example of displaying an avatar that draws a visual object based on the second object display mode.

[0017] Figure 8 illustrates an example of displaying a visual object drawn by an avatar along with the back of the avatar, based on the second object display mode.

[0018] Figure 9 illustrates examples of components of a wearable device.

[0019] FIG. 10 is a block diagram of an electronic device within a network environment according to various embodiments.

[0020] Figure 11a shows an example of a perspective view of a wearable device.

[0021] FIG. 11b illustrates an example of one or more hardware devices arranged within a wearable device.

[0022] Figures 12a and 12b show an example of the appearance of a wearable device.

[0023] Figure 13 shows an example of a block diagram of a wearable device.

[0024] Fig. 14 shows an example of a block diagram of an electronic device for displaying an image in a virtual space.

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0026] The metaverse is a portmanteau of the English word "meta," meaning "virtual" or "transcendence," and "universe," and can refer to a three-dimensional virtual world where social, economic, and cultural activities similar to those in the real world take place. The metaverse is a concept that is one step more advanced than virtual reality (VR, a cutting-edge technology that allows people to have realistic experiences in a computer-generated virtual world), and can have the characteristic of allowing people to engage in social and cultural activities similar to those in the real world, rather than simply enjoying games or virtual reality using avatars. The metaverse service can provide media content to enhance immersion in the virtual world based on augmented reality (AR), virtual reality environment (VR), mixed reality (MR), and / or extended reality (XR).

[0027] For example, media content provided by a metaverse service may include social interaction content, including avatar-based games, concerts, parties, and / or conferences. For example, the media content may include advertisements, user-created content, and / or information for economic activities such as the sale and / or shopping of products. Ownership of the user-created content may be proven by a blockchain-based non-fungible token (NFT). The metaverse service may support economic activities based on real-world currency and / or cryptocurrency. The metaverse service may provide virtual content linked to the real world, such as digital twins or life logging.

[0028] Figure 1 illustrates an example of a visual object corresponding to an external object that is not displayed.

[0029] Referring to FIG. 1, a wearable device (100) may include a head-mounted display (HMD) that can be worn on the head of a user (105). The wearable device (100) may be described as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device.

[0030] The wearable device (100) may include a display assembly including a display (e.g., the display assembly (240) of FIG. 2) and one or more cameras (e.g., one or more cameras (230) of FIG. 2). The wearable device (100) may display an avatar (125) corresponding to a user (105) through the display assembly. For example, the avatar (125) displayed through the display assembly may be visible to other users.

[0031] According to one embodiment, the wearable device (100) may acquire images of a space in front of the wearable device (100) using one or more cameras while displaying an avatar (125) corresponding to a user (105). The wearable device (100) may identify movements of the user (105) using at least some of the images. The wearable device (100) may display an avatar (125) that moves according to the movements of the user (105) identified using at least some of the images.

[0032] In one embodiment, a user (105) may grip an external object (110) with a hand (115) of the user (105). The wearable device (100) may refrain from displaying a visual object (135) corresponding to the external object (110) even if the user (105) grips the external object (110) with the hand (115) of the user (105). By the wearable device (100) refraining from displaying the visual object (135) corresponding to the external object (110), the visual object (135) may not be displayed on a portion (130) of the avatar (125) corresponding to the hand (115) of the user (105) that has a shape that grips the visual object (135). For example, since the wearable device (100) refrains from displaying a visual object (135) corresponding to an external object (110), the visual object (135) may not be visible to other users, which may cause other users to feel uncomfortable. For example, since the visual object (135) is not visible to other users, the user (105) may feel uncomfortable because the user (105) cannot introduce or show the external object (110) to other users.

[0033] A method may be required to resolve the inconvenience of other users and the inconvenience of the user (105) caused by the visual object (135) not being displayed or not being visible to other users. To resolve this inconvenience, the wearable device (100) may display a visual object (135) corresponding to an external object (110) gripped by the hand (115) of the user (105). To display the visual object (135), at least a portion of images acquired using one or more cameras may be utilized.

[0034] The wearable device (100) can execute the operations exemplified in the description of FIGS. 3 to 8 to display a visual object (135) corresponding to an external object (110) gripped by a hand (115) of a user (105). The wearable device (100) can include components for executing the operations. The components can be exemplified in the description of FIG. 2.

[0035] Figure 2 is a simplified block diagram of an exemplary wearable device.

[0036] According to one embodiment, referring to FIG. 2, the wearable device (200) may be described as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device that can be worn on a user's head. An example of the structure of the wearable device (200) that can be worn on a user's head is described with reference to FIGS. 11A, 11B, 12A, and / or 12B. The wearable device (200) may include at least a portion of the electronic device (1001) of FIG. 10, or may correspond to at least a portion of the electronic device (1001) of FIG. 10. A wearable device (200) may include at least one processor (210), memory (220), one or more cameras (230), and a display assembly (240).

[0037] According to one embodiment, at least one processor (210) may include processing circuitry. The at least one processor (210) may include a central processing unit (CPU) (e.g., including processing circuitry). The at least one processor (210) may include a graphic processing unit (GPU) (e.g., including processing circuitry) and a neural processing unit (NPU) (e.g., including processing circuitry). For example, the at least one processor (210) may be configured to control the memory (220), one or more cameras (230), and the display assembly (240). The at least one processor (210) may be configured to individually or collectively execute instructions stored in the memory (220) to cause the wearable device (200) (or the wearable device (100)) to perform at least some of the operations illustrated in the description of FIG. 1. At least one processor (210) may be configured to individually or collectively execute instructions stored in the memory (220) to cause the wearable device (200) to perform at least some of the operations exemplified in the descriptions of FIGS. 3 through 8.

[0038] According to one embodiment, the memory (220) may include one or more storage media. The memory (220) may store various data used by at least one component of the wearable device (200) (e.g., at least one processor (210), the memory (220), one or more cameras (230), and / or the display assembly (240)). For example, the data may include input data or output data for software and commands related thereto. The memory (220) may include volatile memory or non-volatile memory.

[0039] According to one embodiment, the one or more cameras (230) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal representing the color and / or brightness of light. For example, the one or more cameras (230) may be described as an image sensor. For example, the one or more cameras (230) may be available to acquire images of the space in front of the wearable device (200) (or the surrounding environment). For example, at least some of the one or more cameras (230) may have a field of view (FOV) that corresponds to the FOV of a user's eye. For example, the FOV of some of the one or more cameras (230) may be different from the FOV of other some of the one or more cameras (230).

[0040] According to one embodiment, the display assembly (240) may be configured to display an avatar, a visual object, and / or a user interface (UI). The display assembly (240) may include at least one display positioned to face the eyes of a user wearing the wearable device (200).

[0041] The wearable device (200) illustrated in the description of FIG. 2 can execute at least some of the operations illustrated in the description of FIGS. 3 to 8. The operations illustrated in the description of FIGS. 3 to 8 can be caused by (or within) the wearable device (200) under the control of at least one processor (210).

[0042] FIG. 3 is a flowchart illustrating exemplary operations of a wearable device for detecting an external object gripped by a user's hand.

[0043] Referring to FIG. 3, in operation 300, at least one processor (210) may display an avatar (e.g., an avatar (125) of FIG. 1) representing a user (e.g., a user (105) of FIG. 1) on a display assembly (240). According to one embodiment, the avatar may be displayed within a virtual space provided through the display assembly (240). The virtual space may be described as a space for media content provided by a metaverse service. The virtual space may include an immersive environment virtual reality environment and / or a simulated space. At least one processor (210) may perform operation 310 while displaying the avatar.

[0044] In operation 310, according to one embodiment, at least one processor (210) may acquire images using one or more cameras (e.g., 230 of FIG. 2) while displaying an avatar. For example, the images may include images of the space in front of the wearable device (200) and / or images of the surrounding environment of the wearable device (200).

[0045] In operation 320, according to one embodiment, at least one processor (210) may detect an external object gripped by a user's hand using at least a portion of images acquired using one or more cameras (230). Detecting an external object gripped by a user's hand is exemplified within the description of FIG. 4.

[0046] Figure 4 illustrates an example of detecting an external object gripped by a user's hand.

[0047] Referring to FIG. 4, according to one embodiment, a state (400) may be described as a state in which an external object (410) is gripped by a hand (415) of a user (405). Within the state (400), at least one processor (210) may acquire images using one or more cameras (e.g., 230 of FIG. 2). The hand (415) of the user (405) and the external object (410) gripped by the hand (415) of the user (405) may be represented as a visual object (430) corresponding to the hand (415) of the user (405) and a visual object (425) corresponding to the external object (410) within at least a portion (420) of the images. At least one processor (210) may identify a position of the hand (415) of the user (405) and a position of the external object (410) using at least a portion (420) of the images.

[0048] In one embodiment, as the user (405) moves the hand (415) gripping the external object (410), the wearable device (200) may transition from state (400) to state (435). Within state (435), at least one processor (210) may track the hand (415) of the user (405) using images acquired through one or more cameras (230). By tracking the hand (415) of the user (405), the at least one processor (210) may identify that the position of the hand (415) of the user (405) is moving.

[0049] In one embodiment, the user's (405) hand (415) and the external object (410) may move together as the user moves the hand (415) gripping the external object (410). For example, at least one processor (210) may identify that the position of the user's (405) hand (415) and the position of the external object (410) have moved as the position of the visual object (430) corresponding to the user's (405) hand (415) and the position of the visual object (425) corresponding to the external object (410) have moved within at least a portion (440) of the images. At least one processor (210) can identify that the hand (415) of the user (405) and the external object (410) move together by identifying that the position of the visual object (430) corresponding to the hand (415) of the user (405) and the position of the visual object (425) corresponding to the external object (410) move within at least a portion (440) of the images. At least one processor (210) can detect the external object (410) gripped by the hand (415) of the user (405) by identifying that the position of the external object (410) moves according to the movement of the position of the hand (415) of the user (405). For example, the at least one processor (210) can detect the external object (410) gripped by the hand (415) of the user (405) by using a hand tracking processing unit (e.g., the hand tracking processing unit (920) of FIG. 9).

[0050] According to one embodiment, at least one processor (210) may display a visual object (425) corresponding to the external object (410) based on detecting an external object (410) gripped by a hand (415) of a user (405). Displaying the visual object is exemplified within the description of FIG. 5.

[0051] FIG. 5 is a flowchart illustrating exemplary operations of a wearable device for displaying an avatar that draws a visual object.

[0052] Referring to FIG. 5, according to one embodiment, at operation 500, at least one processor (210) may identify a first size of a hand of a user using at least a portion of images acquired through one or more cameras (230) based on detecting an external object (e.g., an external object (410) of FIG. 4) gripped by a hand (e.g., a hand (415) of FIG. 4) of a user (e.g., a user (405) of FIG. 4). At least one processor (210) may identify a second size of the external object using at least a portion of images acquired through one or more cameras (230). At least one processor (210) may identify a ratio of the first size to the second size. The ratio of the first size to the second size may be described as a relative size of the external object with respect to a size of the hand of the user.

[0053] In operation 510, at least one processor (210) can identify a fourth size of a visual object corresponding to an external object based on a first size of the user's hand. To identify the fourth size of the visual object, a relative size of the external object with respect to the size of the user's hand can be utilized. The at least one processor (210) can determine the fourth size by applying a ratio of the first size and the second size to a ratio of the third size of the portion representing the avatar's hand and the fourth size of the visual object. For example, the ratio of the first size of the user's hand and the second size of the external object can correspond to, or substantially correspond to, a ratio of the third size of the portion representing the avatar's hand and the fourth size of the visual object.

[0054] In another embodiment, at least one processor (210) may adjust the second size of the external object to a fourth size based on the third size of the portion representing the avatar's hand. At least one processor (210) may adjust the second size of the external object to a fourth size based on a ratio of the first size of the user's hand to the third size of the portion representing the avatar's hand, thereby displaying the visual object in the fourth size.

[0055] In operation 520, according to one embodiment, at least one processor (210) may display an avatar that has drawn a visual object having a third size on the display assembly (240). By displaying the avatar that has drawn the visual object, the visual object may be visible to other users. Displaying the avatar that has drawn the visual object is exemplified in the description of FIG. 6.

[0056] Figure 6 illustrates an example of displaying an avatar that draws a visual object.

[0057] Referring to FIG. 6, within a state (600), a user (405) may grip an external object (410) with a hand (415) of the user (405). At least one processor (210) may, based on detecting the external object (410) gripped by the hand (415) of the user (405), display an avatar (605) gripping a visual object (610) corresponding to the external object (410) on the display assembly (240).

[0058] According to one embodiment, within the state (600), the hand (415) of the user (405) may have a first size, and the external object (410) may have a second size. At least one processor (210) may identify the first size of the hand (415) and the second size of the external object (410) using images acquired through one or more cameras (230). Within the virtual space, a portion (615) of the avatar (605) representing the hand of the avatar (605) may have a third size. At least one processor (210) may determine a fourth size of the visual object (610) based on the first size, the second size, and the third size. At least one processor (210) may determine the fourth size by causing a ratio of the first size to correspond to a ratio of the third size to the fourth size.

[0059] According to one embodiment, at least one processor (210) can increase a fourth size of the visual object (610) to a fifth size by performing up-sampling on the determined external visual object (610), or can reduce the fourth size of the visual object (610) to a sixth size by performing down-sampling on the visual object (610). The at least one processor (210) can render the visual object (610) having the fifth size based on the fifth size increased by performing up-sampling, or can render the visual object (610) having the sixth size based on the sixth size reduced by performing down-sampling.

[0060] According to one embodiment, at least one processor (210) can display an avatar (605) that draws a visual object (610) having a fourth size. The at least one processor (210) can determine a shape of the visual object (610) based on a shape of an external object (410) included in at least a portion of images acquired through one or more cameras (230). The at least one processor (210) can render the shape of the visual object (610) based on the shape of the external object (410) and the fourth size, thereby displaying a visual object (610) having the shape of the external object (410) and the fourth size on the display assembly (240).

[0061] According to one embodiment, at least one processor (210) can separate the background and the external object (410) from at least a portion of the images and extract the external object (410). At least one processor (210) can use at least a portion of the images to identify the visual object (610) and the background, thereby separating the visual object (610) from the background and avoiding (or not displaying) the background in the virtual space, and display the visual object (610). At least one processor (210) can display the visual object (610) so that the visual object (610) is naturally connected with the background and the part (615) of the avatar (605) that represents the hand of the avatar (605) by performing blurring on the edge of the visual object (610). For example, at least one processor (210) can perform blurring on the edge of the visual object (610) using a punch-through control unit (e.g., the punch-through control unit (950) of FIG. 9).

[0062] In one embodiment, at least one processor (210) displays an avatar (605) that has drawn a visual object (610), so that the visual object (610) can be shown to other users. By showing the visual object (610) to other users, the user (405) can introduce or show the external object (410) to other users. By showing the part (615) representing the hand of the avatar (605) together with the visual object (610) to other users, the movement of the avatar (605) that has drawn the visual object (610) can be naturally shown to other users.

[0063] In one embodiment, a user (405) can release an external object (410). At least one processor (210) can detect, using at least a portion of the images, the external object (410) released by the hand (415) of the user (405) while displaying an avatar (605) that has drawn a visual object (610). At least one processor (210) can detect, using at least a portion of the images, the external object (410) released by the hand (415) of the user (405) based on identifying, using at least a portion of the images, an external object (410) that moves differently from the position of the hand (415) of the user (405). At least one processor (210) can display an avatar (605) that released a visual object (610) on the display assembly (240) based on detecting an external object (410) released by a hand (415) of a user (405), and can stop displaying the visual object (610).

[0064] In another embodiment, at least one processor (210) can identify whether a reference time has elapsed from the time of detecting the external object (410) released by the hand (415) of the user (405) based on detecting the external object (410) released by the hand (415) of the user (405). The at least one processor (210) can maintain displaying the external object (410) released by the hand (415) of the user (405) until the reference time has elapsed from the time of detecting the external object (410) released by the hand (415) of the user (405), and can stop displaying the visual object (610) based on identifying that the reference time has elapsed from the time of detecting the external object (410) released by the hand (415) of the user (405). When an external object (410) gripped by a hand (415) of a user (405) is released against the user's (405) intention, the external object (410) released by the user's (405) hand (415) is maintained to be displayed until a reference time elapses from the time of detecting the external object (410) released by the user's (405) hand (415), thereby allowing other users to continuously see the visual object (610). For example, when the external object (410) is released against the user's (405) intention, the user (405) may grip the external object (410) again. At least one processor (210) can maintain displaying the external object (410) released by the hand (415) of the user (405) until the user (405) grips the external object (410) again (until a reference time elapses from the time of detecting the external object (410) released by the hand (415) of the user (405)), as the user (405) releases the external object (410) unintentionally.

[0065] In one embodiment, even if an avatar (605) drawing a visual object (610) is displayed, a portion of the visual object (610) shown by the user (405) may be different from a portion of the visual object (610) shown to other users. Since the portion of the visual object (610) shown by the user (405) is different from the portion of the visual object (610) shown to other users, the user (405) may make an error in introducing or explaining the external object (410) to other users. To compensate for such an error, in another embodiment, it may be required to display an avatar (605) drawing a visual object (610) so that a portion of the visual object (610) shown by the user (405) is visible to other users.

[0066] According to one embodiment, the state (600) may be described as a state in which an avatar (605) that has drawn a visual object (610) is displayed based on a first object display mode. At least one processor (210) may display the visual object (610) drawn by the avatar (605) in a first orientation within the first object display mode so that a portion of the visual object (610) corresponding to a portion of the external object (410) facing the user (405) is shown to the user (405). In order for the portion of the visual object (610) shown by the user (405) to be shown to other users, a second object display mode different from the first object display mode may be required. Displaying an avatar that has drawn a visual object based on the second object display mode is exemplified within the description of FIG. 7.

[0067] Figure 7 illustrates an example of displaying an avatar that draws a visual object based on the second object display mode.

[0068] Referring to FIG. 7, within a state (700), a user (405) may grip an external object (410) with a hand (415) of the user (405). The external object (410) may include a portion of the external object (410) facing the user (405). At least one processor (210) may identify a rotation of the external object (410) by identifying a portion of the external object (410) facing the user (405) using at least a portion of the images. The at least one processor (210) may execute a second object display mode based on detecting the external object (410) gripped by the hand (415) of the user (405).

[0069] According to one embodiment, at least one processor (210) may display an avatar (605) representing a user (405) who has drawn a visual object (610) corresponding to an external object (410) on the display assembly (240) based on the second object display mode. Within the second object display mode, a portion (706) of the visual object (610) corresponding to a portion of the external object (410) facing the user (405) may be drawn by the avatar (605) so as to be visible to other users. By drawing the portion (706) of the visual object (610) by the avatar (605) so as to be visible to other users, the visual object (610) may be displayed in a second orientation different from the first orientation. By displaying the visual object (610) in the second orientation, the portion (706) of the visual object (610) corresponding to the portion of the external object (410) facing the user (405) can face the front direction of the avatar (605).

[0070] According to one embodiment, at least one processor (210) may display a portion (615) of the avatar (605) corresponding to a hand of the avatar (605) to grip a visual object (610) displayed in a second orientation. For example, in the second orientation, the visual object (610) may be positioned such that a portion (706) of the visual object (610) (e.g., a front portion of the visual object (610)) faces the front direction of the avatar, and in the first orientation, the visual object (610) may be positioned such that a portion (706) of the visual object (610) faces the opposite direction from the visual object (610) in the second orientation. According to one embodiment, based on the second object display mode, the visual object (610) may change the orientation of the visual object (610) so that a portion (e.g., a portion (706) of the visual object (610)) that the user wants to share with other users is visible. The orientation of the visual object (610) to be changed (or adjusted) based on the second object display mode may be determined by artificial intelligence, or may be determined by a user (405) gesture (or a user request, or a user (405) input). At least one processor (210) may display an avatar (605) that naturally grips a visual object (610) displayed in the second orientation by changing the orientation of a part (615) of the avatar (605) corresponding to the hand of the avatar (605) that grips the visual object (610). At least one processor (210) may further change the orientation of a part of the avatar (605) corresponding to the arm of the avatar (605) that grips the visual object (610) to display an avatar (605) that naturally grips a visual object (610) displayed in the second orientation.

[0071] According to one embodiment, by displaying a portion (706) of a visual object (610) corresponding to a portion of an external object (410) facing the user (405) so that the portion of the external object (410) visible to the user (405) is visible to other users, the portion of the external object (410) visible to the user (405) can be visible to other users. By making the portion of the external object (410) visible to the user (405) visible to other users, the user (405) can introduce or provide the portion of the external object (410) visible to the user (405) to other users.

[0072] In another embodiment, at least one processor (210) can identify a user gesture for executing a second object display mode while displaying an avatar (605) that has drawn a visual object (610) such that a portion (706) of the visual object (610) corresponding to a portion of the external object (410) facing the user (405) is shown to the user (405) based on the first object display mode. Based on the user gesture, the wearable device (200) can switch from the first object execution mode to the second object execution mode. Based on the second object display mode changed from the first object execution mode, the at least one processor (210) can change the orientation of the visual object (610) drawn by the avatar (605) such that a portion of the external object (410) facing the user (405) is shown to another user.

[0073] According to one embodiment, at least one processor (210) may display a user interface (UI) (705) on the display assembly (240) to indicate that the second object display mode is being executed while displaying an avatar (605) that draws a visual object (610) based on the second object display mode. For example, the user interface (705) may be displayed together with the avatar (605) while the second object display mode is being executed. By displaying the user interface (705), the at least one processor (210) may indicate to the user (405) and other users that the second object display mode is being executed. By indicating to the user (405) and other users that the second object display mode is being executed, the user (405) and other users may recognize that the visual object (610) is being displayed in the second orientation.

[0074] Figure 8 illustrates an example of displaying a visual object drawn by an avatar along with the back of the avatar, based on the second object display mode.

[0075] Referring to FIG. 8, within a state (800), a user (405) may grip an external object (410) with a hand (415) of the user (405). The external object (410) may include a portion (806) of the external object (410) facing the user (405). At least one processor (210) may execute a third object display mode based on detecting the external object (410) gripped by the hand (415) of the user (405).

[0076] According to one embodiment, at least one processor (210) may display, on the display assembly (240), an avatar (605) representing a user (405) who has drawn a visual object (610) corresponding to an external object (410) based on a third object display mode. Within the third object display mode, at least one processor (210) may display, on the display assembly (240), an avatar (605) that has drawn a visual object (610) from behind the avatar (605) and from a point of view facing the avatar (605) (or from a point of view photographing the avatar (605) from behind the avatar (605). By displaying an avatar (605) drawn with a visual object (610) from behind the avatar (605) and facing the avatar (605), a portion (807) of the visual object (610) corresponding to a portion (806) of the external object (410) facing the user (405) can be displayed together with the back of the avatar (605) so that it can be seen by other users.

[0077] According to one embodiment, by displaying a portion (807) of a visual object (610) corresponding to a portion (806) of an external object (410) facing the user (405) so that the portion (806) of the external object (410) visible to the user (405) is visible to other users, the portion (806) of the external object (410) visible to the user (405) can be visible to other users. By displaying the portion (806) of the external object (410) visible to the user (405) to other users, the user (405) can introduce or provide the portion (806) of the external object (410) visible to the user (405) to other users.

[0078] According to one embodiment, at least one processor (210) may refrain from displaying (or stop displaying, or not displaying) a portion of a portion (807) of a visual object (610) based on a third object display mode. The portion of the portion (806) of the visual object (610) may be described as a portion of the portion (806) of the external object (410) that is predetermined by the user (405) (or a portion that the user (405) does not want to be shown to other users). The at least one processor (210) may refrain from displaying (or stop displaying, or not displaying) a portion of the portion (806) of the visual object (610) by performing a mosaic (or blur) process on the portion of the portion (807) of the visual object (610) or by changing the orientation of the avatar (605) (or the visual object (610)).

[0079] In another embodiment, at least one processor (210) may identify a user gesture for executing a third object display mode while displaying an avatar (605) that draws a visual object (610) such that a portion (807) of the visual object (610) corresponding to a portion (806) of the external object (410) facing the user (405) is shown to the user (405), based on the first object display mode. Based on the user gesture, the wearable device (200) may switch from the first object execution mode to the third object execution mode.

[0080] According to one embodiment, at least one processor (210) may, based on the third object display mode, display a user interface (UI) (805) on the display assembly (240) while displaying an avatar (605) that has drawn a visual object (610) to indicate that the third object display mode is being executed. For example, the user interface (805) may be displayed together with the avatar (605) while the third object display mode is being executed. By displaying the user interface (805), the at least one processor (210) may indicate to the user (405) and other users that the second object display mode is being executed. By indicating to the user (405) and other users that the third object display mode is being executed, the user (405) and other users may recognize that the avatar (605) that has drawn a visual object (610) is being displayed from behind the avatar (605) and from a viewpoint facing the avatar (605).

[0081] Figure 9 illustrates examples of components of a wearable device.

[0082] Referring to FIG. 9, according to one embodiment, a wearable device (200) may include a camera unit (900), an input information processing unit (910), a hand tracking processing unit (920), a static object detection unit (930), a gripped object selection unit (940), a punch-through control unit (950), a storage data processing unit (960), an avatar rendering unit (970), an AR rendering unit (980), and / or a display unit (990). The camera unit (900), the input information processing unit (910), the hand tracking processing unit (920), the static object detection unit (930), the gripped object selection unit (940), the punch-through control unit (950), the storage data processing unit (960), the avatar rendering unit (970), the AR rendering unit (980), and / or the display unit (990) may support the function of processing avatars and visual objects through an algorithm stored in the memory (220). Although the camera unit (900), the input information processing unit (910), the hand tracking processing unit (920), the static object detection unit (930), the gripped object selection unit (940), the punch-through control unit (950), the storage data processing unit (960), the avatar rendering unit (970), the AR rendering unit (980), and / or the display unit (990) are described as 'units', they may perform the following functions in software and / or functionally.

[0083] According to one embodiment, the camera unit (900) can identify the position of the user's body using at least some of the images acquired using one or more cameras (230). The camera unit (900) can identify the movement of the user based on the movement of the user's body position. At least one processor (210) can determine the movement of an avatar representing the user based on the movement of the user identified using the camera unit (900). At least one processor (210) can identify a user gesture based on the movement of the user identified using the camera unit (900). The camera unit (900) can identify the user's surrounding environment using at least some of the images acquired using one or more cameras (230). At least one processor (210) can provide information for the user's surrounding environment to interact with the avatar based on the user's surrounding environment identified using the camera unit (900). The camera unit (900) can identify the user's location within the wearable device (200) by tracking the user's location in real time.

[0084] According to one embodiment, the wearable device (200) may further include one or more cameras arranged facing the user's face. The camera unit (900) may identify changes in the user's face using at least some of the images acquired using the one or more cameras arranged facing the user's face. At least one processor (210) may identify the user's facial expression based on the changes in the user's face identified using the camera unit (900). At least one processor (210) may determine the avatar's facial expression based on the user's facial expression.

[0085] According to one embodiment, the input information processing unit (910) may collect or process data about the user and the user's surroundings in real time using images acquired using one or more cameras (230). The input information processing unit (910) may remove noise from the images or correct the resolution and color of the images. The input information processing unit (910) may identify features of the user's face using images acquired using one or more cameras arranged toward the user's face. The input information processing unit (910) may identify the user's facial expression and gesture based on the features of the user's face. The input information processing unit (910) may extract features related to the user's body (e.g., the user's eyes, arms, body, and / or pupils) using an artificial intelligence model and / or a machine learning model. The input information processing unit (910) may predict or analyze the user's next action based on the user's movement identified using the camera unit (900). The input information processing unit (910) can increase the recognition rate of the user's facial expression and user gesture by using an artificial intelligence model and / or a machine learning model, and can predict the user's next action by analyzing the user's continuous movements.

[0086] According to one embodiment, the hand tracking processing unit (920) can identify feature points of a user's hand using images acquired using one or more cameras (230). The hand tracking processing unit (920) can track the position of the user's hand in real time based on the feature points of the user's hand. The hand tracking processing unit (920) can identify the coordinates of the user's hand in a three-dimensional (3D) space by processing the position of the user's hand in real time. At least one processor (210) can identify the movement of the user's hand and the user gesture using the user's hand based on the position of the hand identified through the hand tracking processing unit (920). The hand tracking processing unit (920) can transmit the movement of the user's hand and the user gesture using the user's hand in real time to at least one processor (210) of the wearable device (200), thereby reflecting the movement of the user's hand and the user gesture using the user's hand in interactions with other users through an avatar. At least one processor (210) can identify an external object gripped by a user using a hand tracking processing unit (920).

[0087] According to one embodiment, the static object detection unit (930) can identify a static object that is distinct from the background using images acquired using one or more cameras (230). The static object detection unit (930) can identify the type of the static object by comparing the identified static object with objects (e.g., a desk, a chair, and / or a wall) stored in a database. The static object detection unit (930) can classify the identified static object by the type of objects stored in the database. The static object detection unit (930) can identify the location and size of the static object using images acquired using one or more cameras (230). The static object detection unit (930) can identify the size of the static object by comparing the size of the static object with the size in an actual environment in images acquired using one or more cameras (230).

[0088] According to one embodiment, the gripped object selection unit (940) can identify the position of the user's hand and the position of the external object using images acquired using one or more cameras (230). The gripped object selection unit (940) can identify the external object gripped by the user's hand based on the position of the external object that moves according to the movement of the position of the user's hand. For example, in operation 320 of FIG. 3, at least one processor (210) can use the gripped object selection unit (940) to detect the external object gripped by the user's hand. The gripped object selection unit (940) can determine the external object in contact with the user's hand as the object gripped by the user's hand by identifying whether the user's hand and the external object are in contact. The gripped object selection unit (940) can relatively accurately detect (or identify) the object gripped by the user's hand using (or integrating) data collected from a plurality of cameras.

[0089] According to one embodiment, the punch-through control unit (950) can display a visual object corresponding to an external object gripped by a user's hand on the display assembly (240) by distinguishing the external object from the background. By displaying the visual object without displaying the background, the visual object can be emphasized and shown to the user.

[0090] According to one embodiment, the storage data processing unit (960) can store and process avatar data and animation data. The avatar data can include data about the appearance of the avatar, data about the attributes of the avatar, data about the clothing of the avatar, 3D (three-dimensional) model data of the avatar, texture data of the avatar, and / or color data of the avatar. The animation data can include sequence data of a predetermined animation within the wearable device (200), motion capture data, action data of the character, and / or facial expression data of the character. The storage data processing unit (960) can display the avatar on the display assembly (240) by applying the avatar data and animation data to an avatar representing the user. The storage data processing unit (960) can receive an input for changing or selecting the animation of the avatar through a user interface for controlling the avatar. The storage data processing unit (960) can change the avatar and store or load the changed avatar based on a user input through the user interface.

[0091] According to one embodiment, the avatar rendering unit (970) can render an avatar within a virtual environment based on information obtained from an avatar database. The avatar rendering unit (970) can display the avatar in real time and control the avatar's movements based on the user's movements. The avatar rendering unit (970) can control the avatar's facial expressions and / or movements using a rendering engine based on avatar data and user data.

[0092] According to one embodiment, the AR rendering unit (980) can render an external object by transmitting information about an external object gripped by a user's hand to the AR rendering engine. The AR rendering unit (980) can provide an augmented reality space to the user by displaying the rendered external object within a virtual environment. The AR rendering unit (980) can increase the size of a visual object corresponding to the external object by performing upsampling, or reduce the size of a visual object by performing downsampling on the visual object. The AR rendering unit (980) can render a visual object having an increased size based on the size of the visual object increased by performing upsampling, or can render a visual object having a reduced size based on the size of the visual object reduced by performing downsampling.

[0093] According to one embodiment, the display unit (990) may include a display assembly (240). The display unit (990) may display visual objects rendered by the avatar rendering unit (970) and the AR rendering unit (980) on the display assembly (240). The display unit (990) may display information indicating a user's status, guidance indicating available actions, and / or notification messages.

[0094] FIG. 10 is a block diagram of an electronic device within a network environment according to various embodiments.

[0095] Referring to FIG. 10, in a network environment (1000), an electronic device (1001) may communicate with an electronic device (1002) via a first network (1098) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1004) or a server (1008) via a second network (1099) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1001) may communicate with the electronic device (1004) via the server (1008). According to one embodiment, the electronic device (1001) may include a processor (1020), a memory (1030), an input module (1050), an audio output module (1055), a display module (1060), an audio module (1070), a sensor module (1076), an interface (1077), a connection terminal (1078), a haptic module (1079), a camera module (1080), a power management module (1088), a battery (1089), a communication module (1090), a subscriber identification module (1096), or an antenna module (1097). In some embodiments, the electronic device (1001) may omit at least one of these components (e.g., the connection terminal (1078)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1076), camera module (1080), or antenna module (1097)) may be integrated into a single component (e.g., display module (1060)).

[0096] The processor (1020) may, for example, execute software (e.g., a program (1040)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1001) connected to the processor (1020) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1020) may store commands or data received from other components (e.g., a sensor module (1076) or a communication module (1090)) in a volatile memory (1032), process the commands or data stored in the volatile memory (1032), and store result data in a non-volatile memory (1034). According to one embodiment, the processor (1020) may include a main processor (1021) (e.g., a central processing unit or an application processor) or an auxiliary processor (1023) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1021). For example, when the electronic device (1001) includes the main processor (1021) and the auxiliary processor (1023), the auxiliary processor (1023) may be configured to use less power than the main processor (1021) or to be specialized for a given function. The auxiliary processor (1023) may be implemented separately from the main processor (1021) or as a part thereof.

[0097] The auxiliary processor (1023) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1060), the sensor module (1076), or the communication module (1090)) of the electronic device (1001), for example, on behalf of the main processor (1021) while the main processor (1021) is in an inactive (e.g., sleep) state, or together with the main processor (1021) while the main processor (1021) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1023) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1080) or a communication module (1090)). In one embodiment, the auxiliary processor (1023) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1001) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1008)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0098] The memory (1030) can store various data used by at least one component (e.g., the processor (1020) or the sensor module (1076)) of the electronic device (1001). The data can include, for example, software (e.g., the program (1040)) and input data or output data for commands related thereto. The memory (1030) can include volatile memory (1032) or non-volatile memory (1034).

[0099] The program (1040) may be stored as software in memory (1030) and may include, for example, an operating system (1042), middleware (1044), or an application (1046).

[0100] The input module (1050) can receive commands or data to be used in a component of the electronic device (1001) (e.g., a processor (1020)) from an external source (e.g., a user) of the electronic device (1001). The input module (1050) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0101] The audio output module (1055) can output audio signals to the outside of the electronic device (1001). The audio output module (1055) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0102] The display module (1060) can visually provide information to an external party (e.g., a user) of the electronic device (1001). The display module (1060) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1060) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0103] The audio module (1070) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (1070) can acquire sound through the input module (1050), output sound through the sound output module (1055), or an external electronic device (e.g., electronic device (1002)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1001).

[0104] The sensor module (1076) can detect the operating status (e.g., power or temperature) of the electronic device (1001) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1076) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0105] The interface (1077) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1001) to an external electronic device (e.g., the electronic device (1002)). In one embodiment, the interface (1077) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0106] The connection terminal (1078) may include a connector through which the electronic device (1001) may be physically connected to an external electronic device (e.g., the electronic device (1002)). In one embodiment, the connection terminal (1078) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0107] The haptic module (1079) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (1079) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0108] The camera module (1080) can capture still images and videos. In one embodiment, the camera module (1080) may include one or more lenses, image sensors, image signal processors, or flashes.

[0109] The power management module (1088) can manage power supplied to the electronic device (1001). According to one embodiment, the power management module (1088) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0110] A battery (1089) may power at least one component of the electronic device (1001). In one embodiment, the battery (1089) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0111] The communication module (1090) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1001) and an external electronic device (e.g., electronic device (1002), electronic device (1004), or server (1008)), and the performance of communication through the established communication channel. The communication module (1090) may operate independently from the processor (1020) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1090) may include a wireless communication module (1092) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1094) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module can communicate with an external electronic device (1004) via a first network (1098) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1099) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1092) can verify or authenticate the electronic device (1001) within a communication network such as the first network (1098) or the second network (1099) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1096).

[0112] The wireless communication module (1092) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1092) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1092) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1092) may support various requirements specified in the electronic device (1001), an external electronic device (e.g., the electronic device (1004)), or a network system (e.g., the second network (1099)). According to one embodiment, the wireless communication module (1092) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 104 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0113] The antenna module (1097) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1097) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1097) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1098) or the second network (1099), may be selected from the plurality of antennas by, for example, the communication module (1090). A signal or power may be transmitted or received between the communication module (1090) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1097).

[0114] According to various embodiments, the antenna module (1097) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0115] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0116] According to one embodiment, commands or data may be transmitted or received between the electronic device (1001) and an external electronic device (1004) via a server (1008) connected to a second network (1099). Each of the external electronic devices (1002 or 1004) may be the same or a different type of device as the electronic device (1001). According to one embodiment, all or part of the operations executed in the electronic device (1001) may be executed in one or more of the external electronic devices (1002, 1004, or 1008). For example, when the electronic device (1001) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1001) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1001). The electronic device (1001) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1001) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (1004) may include an Internet of Things (IoT) device. The server (1008) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1004) or server (1008) may be included within the second network (1099). The electronic device (1001) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0117] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0118] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0119] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0120] Various embodiments of the present document may be implemented as software (e.g., a program (1040)) including one or more instructions stored in a storage medium (e.g., an internal memory (1036) or an external memory (1038)) readable by a machine (e.g., an electronic device (1001)). For example, a processor (e.g., a processor (1020)) of the machine (e.g., an electronic device (1001)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0121] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0122] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0123] Figure 11a shows an example of a perspective view of a wearable device.

[0124] FIG. 11b illustrates an example of one or more hardware devices arranged within a wearable device.

[0125] FIG. 11A illustrates an example of a perspective view of a wearable device. FIG. 11B illustrates an example of one or more hardware components arranged within the wearable device. According to one embodiment, the wearable device (200) may have a form of glasses that can be worn on a body part (e.g., head) of a user. The wearable device (200) of FIGS. 11A and 11B may be an example of the wearable device (200) of FIG. 2. The wearable device (200) may include a head-mounted display (HMD). For example, the housing of the wearable device (200) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (200) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0126] Referring to FIG. 11A, according to one embodiment, a wearable device (200) may include at least one display (1150) and a frame (1100) supporting at least one display (1150).

[0127] According to one embodiment, the wearable device (200) can be worn on a part of a user's body. The wearable device (200) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (200). For example, the wearable device (200) can display a virtual reality image provided from at least one optical device (1182, 1184) of FIG. 11B on at least one display (1150) in response to a user's designated gesture acquired through the motion recognition cameras (1160-2, 1160-3) of FIG. 11B.

[0128] According to one embodiment, at least one display (1150) may provide visual information to a user. For example, at least one display (1150) may include a transparent or translucent lens. At least one display (1150) may include a first display (1150-1) and / or a second display (1150-2) spaced apart from the first display (1150-1). For example, the first display (1150-1) and the second display (1150-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0129] Referring to FIG. 11B, at least one display (1150) can provide the user with visual information transmitted from external light and other visual information distinct from the visual information through a lens included in the at least one display (1150). The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (1150) can include a first surface (1131) and a second surface (1132) opposite the first surface (1131). A display area can be formed on the second surface (1132) of the at least one display (1150). When the user wears the wearable device (200), external light can be transmitted to the user by being incident on the first surface (1131) and transmitted through the second surface (1132). As another example, at least one display (1150) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1182, 1184) on a real screen transmitted through external light, in a display area formed on the second surface (1132).

[0130] In one embodiment, at least one display (1150) may include at least one waveguide (1133, 1134) that diffracts light emitted from at least one optical device (1182, 1184) and transmits the diffracted light to a user. The at least one waveguide (1133, 1134) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1133, 1134). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1133, 1134) may be propagated to the other end of the at least one waveguide (1133, 1134) by the nano-pattern. At least one waveguide (1133, 1134) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1133, 1134) may be arranged within the wearable device (200) to guide a screen displayed by at least one display (1150) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1133, 1134).

[0131] The wearable device (200) can analyze an object included in a real image collected through a shooting camera (1160-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1150). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (200) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (200) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (200) can view an image displayed on at least one display (1150).

[0132] According to one embodiment, the frame (1100) may be configured as a physical structure that allows the wearable device (200) to be worn on the user's body. According to one embodiment, the frame (1100) may be configured so that, when the user wears the wearable device (200), the first display (1150-1) and the second display (1150-2) can be positioned corresponding to the user's left and right eyes. The frame (1100) may support at least one display (1150). For example, the frame (1100) may support the first display (1150-1) and the second display (1150-2) to be positioned corresponding to the user's left and right eyes.

[0133] Referring to FIG. 11A, the frame (1100) may include a region (1120) that at least partially contacts a part of the user's body when the user wears the wearable device (200). For example, the region (1120) of the frame (1100) that contacts a part of the user's body may include a region that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (200) makes contact with. According to one embodiment, the frame (1100) may include a nose pad (1110) that contacts a part of the user's body. When the wearable device (200) is worn by the user, the nose pad (1110) may contact a part of the user's nose. The frame (1100) may include a first temple (1104) and a second temple (1105) that contact another part of the user's body that is distinct from the part of the user's body.

[0134] For example, the frame (1100) may include a first rim (1101) that surrounds at least a portion of the first display (1150-1), a second rim (1102) that surrounds at least a portion of the second display (1150-2), a bridge (1103) that is disposed between the first rim (1101) and the second rim (1102), a first pad (1111) that is disposed along a portion of the edge of the first rim (1101) from one end of the bridge (1103), a second pad (1112) that is disposed along a portion of the edge of the second rim (1102) from the other end of the bridge (1103), a first temple (1104) that extends from the first rim (1101) and is fixed to a portion of the wearer's ear, and a second temple (1105) that extends from the second rim (1102) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1111) and the second pad (1112) can be in contact with a part of the user's nose, and the first temple (1104) and the second temple (1105) can be in contact with a part of the user's face and a part of the user's ear. The temples (1104, 1105) can be rotatably connected to the rim through the hinge units (1106, 1107) of FIG. 11B. The first temple (1104) can be rotatably connected to the first rim (1101) through the first hinge unit (1106) disposed between the first rim (1101) and the first temple (1104). The second temple (1105) can be rotatably connected to the second rim (1102) via a second hinge unit (1107) disposed between the second rim (1102) and the second temple (1105). In one embodiment, the wearable device (200) can identify an external object (e.g., a user's fingertip) touching the frame (1100) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1100).

[0135] According to one embodiment, the wearable device (200) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 13). For example, the hardwares may include a battery module (1170), an antenna module (1175), at least one optical device (1182, 1184), speakers (e.g., speakers 1155-1, 1155-2), a microphone (e.g., microphones 1165-1, 1165-2, 1165-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (1190) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (1100).

[0136] According to one embodiment, microphones (e.g., microphones 1165-1, 1165-2, 1165-3) of the wearable device (200) may be disposed on at least a portion of the frame (1100) to acquire sound signals. A first microphone (1165-1) disposed on the bridge (1103), a second microphone (1165-2) disposed on the second rim (1102), and a third microphone (1165-3) disposed on the first rim (1101) are illustrated in FIG. 11B , but the number and arrangement of the microphones (1165) are not limited to the embodiment of FIG. 11B . When the number of microphones (1165) included in the wearable device (200) is two or more, the wearable device (200) can identify the direction of a sound signal by using a plurality of microphones placed on different parts of the frame (1100).

[0137] According to one embodiment, at least one optical device (1182, 1184) may project a virtual object onto at least one display (1150) to provide various image information to a user. For example, at least one optical device (1182, 1184) may be a projector. At least one optical device (1182, 1184) may be disposed adjacent to at least one display (1150) or may be included within at least one display (1150) as a part of at least one display (1150). According to one embodiment, the wearable device (200) may include a first optical device (1182) corresponding to a first display (1150-1) and a second optical device (1184) corresponding to a second display (1150-2). For example, at least one optical device (1182, 1184) may include a first optical device (1182) disposed at an edge of a first display (1150-1) and a second optical device (1184) disposed at an edge of a second display (1150-2). The first optical device (1182) may transmit light to a first waveguide (1133) disposed on the first display (1150-1), and the second optical device (1184) may transmit light to a second waveguide (1134) disposed on the second display (1150-2).

[0138] In one embodiment, the camera (1160) may include a recording camera (1160-4), an eye tracking camera (ET CAM) (1160-1), and / or a motion recognition camera (1160-2, 1160-3). The recording camera (1160-4), the eye tracking camera (1160-1), and the motion recognition cameras (1160-2, 1160-3) may be positioned at different locations on the frame (1100) and may perform different functions. The eye tracking camera (1160-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (200). For example, the wearable device (200) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1160-1). The wearable device (200) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1160-1). The wearable device (200) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (200) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1160-1). The wearable device (200) can render an image (or screen) displayed on at least one display (1150) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other.The wearable device (200) can obtain an image having a visual quality of a first area matching the user's gaze and a visual quality of a second area using foveated rendering. For example, if the wearable device (200) supports an iris recognition function, user authentication can be performed based on iris information obtained using a gaze tracking camera (1160-1). An example in which the gaze tracking camera (1160-1) is positioned toward the user's right eye is illustrated in FIG. 11B, but the embodiment is not limited thereto, and the gaze tracking camera (1160-1) can be positioned solely toward the user's left eye, or toward both eyes.

[0139] In one embodiment, the capturing camera (1160-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1160-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1160-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1150). The at least one display (1150) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1160-4) and a virtual image provided through at least one optical device (1182, 1184) are superimposed. The wearable device (200) can compensate for depth information (e.g., the distance between the wearable device (200) and an external object acquired through a depth sensor) using an image acquired through the capturing camera (1160-4). The wearable device (200) can perform object recognition using an image acquired using the capturing camera (1160-4). The wearable device (200) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (1160-4). The wearable device (200) can perform a pass-through function to display an image acquired through the capturing camera (1160-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1150) while displaying the screen. In one embodiment, the shooting camera (1160-4) may be positioned on a bridge (1103) disposed between the first rim (1101) and the second rim (1102).

[0140] The gaze tracking camera (1160-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (200) and matching the user's gaze with visual information provided to at least one display (1150). For example, when the wearable device (200) looks straight ahead, the wearable device (200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1150). The gaze tracking camera (1160-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1160-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1160-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1160-1) may be positioned within the first rim (1101) and / or the second rim (1102) to face the direction in which the user wearing the wearable device (200) is positioned.

[0141] The gesture recognition camera (1160-2, 1160-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (1150). The gesture recognition camera (1160-2, 1160-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1150). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (1160-2, 1160-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1160-2, 1160-3). In one embodiment, the motion recognition cameras (1160-2, 1160-3) may be positioned on the first rim (1101) and / or the second rim (1102).

[0142] The camera (1160) included in the wearable device (200) is not limited to the above-described gaze tracking camera (1160-1) and motion recognition cameras (1160-2, 1160-3). For example, the wearable device (200) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (200) can identify an external object based on a sensor for identifying the distance between the wearable device (200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1160) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (200) may include a camera (1160) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (200).

[0143] Although not shown, in one embodiment, the wearable device (200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (1160). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (1100) and the hinge units (1106, 1107).

[0144] According to one embodiment, the battery module (1170) may supply power to electronic components of the wearable device (200). In one embodiment, the battery module (1170) may be disposed within the first temple (1104) and / or the second temple (1105). For example, the battery module (1170) may be a plurality of battery modules (1170). The plurality of battery modules (1170) may be disposed within each of the first temple (1104) and the second temple (1105). In one embodiment, the battery module (1170) may be disposed at an end of the first temple (1104) and / or the second temple (1105).

[0145] The antenna module (1175) can transmit signals or power to the outside of the wearable device (200), or receive signals or power from the outside. In one embodiment, the antenna module (1175) can be positioned within the first temple (1104) and / or the second temple (1105). For example, the antenna module (1175) can be positioned close to one surface of the first temple (1104) and / or the second temple (1105).

[0146] The speaker (1155) can output an audio signal to the outside of the wearable device (200). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1155) may be positioned within the first temple (1104) and / or the second temple (1105) so as to be positioned adjacent to the ear of a user wearing the wearable device (200). For example, the speaker (1155) may include a second speaker (1155-2) positioned within the first temple (1104) and thus adjacent to the user's left ear, and a first speaker (1155-1) positioned within the second temple (1105) and thus adjacent to the user's right ear.

[0147] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (200) to the user. For example, when the wearable device (200) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1101) and / or the second rim (1102).

[0148] Referring to FIG. 11B, according to one embodiment, a wearable device (200) may include a printed circuit board (PCB) (1190). The PCB (1190) may be included in at least one of the first temple (1104) or the second temple (1105). The PCB (1190) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (200) (e.g., hardwares illustrated by different blocks in FIG. 4) may be disposed on the PCB (1190). The wearable device (200) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0149] According to one embodiment, a wearable device (200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (200) based on the IMU.

[0150] Figures 12a and 12b show an example of the appearance of a wearable device.

[0151] FIGS. 12A and 12B illustrate an example of an exterior appearance of a wearable device (e.g., a wearable device (200)). The wearable device (200) of FIGS. 12A and 12B may be an example of the wearable device (200) of FIG. 1. According to one embodiment, an example of an exterior appearance of a first side (1210) of a housing of a wearable device (200) may be illustrated in FIG. 12A, and an example of an exterior appearance of a second side (1220) opposite to the first side (1210) may be illustrated in FIG. 12B.

[0152] Referring to FIG. 12A, according to one embodiment, a first surface (1210) of a wearable device (200) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (200) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1104) and / or the second temple (1105) of FIGS. 11A and 11B). A first display (1150-1) for outputting an image to a left eye among the user's two eyes, and a second display (1150-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1210). The wearable device (200) is formed on the first surface (1210) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (1150-1) and the second display (1150-2).

[0153] According to one embodiment, the wearable device (200) may include cameras (1160-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (1150-1) and the second display (1150-2). The cameras (1160-1) may be referred to as the gaze tracking camera (1160-1) of FIG. 11B. According to one embodiment, the wearable device (200) may include cameras (1160-5, 1160-6) for photographing and / or recognizing the face of the user. The cameras (1160-5, 1160-6) may be referred to as FT cameras. The wearable device (200) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1160-5, 1160-6). For example, the wearable device (200) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (1160-5, 1160-6) (e.g., FT cameras) and representing a facial expression of a user wearing the wearable device (200).

[0154] Referring to FIG. 12B, a camera (e.g., cameras (1160-7, 1160-8, 1160-9, 1160-10, 1160-11, 1160-12)) and / or a sensor (e.g., a depth sensor (1230)) for obtaining information related to the external environment of the wearable device (200) may be disposed on a second surface (1220) opposite to the first surface (1210) of FIG. 12A. For example, the cameras (1160-7, 1160-8, 1160-9, 1160-10) may be disposed on the second surface (1220) for recognizing external objects. Cameras (1160-7, 1160-8, 1160-9, 1160-10) may be referenced to the motion recognition cameras (1160-2, 1160-3) of FIG. 11b.

[0155] For example, using cameras (1160-11, 1160-12), the wearable device (200) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1160-11) can be placed on the second face (1220) of the wearable device (200) to obtain an image to be displayed through the second display (1150-2) corresponding to the right eye among the two eyes. The camera (1160-12) can be placed on the second face (1220) of the wearable device (200) to obtain an image to be displayed through the first display (1150-1) corresponding to the left eye among the two eyes. The cameras (1160-11, 1160-12) can be referred to as the shooting camera (1160-4) of FIG. 11B.

[0156] According to one embodiment, the wearable device (200) may include a depth sensor (1230) disposed on the second face (1220) to identify a distance between the wearable device (200) and an external object. Using the depth sensor (1230), the wearable device (200) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (200). Although not illustrated, a microphone may be disposed on the second face (1220) of the wearable device (200) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0157] Hereinafter, with reference to FIG. 13, the hardware or software configuration of the wearable device (200) is described.

[0158] Figure 13 shows an example of a block diagram of a wearable device.

[0159] Fig. 13 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (200)). The wearable device (200) of Fig. 13 may be an example of the wearable device (200) of Fig. 2 and the wearable devices (200) of Figs. 11a to 12b.

[0160] Referring to FIG. 13, a wearable device (200) according to one embodiment may include a processor (1310), a memory (1315), a display (1150) (e.g., the first display (1150-1) and / or the second display (1150-2) of FIGS. 11A, 11B, 12A, and 12B), and / or a sensor (1320). The processor (1310), the memory (1315), the display (1150), and / or the sensor (1320) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (1302). In the present disclosure, the operative connection of the electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (200) is not limited to those illustrated in FIG. 13. For example, the wearable device (200) may include only some of the electronic components illustrated in FIG. 13.

[0161] According to one embodiment, a processor (1310) of a wearable device (200) may include a circuit (e.g., a processing circuit) for processing data based on one or more instructions. The circuit for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (200) may include one or more processors. The processor (1310) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (1310) may include a structure (e.g., a big-little structure) based on a plurality of core circuits that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (1310) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (1310).

[0162] A memory (1315) of a wearable device (200) according to one embodiment may include electronic components for storing data and / or instructions input to and / or output from a processor (1310). The memory (1315) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (eMMC). In one embodiment, memory (1315) may be referred to as storage.

[0163] In one embodiment, a display (1150) of a wearable device (200) can output visualized information to a user of the wearable device (200). The display (1150), which is arranged in front of the eyes of a user wearing the wearable device (200), can be arranged on at least a portion of a housing of the wearable device (200) (e.g., the first display (1150-1) and / or the second display (1150-2) of FIGS. 11A, 11B, 12A, and 12B). For example, the display (1150) can be controlled by a processor (1310) including circuits such as a CPU, a GPU (graphics processing unit), and / or a DPU (display processing unit), to output visualized information to the user. The display (1150) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (1150) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, if the wearable device (200) includes a lens for transmitting external light (or ambient light), the display (1150) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (1150) may be referred to as a display panel and / or a display module. The pixels included in the display (1150) may be arranged to face either of the user's eyes when the wearable device (200) is worn by the user.For example, the display (1150) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0164] In one embodiment, the sensor (1320) of the wearable device (200) may generate electrical information that may be processed by the processor (1310) and / or the memory (1315) from non-electronic information related to the wearable device (200). For example, the sensor (1320) may include a global positioning system (GPS) sensor for detecting the geographic location of the wearable device (200). In addition to the GPS method, the sensor (1320) may generate information indicating the geographic location of the wearable device (200) based on a global navigation satellite system (GNSS) such as, for example, Galileo or Beidou (compass). The above information may be stored in memory (1315), processed by processor (1310), and / or transmitted to another electronic device distinct from the wearable device (200) via communication circuitry.

[0165] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (1310) of the wearable device (200) may be stored in the memory (1315) of the wearable device (200). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (200) and / or the processor (1310) may perform at least one of the operations of FIGS. 5 to 15 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (200) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (1315), and that the one or more applications are stored in a format executable by the processor (1310) (e.g., a file having an extension specified by the operating system of the wearable device (200)). As an example, the application may include a program and / or a library related to a service provided to a user.

[0166] Referring to FIG. 13, programs installed in the wearable device (200) may be included in any one of different layers, including the application layer (1340), the framework layer (1350), and / or the hardware abstraction layer (HAL) (1380), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (1150), and / or the sensor (1320)) of the wearable device (200) may be included in the hardware abstraction layer (1380). The framework layer (1350) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 13 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (1315) is separated by the layers.

[0167] For example, within the framework layer (1350), programs designed to target at least one of the hardware abstraction layer (1380) and / or the application layer (1340) (e.g., a position tracker (1371), a space recognizer (1372), a gesture tracker (1373), an eye-gaze tracker (1374), and / or a face tracker (1375)) may be included. The programs included in the framework layer (1350) may provide an application programming interface (API) that is executable (or callable) based on other programs.

[0168] For example, a program designed to target users of a wearable device (200) may be included within the application layer (1340). As an example of programs included in the application layer (1340), an extended reality (XR) system user interface (UI) (1341) and / or an XR application (1342) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (1340) may call an API to cause execution of functions supported by programs included in the framework layer (1350).

[0169] For example, the wearable device (200) may display one or more visual objects on the display (1150) for performing interaction with the user based on the execution of the XR system UI (1341). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (200) may provide the user with functions available within a virtual space based on the execution of the XR system UI (1341).

[0170] Referring to FIG. 13, a lightweight renderer (1343) and / or an XR plug-in (1344) are illustrated to be included within the XR system UI (1341), but are not limited thereto. For example, based on the XR system UI (1341), the processor (1310) may execute a lightweight renderer (1343) and / or an XR plug-in (1344) within the framework layer (1350).

[0171] For example, the wearable device (200) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (1343). The lightweight renderer (1343) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (1343) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (200) may acquire resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (1344). The XR plugin (1344) can be referred to as an open XR native client from the perspective of defining (or configuring) the entire rendering pipeline.

[0172] For example, the wearable device (200) may display a screen representing at least a portion of a virtual space on the display (1150) based on the execution of the XR application (1342). The XR plug-in (1344-1) included in the XR application (1342) may include instructions that support functions similar to those of the XR plug-in (1344) of the XR system UI (1341). Descriptions of the XR plug-in (1344-1) that overlap with those of the XR plug-in (1344) may be omitted. The wearable device (200) may cause the execution of the virtual space manager (1351) based on the execution of the XR application (1342).

[0173] For example, the wearable device (200) may display an image on the display (1150) in a virtual space based on the execution of the application (1345). The application (1345) may be configured to output image information for displaying a two-dimensional image. The wearable device (200) may cause the execution of the virtual space manager (1351) based on the execution of the application (1345). The wearable device (200) may generate dual image information to display the two-dimensional image in a three-dimensional virtual space based on the execution of the application (1345). Here, the dual image information may include first image information for the left eye and second image information for the right eye in consideration of binocular disparity. In order to display the two-dimensional image in the three-dimensional virtual space, the wearable device (200) may generate the dual image information based on the image information for displaying the two-dimensional image.

[0174] According to one embodiment, the wearable device (200) may provide a virtual space service based on the execution of the virtual space manager (1351). For example, the virtual space manager (1351) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (1351), the wearable device (200) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (1330), and may display at least a portion of the virtual space on the display (1150). The virtual space manager (1351) may be referred to as a composition presentation manager (CPM).

[0175] For example, the virtual space manager (1351) may include a runtime service (1352). As an example, the runtime service (1352) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (200) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1352). As an example, the wearable device (200) may perform rendering for a virtual space service to the user based on the execution of the runtime service (1352). For example, a function related to a virtual space, executable by the application layer (1340), may be supported based on the execution of the runtime service (1352).

[0176] For example, the virtual space manager (1351) may include a pass-through manager (1353). Based on the execution of the pass-through manager (1353), the wearable device (200) may display an image and / or video representing an actual space acquired through an external camera on at least a portion of the screen while displaying a screen representing a virtual space on the display (1150).

[0177] For example, the virtual space manager (1351) may include an input manager (1354). The wearable device (200) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (1370) based on the execution of the input manager (1354). The wearable device (200) may use the acquired data to identify user input related to the wearable device (200). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by a sensor (1320) (e.g., an image sensor (1330) such as an external camera). The user input may be identified based on an external electronic device connected (or paired) via a communication circuit.

[0178] For example, the perception abstract layer (1360) can be used for data exchange between the virtual space manager (1351) and the perception service layer (1370). From the perspective of being used for data exchange between the virtual space manager (1351) and the perception service layer (1370), the perception abstract layer (1360) can be referred to as an interface. For example, the perception abstract layer (1360) can be referenced as OpenPX. The perception abstract layer (1360) can be used for a perception client and a perception service.

[0179] According to one embodiment, the recognition service layer (1370) may include one or more programs for processing data acquired from the sensor (1320). The one or more programs may include at least one of a position tracker (1371), a space recognizer (1372), a gesture tracker (1373), and / or an eye tracker (1374). The type and / or number of the one or more programs included in the recognition service layer (1370) are not limited to those illustrated in FIG. 13.

[0180] For example, the wearable device (200) can identify the pose of the wearable device (200) using the sensor (1330) based on the execution of the position tracker (1371). The wearable device (200) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (200) using data acquired using an external camera (e.g., an image sensor (1321)) and / or an IMU (e.g., a motion sensor (1322) including a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (1371). The position tracker (1371) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0181] For example, the wearable device (200) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (200) (or the user of the wearable device (200)) based on the execution of the space recognizer (1372). The wearable device (200) may reproduce the surrounding environment of the wearable device (200) in three dimensions using data obtained using an external camera (e.g., an image sensor (1321)) based on the execution of the space recognizer (1372). The wearable device (200) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (200) reproduced in three dimensions based on the execution of the space recognizer (1372). The space recognizer (1372) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0182] For example, the wearable device (200) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (200) based on the execution of the gesture tracker (1373). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera (e.g., an image sensor (1321)) based on the execution of the gesture tracker (1373). As an example, the wearable device (200) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (1373). The gesture tracker (1373) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0183] For example, the wearable device (200) may identify (or track) eye movements of a user of the wearable device (200) based on the execution of the gaze tracker (1374). As an example, the wearable device (200) may identify eye movements of the user using data acquired from a gaze tracking camera (e.g., an image sensor (1321)) based on the execution of the gaze tracker (1374). The gaze tracker (1374) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0184] For example, the recognition service layer (1370) of the wearable device (200) may further include a face tracker (1375) for tracking the user's face. For example, the wearable device (200) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (1375). The wearable device (200) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (1375). As an example, the wearable device (200) may identify the movement of the user's face and / or the user's expression based on data (e.g., images and / or videos) acquired using a camera (1325) (e.g., a camera facing at least a portion of the user's face) based on the execution of the face tracker (1375).

[0185] Referring to FIG. 13, the renderer (1390) may include instructions for rendering images in a three-dimensional virtual space. The processor (1310) executing the renderer (1390) may obtain at least one image to be at least partially displayed in the display area of ​​the display (1150) in a software application. For example, the processor (1310) executing the renderer (1390) may determine the location of the area where an application (e.g., XR application (1342), application (1345)) is to be rendered. The processor (1310) executing the renderer (1390) may generate an image of the application to be displayed on the display (1150). The renderer (1390) may synthesize images to generate a composite image to be displayed on the display (1150).

[0186] For example, the processor (1310) executing the renderer (1390) can divide the display area of ​​the display (1150) into a foveated portion (or may be referred to as the foveated area) and a peripheral portion (or may be referred to as the residual area) using the gaze position calculated using the position tracker (1371) and / or the gaze tracker (1374). For example, the processor (1310) detecting the coordinate values ​​of the gaze position can determine the portion of the display area including the coordinate values ​​as the foveated area. The DPU executing the renderer (1390) can obtain at least one image corresponding to each of the foveated area and the residual area, and having a size smaller than the size of the entire display area of ​​the display (1150) or a resolution smaller than the resolution of the display area.

[0187] The processor (1310) executing the renderer (1390) may obtain or generate a composite image to be displayed on the display (1150) by synthesizing an image corresponding to the foveated area and an image corresponding to the peripheral area. For example, the processor (1310) may perform upscaling to enlarge the image corresponding to the peripheral area to the size of the entire display area of ​​the display (1150). On the enlarged image, the processor (1310) may combine the image corresponding to the foveated area to generate a composite image to be displayed on the display (1150). Along the boundary line of the image corresponding to the foveated area, the processor (1310) may apply a visual effect, such as blur, to blend the enlarged image and the image corresponding to the foveated area.

[0188] Fig. 14 shows an example of a block diagram of an electronic device for displaying an image in a virtual space.

[0189] In Fig. 14, an example is described in which multiple programs / instructions are executed for displaying an image in a virtual space. The multiple programs / instructions may be executed entirely on a single processor (e.g., an AP) or by multiple processors (e.g., an AP, a GPU (graphics processing unit), or an NPU (neural processing unit)). The meaning of being executed by multiple processors means that some programs / instructions may be executed by a first processor, and other programs / instructions may be executed by a second processor different from the first processor.

[0190] Referring to FIG. 14, the electronic device (1001) may execute a virtual space manager (1450) (e.g., the virtual space manager (1351) of FIG. 13, CPM) to render an image in a virtual space. For the virtual space manager (1450), at least some of the descriptions of the virtual space manager (1351) of FIG. 13 may be referenced. The virtual space manager (1450) may include a platform for supporting a virtual space service. The virtual space manager (1450) may include a runtime service (1451) (e.g., open XR runtime), a panel renderer (1452) (e.g., 2D panel render), and an XR compositor (1453). The electronic device (1001) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (1451). For the runtime service (1451), at least some of the descriptions of the runtime service (1352) of FIG. 13 may be referred to. The electronic device (1001) may display at least one image (video) on a panel (e.g., a 2D panel) to implement a virtual space through the display based on the execution of the panel rendering (1452). For example, the electronic device (1001) may display a rendering image corresponding to RGB information (1466) for the panel from the spatialization manager (1440) described below through the display (e.g., the display (1150)). The electronic device (1001) may synthesize an image of an actual area captured by a camera in the virtual space (hereinafter, a pass-through image) with a virtual area image based on the execution of the XR compositor (1453). For example, the electronic device (1001) can generate a composite image by merging the pass-through image and the virtual area image based on the execution of the XR synthesis unit (1453).The electronic device (1001) may transmit the generated composite image to a display buffer so that the composite image is displayed. The electronic device (1001) may identify a virtual space through a virtual space manager (1450) and display at least a portion of the virtual space on the display (1150). The virtual space manager (1450) may be referred to as a CPM. The electronic device (1001) may execute the virtual space manager (1450) to render an image corresponding to at least a portion of the virtual space.

[0191] According to one embodiment, the electronic device (1001) may execute a spatialization manager (1440). The spatialization manager (1440) may perform processes for displaying an image in a three-dimensional virtual space. The electronic device (1001) may perform preprocessing based on the execution of the spatialization manager (1440) so that the image can be rendered in a three-dimensional virtual space through the virtual space manager (1450). For example, the electronic device (1001) may perform at least some of the functions of the renderer (1390) of FIG. 13 based on the execution of the spatialization manager (1440). The electronic device (1001) may process image information provided by an application (e.g., an XR application (1410), an application (1420) that provides a general 2D screen other than XR, and an application that provides a system UI (1430)) based on the execution of the spatialization manager (1440). A spatialization manager (1440) (e.g., space flinger) may include a system scene manager (1441) (e.g., system scene), an input manager (1442) (e.g., input routing), and a lightweight rendering engine (1443) (e.g., impress engine). The system scene manager (1441) may be executed to display a system UI (1430). System UI-related information (1464) may be transmitted to the system scene manager (1441) from a program (e.g., API) that provides the system UI (1430). The system UI-related information (1464) may be obtained through a spatializer API and / or a same-process private API. The spatialization manager (1440) may determine the layout (e.g., location, display order) of the screen of the system UI (1430) in a three-dimensional space through pre-allocated resources.The system screen manager (1441) may transmit image information (1467) for rendering the screen of the system UI (1430) to the virtual space manager (1450) according to the layout. The input manager (1442) may be configured to process user input (e.g., user input on a system screen or an app screen). The impression engine (1443) may be a renderer for image generation (e.g., a lightweight renderer (1343)). For example, the impression engine (1443) may be used to display the system UI (1430). According to one embodiment, the spatialization manager (1440) may include a lightweight rendering engine (1443) for rendering the system UI. According to one embodiment, when the lightweight rendering engine (1443) does not have sufficient resources to render an avatar used in the HMD, at least one external rendering engine may be used. At this time, to resolve compatibility issues with external rendering (e.g., 3rd party engines), an external rendering engine support module may be added within the spatialization manager (1440).

[0192] According to one embodiment, the electronic device can execute an application. For example, in response to the execution of an XR application (1410) (e.g., an XR application (1342), a 3D game, an XR map, or other immersive application), the electronic device can execute a virtual space manager (1450). The electronic device (1001) can provide dual image information (1461) provided from the XR application (1410) to the virtual space manager (1450). In order to display an image in a three-dimensional space, the dual image information (1461) can include two pieces of image information that take binocular parallax into account. For example, the dual image information (1461) can include first image information for the user's left eye and second image information for the user's right eye for rendering in a three-dimensional virtual space. Hereinafter, in the present disclosure, the term dual image information is used to refer to image information for displaying images for both eyes in a three-dimensional space. In addition to dual image information, the above dual image information may also include binocular image information, dual image information, dual image data, dual images, binocular image data, stereoscopic image information, 3D image information, spatial image information, spatial image data, 2D-3D conversion data, dimensional conversion image data, binocular parallax image data, and / or equivalent technical terms. The electronic device (1001) may generate a composite image by merging image layers through a virtual space manager (1450). The electronic device (1001) may transmit the generated composite image to a display buffer. The composite image may be displayed on the display (1150) of the electronic device (1001).

[0193] According to one embodiment, the electronic device can execute at least one application among an XR application (1410) and other applications (1420) (e.g., a first application (1420-1), a second application (1420-2), ..., an Nth application (1420-N)). According to one embodiment, the application (1420) can be configured to output image information for displaying a two-dimensional image. In other words, the application (1420) can provide a two-dimensional image. For example, the application (1420) can be a video application, a schedule application, or an application (1420) can be an Internet browser application. If it is assumed that in response to the execution of the application (1420), image information (1462) provided from the application (1420) is provided to the virtual space manager (1450). Since the image information (1462) only has x-coordinates and y-coordinates within a two-dimensional plane, it may be difficult to consider the chronological relationship (i.e., the distance from the user) between other applications centered on the user. Even when displaying an application (1420) that provides a general 2D screen, the electronic device (1001) may execute the spatialization manager (1440) to provide dual image information to the virtual space manager (1450). For example, based on the execution of the spatialization manager (1440), the electronic device (1001) may receive application-related information (1463) from the first application (1420-1). For example, the application-related information (1463) may include image information representing a two-dimensional image of the first application (1420-1) (e.g., information including RGB for each pixel) and / or content information in the first application (1420-1) (e.g., characteristics of content executed in the first application, type of content). Application related information (1463) can be obtained through the spatializer API.Based on the execution of the spatialization manager (1440), the electronic device (1001) can identify information about the location of the area to be rendered by the first application (1420-1) and the size of the area to be rendered (hereinafter, location information). Based on the execution of the spatialization manager (1440), the electronic device (1001) can generate dual image information (1465, e.g., RGBx2) that takes into account the user's binocular disparity through the image information and the location information. Based on the execution of the spatialization manager (1440), the electronic device (1001) can provide the dual image information (1465) to the virtual space manager (1450). By converting a simple two-dimensional image into the dual image information (1465), a problem that occurs when the image information (1462) is directly transmitted to the virtual space manager (1450) can be resolved. Additionally, since at least some of the functions for displaying images in a virtual space are performed by the spatialization manager (1440) instead of the virtual space manager (1450), the burden on the virtual space manager (1450) can be reduced.

[0194] As described above, the wearable device (e.g., the wearable device (200) of FIG. 2) may include a memory (e.g., the memory (220) of FIG. 2) storing instructions and including one or more storage media, one or more cameras (e.g., the one or more cameras (230) of FIG. 2), a display assembly including a display (e.g., the display assembly (240) of FIG. 2), and at least one processor (e.g., the at least one processor (210) of FIG. 2) including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, on the display assembly, an avatar representing a user (e.g., the avatar (605) of FIG. 6). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect, using at least a portion of the images, an external object (e.g., an external object (410) of FIG. 4) gripped by a hand (e.g., a hand (415) of FIG. 4) of the user (e.g., the user (405) of FIG. 4). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using at least a portion of the images, a first size of the hand of the user based on the detection. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on the first size, a second size of a visual object (e.g., a visual object (610) of FIG. 6) corresponding to the external object.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the avatar on the display assembly, wherein the avatar has the visual object having the second size.

[0195] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a location of the hand of the user using at least some of the images. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect an external object gripped by the hand of the user based on identifying a location of the external object that moves in accordance with movement of the location of the hand of the user.

[0196] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect, using at least a portion of the images, the external object released by the user's hand while displaying the avatar that has drawn the visual object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to stop displaying the visual object based on detecting the external object released by the user's hand.

[0197] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect, using at least a portion of the images, the external object released by the user's hand while displaying the avatar gripping the visual object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display, on the display assembly, the avatar that has released the visual object having the second size based on detecting the external object released by the user's hand. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to stop displaying the visual object based on identifying that a reference time has elapsed since detecting the external object released by the user's hand.

[0198] For example, the ratio of the first size and the third size of the external object may correspond to the ratio of the fourth size of the part corresponding to the hand of the avatar and the second size.

[0199] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, based on a first object display mode, display the visual object gripped by the avatar in a first orientation such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, based on a second object display mode, display the visual object gripped by the avatar in a second orientation such that the portion of the visual object is visible to another user.

[0200] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in the second orientation, such that the portion of the visual object faces the frontal direction of the avatar, based on the second object display mode.

[0201] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in the second orientation so that the back of the avatar, together with the portion of the visual object, is visible to the other user, based on the second object display mode.

[0202] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a user interface (UI) on the display assembly to indicate that the second object display mode is being executed, along with the visual object of the second orientation, based on the second object display mode.

[0203] As described above, the method may be performed in a wearable device including a display assembly including one or more cameras and a display. The method may include an operation of displaying an avatar representing a user on the display assembly. The method may include an operation of acquiring images using the one or more cameras while displaying the avatar. The method may include an operation of detecting an external object gripped by a hand of the user using at least a portion of the images. The method may include an operation of identifying a first size of the hand of the user using at least a portion of the images based on the detection. The method may include an operation of identifying a second size of a visual object corresponding to the external object based on the first size. The method may include an operation of displaying the avatar gripping the visual object having the second size on the display assembly.

[0204] For example, the method may include an operation of identifying a position of the hand of the user using at least a portion of the images. The method may include an operation of detecting the external object gripped by the hand of the user based on identifying a position of the external object that moves according to the movement of the position of the hand of the user.

[0205] For example, the method may include detecting, using at least a portion of the images, an external object released by the user's hand while displaying the avatar that has drawn the visual object. The method may include stopping displaying the visual object based on detecting the external object released by the user's hand.

[0206] For example, the method may include an operation of detecting, using at least a portion of the images, the external object released by the user's hand while displaying the avatar that has drawn the visual object. The method may include an operation of displaying, on the display assembly, the avatar that has released the visual object having the second size based on the detection of the external object released by the user's hand. The method may include an operation of stopping displaying the visual object based on identifying that a reference time has elapsed from the time of detecting the external object released by the user's hand.

[0207] For example, the ratio of the first size and the third size of the external object may correspond to the ratio of the fourth size of the part corresponding to the hand of the avatar and the second size.

[0208] For example, the method may include an action of displaying the avatar that has drawn the visual object by performing blur processing on an edge of the visual object.

[0209] For example, the method may include, based on a first object display mode, an operation of displaying the visual object gripped by the avatar in a first orientation such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user. The method may include, based on a second object display mode, an operation of displaying the visual object gripped by the avatar in a second orientation such that the portion of the visual object is visible to another user.

[0210] For example, the method may include an action of displaying the visual object gripped by the avatar in the second orientation such that the portion of the visual object faces the front direction of the avatar, based on the second object display mode.

[0211] For example, the method may include an action of displaying the visual object drawn by the avatar in the second orientation so that the back of the avatar together with the portion of the visual object is visible to the other user, based on the second object display mode.

[0212] For example, the method may include, based on the second object display mode, displaying a user interface (UI) on the display assembly to indicate that the second object display mode is being executed together with the visual object of the second orientation.

[0213] As described above, the non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a display assembly including one or more cameras and a display, cause the wearable device to display an avatar representing the user on the display assembly. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify a first size of the hand of the user using at least a portion of the images based on the detection. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify a second size of a visual object corresponding to the external object based on the first size. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, on the display assembly, the avatar gripping the visual object having the second size.

[0214] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify a location of the hand of the user using at least a portion of the images. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect an external object gripped by the hand of the user based on identifying a location of the external object that moves in accordance with movement of the location of the hand of the user.

[0215] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect, using at least a portion of the images, the external object released by the user's hand while displaying the avatar that has gripped the visual object. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to stop displaying the visual object based on detecting the external object released by the user's hand.

[0216] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect, using at least a portion of the images, the external object released by the user's hand while displaying the avatar gripping the visual object. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display, on the display assembly, the avatar that has released the visual object having the second size based on detecting the external object released by the user's hand. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to stop displaying the visual object based on identifying that a reference time has elapsed since detecting the external object released by the user's hand.

[0217] For example, the ratio of the first size and the third size of the external object may correspond to the ratio of the fourth size of the part corresponding to the hand of the avatar and the second size.

[0218] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the avatar that has drawn the visual object by performing a blur process on an edge of the visual object.

[0219] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to, based on a first object display mode, display the visual object gripped by the avatar in a first orientation such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to, based on a second object display mode, display the visual object gripped by the avatar in a second orientation such that the portion of the visual object is visible to another user.

[0220] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in the second orientation such that the portion of the visual object faces the frontal direction of the avatar, based on the second object display mode.

[0221] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in the second orientation so that the back of the avatar, together with the portion of the visual object, is visible to the other user, based on the second object display mode.

[0222] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a user interface (UI) on the display assembly to indicate that the second object display mode is being executed along with the visual object of the second orientation, based on the second object display mode.

[0223] As described above, the wearable device may include a memory storing instructions and including one or more storage media, a display assembly including one or more cameras, a display, and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display an avatar representing a user on the display assembly. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain a visual object corresponding to the external object using at least a portion of the images based on the detection. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in a first orientation such that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user, based on a first object display mode.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in a second orientation so that the portion of the visual object is visible to another user, based on a second object display mode.

[0224] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in the second orientation, such that the portion of the visual object faces the frontal direction of the avatar, based on the second object display mode.

[0225] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the visual object gripped by the avatar in the second orientation so that the back of the avatar, together with the portion of the visual object, is visible to the other user, based on the second object display mode.

[0226] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a user interface (UI) on the display assembly to indicate that the second object display mode is being executed, along with the visual object of the second orientation, based on the second object display mode.

[0227] As described above, the method may be performed in a wearable device including a display assembly including one or more cameras and a display. The method may include an operation of displaying an avatar representing a user on the display assembly. The method may include an operation of acquiring images using the one or more cameras while displaying the avatar. The method may include an operation of detecting an external object gripped by a hand of the user using at least a portion of the images. The method may include an operation of acquiring a visual object corresponding to the external object using at least a portion of the images based on the detection. The method may include an operation of displaying the visual object gripped by the avatar in a first orientation so that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user, based on a first object display mode. The method may include an operation of displaying the visual object gripped by the avatar in a second orientation so that the portion of the visual object is visible to another user, based on a second object display mode.

[0228] For example, the method may include an action of displaying the visual object gripped by the avatar in the second orientation such that the portion of the visual object faces the front direction of the avatar, based on the second object display mode.

[0229] For example, the method may include an action of displaying the visual object drawn by the avatar in the second orientation so that the back of the avatar together with the portion of the visual object is visible to the other user, based on the second object display mode.

[0230] For example, the method may include, based on the second object display mode, displaying a user interface (UI) on the display assembly to indicate that the second object display mode is being executed together with the visual object of the second orientation.

[0231] As described above, the non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a display assembly including one or more cameras and a display, cause the wearable device to display an avatar representing the user on the display assembly. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to acquire images using the one or more cameras while displaying the avatar. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to detect an external object gripped by a hand of the user using at least a portion of the images. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to obtain a visual object corresponding to the external object using at least a portion of the images based on the detection. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in a first orientation so that a portion of the visual object corresponding to a portion of the external object facing the user is visible to the user, based on a first object display mode. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in a second orientation so that the portion of the visual object is visible to another user, based on a second object display mode.

[0232] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in the second orientation such that the portion of the visual object faces the frontal direction of the avatar, based on the second object display mode.

[0233] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display the visual object gripped by the avatar in the second orientation so that the back of the avatar, together with the portion of the visual object, is visible to the other user, based on the second object display mode.

[0234] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a user interface (UI) on the display assembly to indicate that the second object display mode is being executed along with the visual object of the second orientation, based on the second object display mode.

Claims

1. In a wearable device (200), A memory (220) storing instructions and including one or more storage media; One or more cameras (230); A display assembly (240) including a display; and At least one processor (210) comprising a processing circuit, The above instructions, when executed individually or collectively by the at least one processor (210), On the above display assembly (240), an avatar (605) representing the user (405) is displayed, While displaying the above avatar (605), images are acquired using one or more cameras (230), Using at least some of the above images, detecting an external object (410) gripped by the hand (415) of the user (405), Based on the above detection, using at least some of the images, a first size of the hand (415) of the user (405) is identified, According to the first size, a second size of a visual object (610) corresponding to the external object (410) is identified, and On the display assembly (240), to display the avatar (605) that has the visual object (610) having the second size, causing the above wearable device (200), Wearable device (200).

2. In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), Using at least some of the above images, identifying the position of the hand (415) of the user (405), and To detect the external object (410) gripped by the hand (415) of the user (405) based on identifying the position of the external object (410) that moves according to the movement of the position of the hand (415) of the user (405). causing the above wearable device (200), Wearable device (200).

3. In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), While displaying the avatar (605) that has drawn the visual object (610), detecting the external object (410) released by the hand (415) of the user (405) using at least a part of the images, and To stop displaying the visual object (610) based on detecting the external object (410) released by the hand (415) of the user (405). causing the above wearable device (200), Wearable device (200).

4. In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), While displaying the avatar (605) that has drawn the visual object (610), detecting the external object (410) released by the hand (415) of the user (405) using at least a part of the images, Based on detecting the external object (410) released by the hand (415) of the user (405), displaying the avatar (605) that has released the visual object (610) having the second size on the display assembly (240), and To stop displaying the visual object (610) based on identifying that a reference time has elapsed from the time of detecting the external object (410) released by the hand (415) of the user (405). causing the above wearable device (200), Wearable device (200).

5. In claim 1, The ratio of the first size and the third size of the external object (410) is The fourth size of the part of the avatar (605) corresponding to the hand of the avatar (605) corresponds to the ratio of the second size, Wearable device (200).

6. In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), By performing blur processing on the edge of the visual object (610), the avatar (605) that draws the visual object (610) is displayed. causing the above wearable device (200), Wearable device (200).

7. In claim 1, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the first object display mode, the visual object (610) drawn by the avatar (605) is displayed in a first orientation so that a portion of the visual object (610) corresponding to a portion of the external object (410) facing the user (405) is shown by the user (405), and Based on the second object display mode, the visual object (610) drawn by the avatar (605) is displayed in a second orientation so that the part of the visual object (610) is shown to another user (405). causing the above wearable device (200), Wearable device (200).

8. In claim 7, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the second object display mode, the visual object (610) drawn by the avatar (605) is displayed in the second orientation so that the portion of the visual object (610) faces the front direction of the avatar (605). causing the above wearable device (200), Wearable device (200).

9. In claim 7, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the second object display mode, the back of the avatar (605) together with the part of the visual object (610) is displayed in the second orientation so that the other user (405) can see the visual object (610). causing the above wearable device (200), Wearable device (200).

10. In claim 7, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the second object display mode, a user interface (UI) (405) is displayed on the display assembly (240) to indicate that the second object display mode is being executed together with the visual object (610) of the second orientation. causing the above wearable device (200), Wearable device (200).

11. In a wearable device (200), A memory (220) storing instructions and including one or more storage media; One or more cameras (230); A display assembly (240) including a display; and At least one processor (210) comprising a processing circuit, The above instructions, when executed individually or collectively by the at least one processor (210), On the above display assembly (240), an avatar (605) representing the user (405) is displayed, While displaying the above avatar (605), images are acquired using one or more cameras (230), Using at least some of the above images, detecting an external object (410) gripped by the hand (415) of the user (405), Based on the above detection, using at least a part of the images, a visual object (610) corresponding to the external object (410) is obtained, Based on the first object display mode, the visual object (610) drawn by the avatar (605) is displayed in a first orientation so that a portion of the visual object (610) corresponding to a portion of the external object (410) facing the user (405) is shown by the user (405), and Based on the second object display mode, the visual object (610) drawn by the avatar (605) is displayed in a second orientation so that the part of the visual object (610) is shown to another user (405). causing the above wearable device (200), Wearable device (200).

12. In claim 11, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the second object display mode, the visual object (610) drawn by the avatar (605) is displayed in the second orientation so that the portion of the visual object (610) faces the front direction of the avatar (605). causing the above wearable device (200), Wearable device (200).

13. In claim 11, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the second object display mode, the back of the avatar (605) together with the part of the visual object (610) is displayed in the second orientation so that the other user (405) can see the visual object (610). causing the above wearable device (200), Wearable device (200).

14. In claim 11, The above instructions, when executed individually or collectively by the at least one processor (210), Based on the second object display mode, a user interface (UI) (405) is displayed on the display assembly (240) to indicate that the second object display mode is being executed together with the visual object (610) of the second orientation. causing the above wearable device (200), Wearable device (200).

15. A method executed within a wearable device (200) comprising a display assembly (240) including one or more cameras (230) and a display, the method comprising: An operation of displaying an avatar (605) representing a user (405) on the above display assembly (240), While displaying the above avatar (605), an operation of acquiring images using one or more cameras (230), An operation of detecting an external object (410) gripped by a hand (415) of the user (405) using at least a portion of the above images, Based on the above detection, an operation of identifying a first size of the hand (415) of the user (405) using at least some of the images; An operation of identifying a second size of a visual object (610) corresponding to the external object (410) according to the first size, and An operation of displaying the avatar (605) that draws the visual object (610) having the second size on the display assembly (240), method.

Images