Electronic device, method, and computer-readable medium for displaying virtual object

Abstract

In embodiments, an electronic device is provided. The electronic device may comprise: a camera; a display; at least one sensor; a memory for storing instructions; and at least one processor. The instructions, when executed by the at least one processor, may cause the electronic device to: obtain an external image of the electronic device through the camera while displaying a virtual image of a virtual environment through the display; display, through the display, at least a part of the external image in a virtual area according to first depth information in the virtual environment, while displaying the virtual image; obtain a distance from an external object included in at least the part of the external image through the at least one sensor; and display a virtual object corresponding to the external object through the display while displaying the virtual image and at least the part of the external image, on the basis that a position corresponding to the distance from the external object is within a space according to second depth information.

Description

Electronic device, method, and computer-readable medium for displaying virtual objects

[0001] The present disclosure relates to an electronic device, method, and computer-readable medium for displaying a virtual object.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices can provide AR services to users by utilizing virtual objects corresponding to the user.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] In embodiments, an electronic device is provided. The electronic device may include a camera, a display, at least one sensor, a memory storing instructions, and at least one processor. The instructions, when executed by the at least one processor, may cause the electronic device to: acquire an external image of the electronic device through the camera while displaying a virtual image of a virtual environment through the display; display at least a portion of the external image through the display in a virtual area in the virtual environment according to first depth information while displaying the virtual image; acquire a distance to an external object included in at least a portion of the external image through the at least one sensor; and display a virtual object corresponding to the external object through the display while displaying the virtual image and at least a portion of the external image based on a location corresponding to the distance to the external object being within a space according to second depth information.

[0005] In embodiments, an electronic device is provided. The electronic device may include a camera, a display, at least one sensor, a memory storing instructions, and at least one processor. The instructions, when executed by the at least one processor, may cause the electronic device to: acquire an external image representing at least a portion of a real environment around the electronic device through the camera while displaying a virtual image of a virtual environment through the display; display the external image through the display in a virtual area according to first depth information in the virtual environment while displaying the virtual image; acquire location information of the external object based on a distance from the external object acquired through the at least one sensor; display a virtual object corresponding to the external object through the display while displaying the virtual image and at least a portion of the external image when the location information of the external object is within a space according to second depth information; and stop displaying the virtual object when the location information of the external object is outside the space according to the second depth information and is located between the first depth information and the second depth information. The second depth information may indicate a distance closer than the distance corresponding to the first depth information from the reference position of the virtual environment.

[0006] In embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory including instructions. The instructions, when executed by at least one processor, may cause an electronic device to, while displaying a virtual image of a virtual environment through a display, acquire an external image of the electronic device through a camera; while displaying the virtual image, display the external image through at least a portion of the display in a virtual area according to first depth information in the virtual environment; acquire a distance to an external object included in at least a portion of the external image through at least one sensor; and, based on a location corresponding to the distance to the external object being within a space according to second depth information, display a virtual object corresponding to the external object through the display while displaying the virtual image and at least a portion of the external image.

[0007] Figure 1 illustrates an example of an operation of displaying an external image acquired through a camera on a virtual environment.

[0008] Figure 2a illustrates an example of a perspective view of a wearable device.

[0009] FIG. 2b illustrates an example of one or more hardware devices arranged within a wearable device.

[0010] Figures 3a and 3b illustrate an example of the appearance of a wearable device.

[0011] Figure 4 illustrates an example of a block diagram of a wearable device.

[0012] Figure 5 illustrates an example of a pass-through operation of a wearable device.

[0013] Figure 6 illustrates an example of depth information for pass-through operation of a wearable device.

[0014] Figure 7 shows an example of displaying a virtual object according to depth information.

[0015] Figure 8 illustrates an example of displaying external images and virtual objects.

[0016] Figure 9 illustrates an example of displaying an external image and a virtual object according to the movement of the virtual object.

[0017] Figure 10 illustrates an example of displaying a virtual object in relation to the movement of the virtual object.

[0018] Figures 11a and 11b illustrate examples of pass-through surfaces.

[0019] Figure 12 illustrates the operation flow of a wearable device for displaying a virtual object on an external image.

[0020] Figure 13 illustrates an example of a boundary area for displaying a virtual object.

[0021] Figure 14 illustrates an example of a border area and virtual environment for displaying a virtual object.

[0022] Figure 15 shows an example of a boundary area of ​​a curved surface.

[0023] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0024] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0025] In the following description, terms referring to space (e.g., plane, object, shape, surface, figure, solid figure, area, occupied area, position, depth, distance), terms referring to distance (e.g., z-index, z-order, z-depth, z-axis value, data, depth data, distance information, distance value, depth value, position information, position data), terms referring to pass-through surfaces (e.g., pass-through area, virtual surface, pass-through surface, view surface, virtual area, view area, transition surface, transition space, transition area, pass surface, pass area), terms referring to values ​​(e.g., threshold value, reference value, reference area, reference range, level, threshold, range, value, area), terms for operation states (e.g., step, operation, procedure), terms referring to network entities, terms referring to components of a device, etc., are examples for convenience of description. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used.

[0026] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0027] FIG. 1 illustrates an example of an operation of displaying an external image acquired through a camera on a virtual environment. The wearable device (101) of FIG. 1 may include a head-mounted display (HMD) that is wearable on the head of a user (105). The wearable device (101) may include a camera (e.g., camera (425) of FIG. 4) positioned toward the front of the user (105) when the device is worn by the user (105). The front of the user (105) may include the head of the user (105) and / or the direction in which the gaze of the user (105) is directed. The wearable device (101) may include a sensor for identifying the motion of the head of the user (105) and / or the wearable device (101) when the device is worn by the user (105). The wearable device (101) can identify the angle of the wearable device (101) based on the data of the sensor. In order to provide a user interface (UI) based on virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) to a user (105) wearing the wearable device (101), the wearable device (101) can control the camera and / or the sensor. The UI can be related to a metaverse service provided by the wearable device (101), and / or a server connected to the wearable device (101), and / or a notification service.

[0028] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). Referring to FIG. 1, when a user (105) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the eyes of the user (105). Ambient light passing through the lens of the wearable device (101) may be combined (or mixed) with light emitted from a display of the wearable device (101) (e.g., display (420) of FIG. 4). A display area of ​​the display may be formed in at least a portion of the lens through which the ambient light passes. Because the wearable device (101) displays at least part of an image generated based on ambient light and at least part of the light emitted from the display, the user (105) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display.

[0029] The wearable device (101) can execute functions related to video see-through (VST) and / or virtual reality (VR). For example, the VST function may include an operation of rendering an image to be expressed as augmented reality on an image of a real background captured by a camera and showing the final result to the user. For example, the VR-related function may include eye tracking and / or hand tracking. When the user (105) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (105). In the state, the wearable device (101) may include a display arranged on a first surface facing the eyes (e.g., the first surface (310) of FIG. 3A). The wearable device (101) may include a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12 of FIG. 3B) disposed on a second surface opposite to the first surface (e.g., the second surface (320) of FIG. 3A). Using the camera, the wearable device (101) may acquire frame images including ambient light. The wearable device (101) may output the frame images to a display disposed on the first surface, thereby allowing a user (105) to recognize the ambient light through the display. A display area of ​​the display disposed on the first surface may be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object within frame images output through the display to enable the user (105) to recognize the virtual object together with a real object recognized by ambient light.

[0030] According to one embodiment, a wearable device (101) may provide a user experience based on mixed reality (MR) by using a virtual environment (or virtual space). The wearable device (101) may recognize an external space (e.g., a real space) in which the wearable device (101) is included, and generate a virtual space mapped to the external space. The wearable device (101) recognizing the external space may include an operation of obtaining information about the size of the external space (e.g., the size of the external space defined by a side wall, a floor surface, and / or a ceiling surface). The wearable device (101) recognizing the external space may include an operation of identifying an external plane (e.g., a ceiling and / or a floor) included in the external space. The wearable device (101) identifying the external space may include an operation of identifying a virtual object (e.g., a user interface (UI) for displaying at least one image) displayed within a display.

[0031] According to one embodiment, the wearable device (101) can provide an external object (or an existence object) recognized by ambient light and a visual object (or a virtual object) displayed by the display to the user (105). For example, the wearable device (101) can obtain an image (110) corresponding to the gaze of the user (105) obtained using a sensor. The image (110) can be obtained using a camera of the wearable device (101). For example, the image (110) can correspond to the field-of-view (FoV) of the camera of the user (105) or the wearable device (101). The image (110) can be referred to as an external image (110) in the sense that it represents at least a portion of the actual environment around the wearable device identified through the camera. For example, an image (110) may be referred to as a frame image from the perspective of corresponding to at least one of a plurality of frame images acquired through a camera.

[0032] According to one embodiment, the wearable device (101) may receive an input for acquiring an external image (110) while displaying a three-dimensional image (150) corresponding to a virtual environment (or virtual space) through a display. The three-dimensional image (150) may be acquired based on the execution of at least one software application installed in the wearable device (101).

[0033] According to one embodiment, the wearable device (101) may initiate operation of the camera based on receiving an input for acquiring an external image (110). The wearable device (101) may acquire an external image (110) corresponding to the gaze of the user (105) using the camera. For example, the external image (110) may be acquired based on color information among color information or depth information. The wearable device (101) may store the external image (110) in a command buffer that is distinct from a display buffer corresponding to a display in the memory of the wearable device (101). The display buffer may store information about a screen (130) to be displayed on the display. For example, the display buffer may refer to an area of ​​memory where information about the screen is at least temporarily stored in order to display the screen on the display. The wearable device (101) can display the screen (130) on the display using information about the screen (130) stored in the display buffer. For example, the screen (130) may refer to a user interface (UI) displayed in at least a portion of the display. The screen (130) may, within the screen (130), the wearable device (101) may display one or more visual objects (or one or more virtual objects). The visual objects may refer to objects that can be deployed within the screen for transmitting and / or interacting with information, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables.A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element.

[0034] According to one embodiment, a command buffer may be used by the wearable device (101) to render a screen to be displayed on the display. The command buffer may correspond to each of the images acquired by the wearable device (101) using each of one or more programs (e.g., software applications, libraries, and / or services) installed in the wearable device (101). The command buffer may store an image layer representing an image used to render a screen to be displayed on the display. For example, the command buffer may refer to an area of ​​memory where the image layer is at least temporarily stored to render the screen.

[0035] The wearable device (101) can obtain a screen to be displayed on the display by rendering image layers corresponding to each of the acquired images. The screen to be obtained by the wearable device (101) may vary based on the order of the image layers.

[0036] The screen (130) provided by the wearable device (101) may include a three-dimensional image (150) and a virtual area (155). The wearable device (101) may display a virtual area (155) to provide at least a portion of an external image (110) to the user (105). The wearable device (101) may display the virtual area (155) while displaying the three-dimensional image (150). Through the virtual area (155), at least a portion of the external image (110) may be exposed. Through the virtual area (155), the user may perceive a real environment in addition to a virtual environment. For example, the wearable device (101) may obtain spatial information about a virtual environment corresponding to the three-dimensional image (150) in order to display the virtual area (155) within the three-dimensional image (150). The wearable device (101) can identify the location of a virtual area (155) within the virtual environment using the spatial information. Based on the location of the virtual area (155), the wearable device (101) can display at least a portion of an external image (110) through a display (e.g., display (420) of FIG. 4). Based on displaying at least a portion of the external image (110) on the virtual area (155), the wearable device (101) can provide a user experience capable of interacting with the external environment.

[0037] The wearable device (101) can perform rendering to obtain a screen (130) to be stored in a display buffer and displayed through a display. The wearable device (101) can place an image layer corresponding to an external image (110) on an image layer corresponding to the virtual area (155) to display the external image (110) on at least a portion of the virtual area (155). The wearable device (101) can perform rendering to display the virtual area (155) and the external image (110) in an overlapping manner on a three-dimensional image (150), thereby obtaining the screen (130). The screen (130) can include the external image (110), the virtual area (155), and the three-dimensional image (150) that are overlaid based on the above order. The external image (110) may be referred to as a pass-through image from the perspective that the surrounding environment of the wearable device (101) is shown to the user (105) by passing through the three-dimensional image (150). The external image (110) may also be referred to as a punch-through image from the perspective that at least a portion of the three-dimensional image (150) is passed through and at least a portion of the external image (110) is shown to the user (105). In addition, an operation of displaying a virtual area (155) and providing at least a portion of the external image (110) to the user (105) may be referred to as a pass-through operation or a punch-through operation. In addition to a virtual area, the virtual area (155) may also be referred to as a virtual surface, a punch-through area, a virtual surface, a punch-through surface, a pass-through area, a pass-through surface, a viewing surface, a viewing area, a transition surface, a transition space, a transition area, a passing surface, a passing area, or terms having equivalent technical meanings thereto.

[0038] Although a single virtual area (155) is illustrated in FIG. 1, embodiments of the present disclosure are not limited thereto. For example, the screen (130) may include multiple virtual areas (not shown). Each virtual area of ​​the multiple virtual areas may display at least a portion of an external image (110).

[0039] FIG. 2A illustrates an example of a perspective view of a wearable device. FIG. 2B illustrates an example of one or more hardware components arranged within the wearable device. According to one embodiment, the wearable device (101) may have the form of glasses that are wearable on a body part (e.g., head) of a user. The wearable device (101) of FIGS. 2A and 2B may be an example of the wearable device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (101) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0040] Referring to FIG. 2A, according to one embodiment, a wearable device (101) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0041] According to one embodiment, a wearable device (101) can be worn on a part of a user's body. The wearable device (101) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0042] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0043] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to the user through a lens included in the at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (101), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0044] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (101) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0045] The wearable device (101) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view an image displayed on at least one display (250).

[0046] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0047] Referring to FIG. 2A, the frame (200) may include a region (220) that is in contact with at least a portion of a user's body when the user wears the wearable device (101). For example, the region (220) of the frame (200) that is in contact with a portion of the user's body may include a region that is in contact with a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (101) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a portion of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may be in contact with a portion of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another portion of the user's body that is distinct from the portion of the user's body.

[0048] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (101) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0049] According to one embodiment, the wearable device (101) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0050] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (101) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B , but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B . When the number of microphones (265) included in the wearable device (101) is two or more, the wearable device (101) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0051] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or may be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0052] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (101) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (101) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0053] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (101) can perform object recognition through an image acquired using the capturing camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (260-4). The wearable device (101) can perform a pass-through function to display an image acquired through the capturing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the capturing camera (260-4) can be disposed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0054] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (101) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (101) looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is positioned.

[0055] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the motion recognition camera (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0056] The camera (260) included in the wearable device (101) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (101) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (101) can identify an external object based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (101).

[0057] Although not shown, in one embodiment, the wearable device (101) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0058] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (101). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0059] The antenna module (275) can transmit signals or power to the outside of the wearable device (101), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0060] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0061] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, when the wearable device (101) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0062] Referring to FIG. 2B, according to one embodiment, a wearable device (101) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (101) (e.g., hardwares illustrated by different blocks in FIG. 4A) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0063] According to one embodiment, a wearable device (101) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (101) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (101) based on the IMU.

[0064] FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (e.g., wearable device (101)). The wearable device (101) of FIGS. 3A and 3B may be an example of the wearable device (101) of FIG. 1 . According to one embodiment, an example of an exterior appearance of a first side (310) of a housing of the wearable device (101) is illustrated in FIG. 3A , and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B .

[0065] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (101) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (101) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (101) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0066] According to one embodiment, the wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0067] Referring to FIG. 3B, a camera (e.g., cameras (260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (101) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0068] For example, using cameras (260-11, 260-12), the wearable device (101) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (101) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (101) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0069] According to one embodiment, the wearable device (101) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (101) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0070] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) is described.

[0071] Fig. 4 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (101)). The wearable device (101) of Fig. 4 may be an example of the wearable device (101) of Fig. 1 or the wearable devices (101) of Figs. 2A to 3B.

[0072] Referring to FIG. 4, a wearable device (101) according to one embodiment may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (480). The processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (480) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 4. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 4. The elements within the memory described below (e.g., layers and / or modules) may be logically distinct, but are not limited thereto.

[0073] According to one embodiment, the processor (410) of the wearable device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0074] The memory (415) of the wearable device (101) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410). The memory (415) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi-media card (eMMC).

[0075] In one embodiment, the display (420) of the wearable device (101) can output visualized information to a user of the wearable device (101). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0076] In one embodiment, the camera (425) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0077] According to one embodiment, the wearable device (101) may include a plurality of cameras (not shown) arranged in different directions, as an example of a camera (425). A first camera among the plurality of cameras may be referred to as a motion recognition camera, and a second camera may be referred to as an eye tracking camera. The wearable device (100) may identify a position, shape, and / or gesture of a hand using an image of the first camera (e.g., image (110) of FIG. 1). The wearable device (101) may identify a direction of a user's (e.g., user (105) of FIG. 1) gaze using an image of the second camera (not shown). For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0078] According to one embodiment, a sensor (430) of a wearable device (101) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (101) from non-electronic information related to the wearable device (101). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (101).

[0079] In one embodiment, the communication circuit (480) of the wearable device (101) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device (e.g., the server (405) of FIG. 4). The communication circuit (480) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (480) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0080] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (410) of the wearable device (101) may be stored in the memory (415) of the wearable device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (101) and / or the processor (410) may perform at least one of the operations described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, an application may include a program and / or a library related to a service provided to a user.

[0081] Referring to FIG. 4, programs installed in the wearable device (101) may be classified into any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (460), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (101) (e.g., a display (420), a camera (425), and / or a sensor (430)) may be classified within the hardware abstraction layer (460). For example, although FIG. 4 illustrates the layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0082] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (460) and / or the application layer (440) (e.g., a position tracker (453-1), a space recognizer (453-2), a gesture tracker (453-3), an eye tracker (453-4), a controller tracker (453-5), a virtual space analyzer (455), and / or a renderer (454)) may be classified. Programs classified into the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0083] For example, within the application layer (440), programs designed to target a user (e.g., a user (105) of FIG. 1) controlling a wearable device (101) may be classified. As an example of programs classified into the application layer (440), the first software application (441) is exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) classified into the application layer (440) may call an API to cause the execution of functions supported by programs classified into the framework layer (450). For example, the wearable device (101) may display the screen (130) of FIG. 1 representing a virtual environment on the display (420) based on the execution of the first software application (441).

[0084] According to one embodiment, the virtual environment manager (451) may be configured for a virtual environment service. For example, the virtual environment manager (451) may include a platform (e.g., an Android platform) for supporting the virtual environment service. The virtual environment manager (451) may display the posture of a virtual object representing the user's posture, rendered using data acquired through the sensor (430), on the display. The virtual environment manager (451) may be referred to as a composition presentation manager.

[0085] For example, the virtual environment manager (451) may include a runtime service (451-1). As an example, the runtime service (451-1) may be referred to as an OpenXR runtime module. The runtime service (451-1) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (101). As an example, the runtime service (451-1) may be used to perform rendering for a virtual environment service to the user. For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the runtime module (451-1).

[0086] For example, the perception abstract layer (452) can be used for data exchange between the virtual environment manager (451) and the perception service layer (453). From the perspective of being used for data exchange between the virtual environment manager (451) and the perception service layer (453), the perception abstract layer (452) can be referred to as an interface. For example, the perception abstract layer (452) can be referenced as OpenPX. The perception abstract layer (452) can be used for a perception client and a perception service.

[0087] According to one embodiment, the recognition service layer (453) may include one or more programs for processing data acquired from a sensor (430) (or camera (425)). The one or more programs may include at least one of a position tracker (453-1), a spatial recognizer (453-2), a gesture tracker (453-3), a gaze tracker (453-4), and a controller tracker (453-5).

[0088] For example, the position tracker (453-1) can identify the posture of the wearable device (101) by using at least one sensor of the wearable device (101). As an example, the position tracker (453-1) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) based on data acquired using a camera (e.g., camera (425) of FIG. 4) and an IMU. The position tracker (453-1) may be referred to as a head tracking (HeT) module.

[0089] For example, the space recognizer (453-2) may be used to construct a three-dimensional virtual space surrounding the wearable device (101) (or a user of the wearable device (101)). The space recognizer (453-2) may be used to reconstruct the three-dimensional surroundings of the wearable device (101) based on data acquired using a camera (e.g., the camera (425) of FIG. 4). The space recognizer (453-2) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (101). The space recognizer (453-2) may be referred to as a scene understanding (SU) module.

[0090] For example, the gesture tracker (453-3) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of a wearable device (101). As an example, the gesture tracker (453-3) may identify a pose and / or gesture of a hand of a user based on data acquired from at least one sensor. As an example, the gesture tracker (453-3) may identify a pose and / or gesture of a hand of a user based on data acquired using a camera (e.g., an image (110)). The gesture tracker (453-3) may be referred to as a hand tracking (HaT) module.

[0091] For example, the gaze tracker (453-4) may be used to identify (or track) eye movements of a user of the wearable device (101). For example, the gaze tracker (453-4) may identify eye movements of the user based on data acquired from at least one sensor. For example, the gaze tracker (453-4) may identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED). The gaze tracker (453-4) may be referred to as an eye tracking (ET) module.

[0092] For example, the controller tracker (453-4) may be used to identify (or track) the movement of a controller connected to a wearable device (101). As an example, the controller tracker (453-4) may identify the movement of the controller based on receiving sensor data included in the controller through a communication circuit.

[0093] For example, the recognition service layer (453) of the wearable device (101) may further include a face tracking module (not shown) for tracking the user's face. For example, the face tracking module may be used to identify (or track) the user's facial movements and / or the user's facial expressions. The face tracking module may estimate the user's facial expressions based on the user's facial movements. As an example, the face tracking module may identify the user's facial movements and / or the user's facial expressions based on data (e.g., images) acquired using a camera.

[0094] For example, the wearable device (101) may obtain data representing the user's posture and / or movement from one or more programs included in the recognition service layer (453) based on the execution of the renderer (454), thereby rendering a screen including a virtual object representing the user's posture within a virtual environment. The wearable device (101) may render one or more images (e.g., virtual object animation) for changing the posture of the virtual object using the posture information (470) based on the execution of the renderer (454). The wearable device (101) may change the posture of the entire virtual object based on inverse kinematics by identifying the posture of at least a portion of the user. When changing a portion of the virtual object, the wearable device (101) may change the posture of the entire virtual object by inferring the position at which another portion of the virtual object will change based on position information within the virtual environment using the inverse kinematics.

[0095] For example, the wearable device (101) can identify virtual environment information based on the execution of the virtual environment analyzer (455). As an example, the virtual environment information can be received from the server (405) to the wearable device (101).

[0096] For example, the posture information (470) may include one or more images representing the posture of the virtual object. As an example, the wearable device (101) may receive the posture information (470) from the server (405). The wearable device (101) may use the posture information (470) to identify the posture of the virtual object corresponding to the user's posture, thereby controlling the posture of the virtual object displayed on the display to be changed.

[0097] Referring to FIG. 4, the wearable device (101) and the server (405) may be connected to each other based on a wired network and / or a wireless network. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the wearable device (101) and the server (405) are illustrated as being directly connected, the wearable device (101) and the server (405) may be indirectly connected via an intermediate node within the network. The server (405) may be referred to as an external electronic device in that it is located outside the wearable device (101).

[0098] Referring to FIG. 4, in one embodiment, the server (405) may include at least one of a processor (410-1), a memory (415-1), and a communication circuit (480-1). Within the server (405), the processor (410-1), the memory (415-1), and the communication circuit (480-1) may be electrically and / or operatively coupled via a communication bus. The processor (410-1), the memory (415-1), and the communication circuit (480-1) included within the server (405) may include hardware components and / or circuits corresponding to the processor (410), the memory (415), and the communication circuit (480) of the wearable device (101). Hereinafter, to reduce repetition, descriptions of hardware and / or software included within the server (405) may be omitted to the extent that they overlap with the electronic device (101).

[0099] According to one embodiment, the server (405) can transmit a screen (e.g., screen (130) of FIG. 1) representing a virtual environment to the wearable device (101) using a virtual environment plug-in (490). The server (405) can provide a virtual environment service to the wearable device (101) (or the user (105)) using the virtual environment plug-in (490). The server (405) can store a program including instructions for interaction with the wearable device (101) in the memory (415-1). As an example of the program, the first software application (441-1) is exemplified, but the embodiment is not limited thereto.

[0100] For example, the server (405) may receive data (e.g., sensor data) acquired from one or more programs included in the recognition service layer (453) of the wearable device (101) from the wearable device (101). The server (405) may use the received data to generate rendering data for changing the posture of a virtual object using the renderer (454-1). For example, the server (405) may generate rendering data for changing the posture of a virtual object to the wearable device (101) based on the execution of the first software application (441-1). The server (405) may transmit the rendering data to the wearable device (101) to display a screen including a virtual object whose posture has been changed using the rendering data on the display (420) of the wearable device (101). However, the present invention is not limited thereto.

[0101] FIG. 5 illustrates an example of a pass-through operation of a wearable device (e.g., wearable device (101)). The wearable device (101) can provide a virtual environment to a user (105). Since the display (e.g., display (420) of FIG. 4) of the wearable device (101) is physically positioned to be isolated from the outside, the user (105) can obtain an immersive user experience based on the virtual environment. In addition, the wearable device (101) can obtain an image corresponding to the field-of-view (FoV) of the user (105) through a camera (e.g., camera (425) of FIG. 4). The pass-through operation can include an operation of displaying at least a portion of the image corresponding to the FOV while displaying a 3D image of the virtual environment. By displaying at least a portion of the image through the display, the wearable device (101) can provide the user (105) with not only a VR experience, but also an augmented reality (AR) and / or mixed reality (MR) experience. The pass-through operation can be utilized to increase the productivity of the wearable device (101). For example, through the pass-through operation, the user (105) of the wearable device (101) can recognize an input device (e.g., a keyboard, a mouse) from an image of a real environment in an office space configured as a virtual environment.

[0102] Referring to FIG. 5, the wearable device (101) can obtain an image (510) through a camera (e.g., the camera (425) of FIG. 4). The image (510) can represent at least a part of the real environment of the user (105) of the wearable device (101). The wearable device (101) can obtain the image (510) based on the gaze direction of the user (105). The wearable device (101) can obtain the image (510) corresponding to the FoV of the user (105). The image (510) can include an external object (520) within the real environment (e.g., the hand of the user (105) of FIG. 1, a controller connected to the wearable device (101), an object positioned within the FoV of the user).

[0103] A wearable device (101) can provide a virtual environment to a user (105). The wearable device (101) can generate a virtual image (570) (e.g., a 3D image) for the virtual environment. The wearable device (101) can display the virtual image (570) through a display (e.g., the display (420) of FIG. 4). The user (105) of the wearable device (101) may want to check at least a portion of the actual environment in addition to the virtual environment. The wearable device (101) can perform a pass-through operation. For example, the wearable device (101) can operate the camera (425) based on a user input. The wearable device (101) can acquire an image (510) through the camera (425). The wearable device (101) can display a screen (560) through the display (420) through the pass-through operation. The screen (560) can include a virtual image (570) representing a virtual environment. The screen (560) can include a virtual area (580) for representing at least a portion of a real environment. For example, the virtual area (580) can include at least a portion of an image (510) acquired through a camera (425) (e.g., an external image (575)). The wearable device (101) can display the external image (575) within the virtual area (580). The external image (575) represents at least a portion of the image (510). The external image (575) can include an external object (520).

[0104] A virtual area (580) for displaying an external image (575) exists at a specific location within a virtual environment. The wearable device (101) can display the virtual area (580) based on location information for a pass-through operation. The wearable device (101) can obtain the external image (575) from the image (510) based on the location information of the virtual area (580). The image (510) can correspond to the FoV of the user (105). The external image (575) can include at least a portion of the image (510) corresponding to the direction in which the virtual area (580) is located. The wearable device (101) can display the virtual image (570) outside the virtual area (580) while displaying the external image (575) within the virtual area (580).

[0105] To solve the above-described problem, the wearable device (101) according to embodiments of the present disclosure can display a virtual object corresponding to an external object (520) while performing a pass-through operation, so as to reduce the sense of incongruity between the real environment and the virtual environment due to the external object (520). The wearable device (101) can display the virtual object while the virtual image (570) and the external image (575) are displayed.

[0106] FIG. 6 illustrates an example of depth information for a pass-through operation of a wearable device (e.g., the wearable device (101) of FIG. 1). For a description of the pass-through operation of the wearable device (101), reference may be made to FIG. 5 . The same reference numerals may indicate that the same description is applicable.

[0107] Referring to FIG. 6, the wearable device (101) can display a virtual image (570) representing a virtual environment. Unlike a two-dimensional image corresponding to a plane (e.g., an xy plane), the virtual image (570) can include depth information (e.g., coordinate information on the z-axis) that forms a space. The depth information can indicate a distance from a user (105) of the wearable device (101) to a rendering target for 3D visualization. For example, in order to display a sense of perspective of a virtual object in a virtual environment, the depth information of one side of the virtual object and the depth information of another side of the virtual object can be different. Even if the same virtual object is rendered in a virtual environment, the user of the wearable device (101) can recognize that the virtual object occupies a designated space within the virtual environment due to different depth information.

[0108] According to one embodiment, the wearable device (101) may use image layers having different depth information to express a target for rendering (e.g., an object, an image) in a virtual space with a sense of distance. Due to the different depth information, a sense of perspective may be expressed in 3D visualization. The wearable device (101) may generate the image layers based on the location of the user (105). For example, a reference layer (620) may be defined based on the user (105). Based on the location of the user (105), the wearable device (101) may generate a plurality of image layers. Each of the plurality of image layers may have depth information based on the user (105). The plurality of image layers may be aligned and stacked for rendering. For example, an image layer corresponding to a location where the user's (105's) field of view begins in the virtual space may be a reference layer (620). For example, the depth information (e.g., z-axis value) of the reference layer (620) may be '0'. The wearable device (101) may generate an image layer (e.g., a first image layer (621)) corresponding to the virtual image (570) in order to render the virtual image (570). Since the virtual image (570) is located at a certain distance (e.g., about 100 m) or more from the user (105) in the virtual space, it may have depth information corresponding to the certain distance. For example, the depth information of the image layer (e.g., the first image layer (621)) corresponding to the virtual image (570) may be '100'.

[0109] According to one embodiment, the wearable device (101) can create a virtual area (580) within a virtual environment through a pass-through operation. The pass-through operation may include an operation of displaying at least a portion of an image (e.g., an external image (575)) corresponding to the FOV while displaying a virtual image (570) of the virtual environment. The wearable device (101) can display at least a portion of a real environment (e.g., an external image (575) of FIG. 5) through the virtual area (580) while displaying the virtual image (570). The wearable device (101) can acquire the external image (575) for the pass-through operation. For example, the wearable device (101) can acquire an image corresponding to the FoV (e.g., an image (510) of FIG. 5) through a camera (e.g., a camera (425) of FIG. 4). According to one embodiment, the wearable device (101) may obtain an external image (575) from the image (510) based on the location information of the virtual area (580). For example, the wearable device (101) may obtain, as the external image (575), an area in the image (510) corresponding to the direction in which the virtual area (580) is located in the virtual space. In addition to the camera (425), the wearable device (101) may obtain an external image in other ways. For example, the wearable device (101) may download information about the external image from an external server. In addition, for example, the wearable device (101) may obtain an image through another camera separately provided from the wearable device (101). The virtual area (580) may have various shapes. For example, the virtual area (580) may include a curved surface or a flat surface including one surface in a three-dimensional space. The wearable device (101) can associate an external image (575) with an image layer (e.g., a second image layer (622)) corresponding to depth information (hereinafter, first depth information (641)) of a virtual area (580).The wearable device (101) can display an external image (575) by rendering the second image layer (622). Since the virtual image (570) is located at a certain distance (e.g., approximately 70 m) or more from the user (105) in the virtual space, it can have depth information corresponding to the certain distance. For example, the first depth information (641) of the image layer (e.g., the second image layer (622)) corresponding to the external image (575) can be '70'. The first depth information (641) can represent approximately 70 m as a distance in the virtual space.

[0110] According to one embodiment, the external image (575) may include an external object (520). The external object (520) may be located within the FoV. The distance perceived by the user (105) in the virtual environment may vary depending on the rendered image layers (e.g., the first image layer (621) and the second image layer (622) of the virtual image (570). For example, the first depth information (641) of the virtual area (580) may indicate about 3 meters (m) in the virtual environment. The wearable device (101) may display the external object (520) of the external image (575) together with the virtual image (570) and the virtual area (580) representing the virtual environment. Since the virtual area (580) is rendered within the virtual environment based on the first depth information (641), the external image (575) within the virtual area (580) may also be displayed based on the first depth information (641). In a real environment, the distance (642) between the wearable device (101) and the external object (520) may be about 30 cm. The user (105) of the wearable device (101) may recognize that the external object (520) on the external image (575) is at a distance of 30 m or more, since the distance from the reference position to the virtual area (580) is about 30 m. If there is a large difference between the distance to the virtual area (580) (e.g., the distance indicated by the first depth information (641)) and the distance (642) of the external object (520) (e.g., the actual distance between the user (105) and the actual external object (520)) (e.g., the difference is greater than a distance threshold value), the user (105) may feel a sense of incongruity between the virtual environment and the real environment.

[0111] To alleviate the above-described problem, the wearable device (101) according to embodiments of the present disclosure may display a virtual object corresponding to the external object (520) within a virtual environment based on an actual distance (e.g., distance (642)) between the external object (520) and the wearable device (101). The wearable device (101) may display the virtual object corresponding to the external object (520) to provide a virtual environment close to the actual environment to the user (105) based on the distance between the external object (520) and the wearable device (101). The wearable device (101) may display the virtual object corresponding to the external object (520) so as to overlap an area in the screen where the external object (520) is displayed.

[0112] The wearable device (101) may identify whether a specified condition is met to determine whether to display the virtual object. According to one embodiment, the wearable device (101) may identify whether a distance from the wearable device (101) to an external object (e.g., an external object (520) of FIG. 5) is less than a threshold value. For example, the threshold value may be set to be smaller than a distance indicated by depth information (e.g., first depth information (641)) of a virtual area (e.g., a virtual area (580) of FIG. 5) for displaying an external image (e.g., an external image (575) of FIG. 5). The wearable device (101) may display a virtual object when the distance from the wearable device (101) to the external object (520) is less than the threshold value. In general, a virtual environment may provide a wider space than an actual environment. As the external object (520) in the real environment is closer to the user (105), a distortion may occur between the distance perceived in the external image (575) of the virtual environment and the actual distance. To mitigate the effects of the distortion, a virtual object may be displayed.

[0113] According to one embodiment, the wearable device (101) can identify whether a difference between a position of the virtual object and a position of a virtual area (e.g., a virtual area (580) of FIG. 5) for displaying an external image (e.g., an external image (575) of FIG. 5) is greater than or equal to a threshold value. For example, the wearable device (101) can identify whether a difference between an actual distance (642) from the wearable device (101) to the virtual object and a distance (hereinafter, referred to as a virtual distance) to the virtual area (580) is greater than or equal to the threshold value. The wearable device (101) can display a commercial virtual object when the difference between the actual distance from the wearable device (101) to the external object and the virtual distance between the virtual area (580) and the wearable device (101) is greater than or equal to the threshold value. This is because, as the difference between the location of the external object (520) and the location of the virtual area (580) in the actual environment increases, distortion may occur in the user's (105) perception of distance. To mitigate the effects of the distortion, a virtual object may be displayed.

[0114] According to one embodiment, the wearable device (101) can identify whether the angle formed by the normal vector (631) of the virtual area (580) and the gaze direction (632) of the user (105) is greater than or equal to a threshold value. For example, the wearable device (101) can display a virtual object when the angle between the gaze direction (632) and the normal vector (631) is within the threshold value. As the virtual area (580) (pass-through surface) is located in front of the user's (105) field of view, the display ratio within the image corresponding to the FoV (e.g., image (510) of FIG. 5) can increase. As the ratio of the image (e.g., image (575) of FIG. 5) displayed through the virtual area (580) increases, distortion may occur in the user's (105) perception of distance. To mitigate the effect of the distortion, a virtual object may be displayed.

[0115] In one embodiment, the wearable device (101) can identify whether the area of ​​the virtual area (580) is greater than or equal to a threshold value. For example, the wearable device (101) can display a virtual object if the area of ​​the virtual area (580) is greater than or equal to the threshold value. As the virtual area (580) (i.e., the pass-through surface) becomes wider, the proportion of an image (e.g., image (575)) displayed within the screen increases. Accordingly, distortion may occur in the user's (105) perception of distance, and thus, a virtual object may be displayed to mitigate the effect of the distortion.

[0116] The wearable device (101) may determine whether to display a virtual object based on at least one of the conditions described above. According to one embodiment, the wearable device (101) may determine whether to display a virtual object through a combination of the conditions described above. For example, the wearable device (101) may display a virtual object when an angle formed by a normal vector (631) of the virtual area (580) and a gaze direction (632) of the user (105) is less than a first threshold value, and a distance from the wearable device (101) to an external object (e.g., an external object (520) of FIG. 5) is less than a second threshold value. For example, the wearable device (101) may display a virtual object when a distance from the wearable device (101) to an external object (e.g., an external object (520) of FIG. 5) is less than the second threshold value, and an area of ​​the virtual area (580) is greater than or equal to a third threshold value. Furthermore, embodiments of the present disclosure are not limited to the conditions described above. If a condition is used to determine whether a distortion in the user's (105) distance perception has occurred, it can be used to determine whether to display a virtual object. This condition may be used in place of the conditions described above, or may be combined with at least one of the conditions described above.

[0117] Fig. 7 illustrates an example of displaying a virtual object according to depth information. The depth information represents a distance along one direction (e.g., the z-axis direction). A method for displaying an external object (e.g., an external object (520) of Fig. 5) in a virtual space may include a method of displaying an external object (520) photographed according to a pass-through operation through a virtual area (e.g., a virtual area (580) of Fig. 5) or a method of displaying a virtual object (720) corresponding to the external object (520). Fig. 7 illustrates an example of displaying a virtual object corresponding to the external object (520) or displaying an external image (e.g., an external image (575) of Fig. 5) without displaying the virtual object, depending on the position of the external object (520).

[0118] Referring to FIG. 7, the wearable device (101) can obtain a distance from a user (105) of the wearable device (101) to an external object (520). The wearable device (101) can obtain the distance between the wearable device (101) and the external object (520) through at least one sensor (e.g., a distance sensor, a depth sensor, an image sensor, a ToF (time of flight) camera). According to one embodiment, the wearable device (101) can determine whether to display a virtual object corresponding to the external object (520) based on the distance between the wearable device (101) and the external object (520).

[0119] According to one embodiment, the wearable device (101) can identify whether the distance from the wearable device (101) to the external object (520) is less than a threshold value (741). The threshold value (741) may be referred to as second depth information. The threshold value (741) may be set to be smaller than a distance indicated by depth information (e.g., first depth information (641) of FIG. 6) of a virtual area (e.g., virtual area (580) of FIG. 5) for displaying an external image (e.g., external image (575) of FIG. 5). In order to explain the relationship between the display of the virtual object and the distance between the wearable device (101) and the external object (520), an area in the virtual space according to the threshold value (741) may be referred to as a boundary area (780).

[0120] According to one embodiment, the distance between the wearable device (101) and the external object (520) may be less than a threshold value (741). Hereinafter, an area where the actual distance from the wearable device (101) is less than the threshold value may be referred to as a first area (731). When the location of the external object (520) is within the first area (731), the wearable device (101) may display a virtual object (720). The wearable device (101) may acquire an external image (775a) through a camera (e.g., the camera (420) of FIG. 4). While displaying the external image (775a) on the virtual area (580), the wearable device (101) may display the virtual area (580) so as to at least partially overlap the external image (775a).

[0121] According to one embodiment, the distance between the wearable device (101) and the external object (520) may be greater than or equal to the threshold value (741), but may be shorter than the distance according to the first depth information (e.g., the first depth information (641) of FIG. 6) of the virtual area (580). Hereinafter, an area in which the actual distance from the wearable device (101) to the external object (520) is greater than or equal to the threshold value (741) and less than the virtual distance corresponding to the first depth information (641) may be referred to as a second area (732). The wearable device (101) may acquire an external image (775b) through the camera (420). When the location of the external object (520) is within the second area (732), the wearable device (101) may display the external image (775b) through the virtual area (580) instead of displaying the virtual object (720).

[0122] According to one embodiment, the distance between the wearable device (101) and the external object (520) may be longer than the distance according to the first depth information (e.g., the first depth information (641) of FIG. 6) of the virtual area (580). Hereinafter, an area in which the actual distance from the wearable device (101) is greater than or equal to the virtual distance corresponding to the first depth information (641) may be referred to as a third area (733). The wearable device (101) may acquire an external image (775c) through the camera (420). When the location of the external object (520) is within the third area (733), the wearable device (101) may display the external image (775c) through the virtual area (580) instead of displaying the virtual object (720). Even if the actual distance from the wearable device (101) is closer or farther than the virtual area (580), the wearable device (101) can directly display the external object (520) instead of the virtual object (720), with only a difference in perspective. This is because the virtual object (720) is located at a distance farther than the virtual area (580) for pass-through operation, so the possibility of the perspective being reduced due to the external object (520) is somewhat low.

[0123] FIG. 8 illustrates an example of displaying an external image (e.g., an external image (575) of FIG. 5) and a virtual object (e.g., a virtual object (720) of FIG. 7).

[0124] Referring to FIG. 8, a wearable device (101) can provide a virtual environment to a user (105). The wearable device (101) can generate a virtual image (570) for the virtual environment. The wearable device (101) can display the virtual image (570) through a display (e.g., the display (420) of FIG. 4). The wearable device (101) can perform a pass-through operation. For example, the wearable device (101) can operate a camera (425) based on a user input. The wearable device (101) can obtain an image (e.g., the image (510) of FIG. 5) through the camera (425). Through the pass-through operation, the wearable device (101) can display a screen (560) through the display (420). The screen (560) may include a virtual image (570) representing a virtual environment. The screen (560) may include a virtual area (580) for representing at least a portion of a real environment. For example, the virtual area (580) may include at least a portion of an image (510) acquired through the camera (425) (e.g., an external image (575) of FIG. 5 ). The virtual area (580) represents a space in which a real image captured by the camera (425) is displayed on a virtual image (570) corresponding to a virtual space through a VST function. The wearable device (101) may display an external image (575) within the virtual area (580). The external image (575) represents at least a portion of the image (510). The external image (575) may include an external object (520).

[0125] According to one embodiment, the wearable device (101) may identify whether a distance from the wearable device (101) to an external object (520) is less than a threshold value (e.g., threshold value (741) of FIG. 7). For example, if the distance between the wearable device (101) and the external object (520) is less than the threshold value (741), the wearable device (101) may display a virtual object (720). For example, the wearable device (101) may display a screen (860) through the display (420) through the pass-through operation. The screen (860) may include a virtual image (570) representing a virtual environment, and a virtual object (720) displayed by overlapping at least a portion of a virtual area (580). The wearable device (101) can display a virtual object (720) while displaying a virtual image (570) and an external image (575). The wearable device (101) can display the virtual object (720) while displaying the external image (575) within a virtual area (580). The wearable device (101) can display the virtual object (720) based on the direction in which the virtual area (580) for the external image (575) is positioned. The virtual object (720) can overlap at least partially with the external image (575). The wearable device (101) can display the virtual object (720) so as to overlap at least partially with a portion in which the external object (520) is displayed within the external image (575).

[0126] In FIG. 8, a condition in which the distance between the external object (520) and the wearable device (101) is less than a threshold value (741) is described as an example as a condition for displaying a virtual object (720), but embodiments of the present disclosure are not limited thereto. According to one embodiment, the wearable device (101) may display the virtual object (720) based on a condition in which a difference between the position of the external object (520) and the position of a boundary surface (e.g., the virtual area (580) of FIG. 5) for a pass-through operation is greater than or equal to the threshold value. According to one embodiment, the wearable device (101) may display the virtual object (720) based on a condition in which the distance between the external object (520) and the wearable device (101) is less than the threshold value (741) and the area of ​​the virtual area (580) is greater than or equal to the size threshold value. According to one embodiment, a virtual object (720) may be displayed based on a condition that the distance between an external object (520) and a wearable device (101) is less than a threshold value (741) and the depth information of a virtual area (580) (e.g., the first depth information (641) of FIG. 6) is greater than or equal to the distance threshold value.

[0127] FIG. 9 illustrates an example of an external image (e.g., an external image (575) of FIG. 5) and a display of a virtual object according to movement of the virtual object (e.g., a virtual object (720) of FIG. 7). The virtual object (720) can move. For example, the virtual object (720) may be a body part (e.g., a hand) of a user (105) of a wearable device (101), or a controller connected to the wearable device (101).

[0128] In FIG. 9, the wearable device (101) can display a screen (910) through a display (e.g., the display (420) of FIG. 4). The screen (910) can include a virtual image (970a) and a virtual area (580). The virtual area (580) can display an external image (975a) captured according to a pass-through operation of the wearable device (101). The external image (975a) can include an external object (520). For example, the external object (520) can include a hand of a user (105). The wearable device (101) can obtain a distance between the wearable device (101) and the external object (520). For example, the user (105) can move his / her hand closer to the head of the user (105) on whom the wearable device (101) is worn. As the hand moves, the external object (520) may come closer to the wearable device (101). The distance between the wearable device (101) and the external object (520) may be less than a specified threshold (e.g., the threshold value (741) of FIG. 7). To indicate the closer external object (520), the wearable device (101) may display a virtual object (720) corresponding to the external object (520) in the virtual environment. The screen (910) may include the virtual object (720). The wearable device (101) may display the virtual object (720) while displaying the virtual image (970a) and the external image (975b).

[0129] In one embodiment, the screen (920) may include a virtual image (970b) and a virtual area (580). The virtual area (580) may display an external image (975b) captured according to a pass-through operation of the wearable device (101). The external image (975b) may include an external object (520). For example, the external object (520) may include a hand of the user (105). The wearable device (101) may obtain a distance between the wearable device (101) and the external object (520). For example, the user (105) may move his / her hand closer to a keyboard on a desk. As the hand moves, the external object (520) may move away from the wearable device (101). The distance between the wearable device (101) and the external object (520) may be greater than or equal to the specified threshold. Since it is sufficient to display a more distant external object (520) through the virtual area (580), the wearable device (101) can stop displaying the virtual object (720). At this time, the screen (920) may not include the virtual object (720). The wearable device (101) can display the virtual image (970b) and the external image (975b) without the virtual object (720).

[0130] According to one embodiment, the screen (930) may include a virtual image (970c) and a virtual area (580). The virtual area (580) may display an external image (975c) captured according to a pass-through operation of the wearable device (101). The external image (975c) may include an external object (520). For example, the external object (520) may include a hand of the user (105). The wearable device (101) may obtain a distance between the wearable device (101) and the external object (520). For example, the user (105) may move his hand closer to the user (105) than the keyboard. As the hand moves, the external object (520) may come closer to the wearable device (101). However, the distance between the wearable device (101) and the external object (520) may be greater than the specified threshold. Even if the wearable device (101) gets closer to the user (105), if the wearable device (101) does not get closer by a predetermined threshold value, the wearable device (101) does not need to display the virtual object (720). The wearable device (101) can continuously display the virtual image (970a) and the external image (975b). The screen (930) may not include the virtual object (720). The wearable device (101) can display the virtual image (970c) and the external image (975c) without the virtual object (720).

[0131] FIG. 10 illustrates an example of displaying a virtual object (e.g., the virtual object (720) of FIG. 7) based on movement of the virtual object. The wearable device (101) may display the virtual object (720) differently based on the distance between the virtual object (720) and the user (105).

[0132] Referring to FIG. 10, the wearable device (101) can display a virtual object (720) corresponding to the external object (520) based on the distance between the wearable device (101) and the external object (520). A situation is described where the external object (520) is gradually moving away from the user (105).

[0133] For example, an external object (520) may be located at a first location (1051). The wearable device (101) may display a virtual object (720a) corresponding to the external object (520) because the distance between the wearable device (101) and the external object (520) is less than a threshold value (741).

[0134] For example, the external object (520) may be located at a second location (1053). The wearable device (101) may display a virtual object (720b) ​​corresponding to the external object (520) because the distance between the wearable device (101) and the external object (520) is less than a threshold value (741). In one embodiment, the wearable device (101) may display the virtual object (720b) ​​differently from the virtual object (720a) at the first location (1051) as the second location (1053) of the external object (520) becomes further from the first location (1051) of the external object (520). For example, the wearable device (101) may display the virtual object (720b) ​​to have different properties (e.g., color, transparency, opacity, resolution, saturation) than the properties of the virtual object (720a). For example, the opacity of the virtual object (720b) ​​may be lower than the opacity of the virtual object (720a). For example, the resolution of the virtual object (720b) ​​may be lower than the resolution of the virtual object (720a). For example, the size of the virtual object (720b) ​​may be smaller than the size of the virtual object (720a). For example, the saturation of the virtual object (720b) ​​may be lower than the saturation of the virtual object (720a).

[0135] For example, the external object (520) may be located at the third location (1055). A portion of the external object (520) may have a distance from the wearable device (101) less than the threshold value (741), and another portion of the external object (520) may have a distance from the wearable device (101) greater than or equal to the threshold value (741). In one embodiment, the external object (520) may display a virtual object (720c). For example, if at least a portion of the external object (520) has a distance from the wearable device (101) less than the threshold value (741), the wearable device (101) may display a virtual object (720c) corresponding to the entire external object (520). For another example, if a part of an external object (520) is less than a threshold value (741) in distance from the wearable device (101), the wearable device (101) may display a virtual object (720c) corresponding to an area in which the distance from the wearable device (101) is less than the threshold value (741). For another example, if a part of an external object (520) is less than a threshold value (741) in distance from the wearable device (101), the wearable device (101) may display a virtual object (720c) with different display properties to indicate that only a part of the external object (520) overlaps. For example, the opacity of the virtual object (720c) may be lower than the opacity of the virtual object (720b). For example, the resolution of the virtual object (720c) may be lower than the resolution of the virtual object (720b). For example, the size of the virtual object (720c) may be smaller than the size of the virtual object (720b). For example, the saturation of the virtual object (720c) may be lower than the saturation of the virtual object (720b).

[0136] For example, the external object (520) may be located at the fourth location (1057). The wearable device (101) may not display a virtual object (720) corresponding to the external object (520) because the distance between the wearable device (101) and the external object (520) is greater than or equal to a threshold value (741).

[0137] In FIGS. 5 to 9, examples of determining whether to display a virtual object corresponding to an external object (520) based on the distance between the wearable device (101) and the external object (520) are described. When the external object (520) is located between the virtual area (580) and the user (105) (or the wearable device (101)), the wearable device (101) is required to obtain the distance to the external object (520) because the external object (520) affects the user's (105) sense of perspective. Meanwhile, even if the external objects are spaced apart from the wearable device (101) by the same distance, the effects they have on the user's (105) sense of perspective may be different. For example, assume a first external object and a second external object. The size of the first external object may be larger than the size of the second external object. The first external object may occupy a larger space in the user's (105) field of view than the second external object. Accordingly, the first external object may have a greater influence on the sense of depth than the second external object. To represent this difference in influence, the wearable device (101) according to embodiments of the present disclosure may set a weight value to the external object. The wearable device (101) may determine whether to display a virtual object based on whether the weight value of the external object exceeds a threshold value. If the weight value of the external object exceeds the threshold value, the wearable device (101) may display the virtual object.

[0138] In one embodiment, the wearable device (101) may set a weight value for the external object (520) based on a distance between the wearable device (101) and the external object (520). As the external object (520) gets closer to the wearable device (101), the weight of the external object (520) may be set higher. As the external object (520) gets farther from the wearable device (101), the weight of the external object (520) may be set lower.

[0139] In one embodiment, the wearable device (101) may set a weight value for the external object (520) based on the size of the external object (520). As the size of the external object (520) increases, the weight of the external object (520) may be set higher. As the size of the external object (520) decreases, the weight of the external object (520) may be set lower.

[0140] According to one embodiment, the wearable device (101) may set a weight value for the external object (520) based on the speed information of the external object (520). The greater the speed information of the external object (520), the higher the weight of the external object (520) may be set. In other words, the faster the external object (520) moves, the higher the weight of the external object (520) may be set. The slower the speed of the external object (520), the lower the weight of the external object (520) may be set.

[0141] In one embodiment, the wearable device (101) may set a weight value for the external object (520) based on color information of the external object (520). For example, the more components of a specified color (e.g., red) the external object (520) contains, the higher the weight of the external object (520) may be set. For example, the higher the color saturation of the external object (520) is, the higher the weight of the external object (520) may be set.

[0142] According to one embodiment, the wearable device (101) may set a weight value for an external object (520) based on a direction of a gaze of a user (105). The wearable device (101) may obtain the direction of the gaze of the user (105). The wearable device (101) may obtain information about a distance at which the external object (520) is located based on the direction of the gaze. The wearable device (101) may set a weight value for the external object (520) based on the information. For example, the closer the external object (520) is to the direction of the gaze, the higher the weight value may be set. For example, the farther the external object (520) is from the direction of the gaze, the higher the weight value of the external object (520) may be set.

[0143] In order to determine the visual impact of an external object on a user (105), the wearable device (101) may determine a weight for the external object based on at least one of the conditions described above. According to one embodiment, the wearable device (101) may determine a weight for the external object through a combination of the conditions described above. For example, the wearable device (101) may set a weight value for the external object (520) based on color information and speed information. For example, the wearable device (101) may set a weight value for the external object (520) based on a distance between the external object (520) and the wearable device (101) and speed information of the external object (520). For example, a weight value for a distant and stationary external object (520) may be set to be smaller than a weight value for a moving external object (520). Embodiments of the present disclosure are not limited to the conditions described above. If there is a condition that can affect the user's (105) vision, the condition may be used to determine the weight value. The condition may be used in place of the conditions described above, or may be combined with at least one of the conditions described above.

[0144] Figures 11a and 11b illustrate examples of pass-through surfaces. The wearable device (101) can perform a pass-through operation. For example, the wearable device (101) can operate a camera (e.g., camera (425) of Figure 4) based on user input. The wearable device (101) can acquire an image through the camera (425). The wearable device (101) can include a virtual area (580) for representing at least a portion of the image. The virtual area (580) can be referred to as a pass-through surface.

[0145] Referring to FIG. 11A, according to one embodiment, a virtual area (580) can be used to transition between a virtual environment (e.g., virtual environment (1100)) and a real environment (e.g., real environment (1150)). The wearable device (101) can display an image representing at least a portion of the real environment (1050) through the display (420) via the virtual area (580). A user (105) of the wearable device (101) can move. For example, the user (105) of the wearable device (101) can move closer to the virtual area (580). The user (105) of the wearable device (101) can pass through the virtual area (580). When the wearable device (101) detects that the user (105) moves across an area corresponding to a virtual area (580) in a virtual space, the wearable device (101) can acquire an image corresponding to the real environment (1150) through the camera (425). The image corresponding to the real environment (1150) can be displayed through the display (420). While the wearable device (101) displays the image corresponding to the real environment (1150), the wearable device (101) can display an image representing at least a part of the virtual environment (1100) through the virtual area (580).

[0146] Referring to FIG. 11B, according to one embodiment, a virtual area (580) may be formed in the form of a stencil within a virtual environment. For example, a user (105) of a wearable device (101) may move around the periphery of the virtual area (580). The wearable device (101) may move counterclockwise around the virtual area (580). The wearable device (101) may display a screen corresponding to the location of the user (105). For example, the wearable device (101) may sequentially display a first screen (1161), a second screen (1162), and a third screen (1163). As displayed through the first screen (1161), the second screen (1162), and the third screen (1163), the virtual area (580) may be positioned at a fixed location within the virtual environment. The shape of the virtual area (580) may be a disk.

[0147] FIG. 12 illustrates an operation flow of a wearable device (e.g., the wearable device (101) of FIG. 1) for displaying a virtual object on an external image (e.g., the external image (575) of FIG. 5).

[0148] Referring to FIG. 12, in operation (1201), the wearable device (101) can acquire an external image (e.g., an external image (575) of FIG. 5) of the wearable device (101) while displaying a virtual image of a virtual environment (e.g., a virtual image (570) of FIG. 5). The external image (575) can represent at least a portion of a surrounding real environment. The wearable device (101) can perform a pass-through operation. The wearable device (101) can provide a virtual environment. The wearable device (101) can drive a camera (e.g., a camera (425) of FIG. 4) to display an image corresponding to the real environment while the virtual image (570) is displayed. The wearable device (101) can acquire the external image (575) by photographing at least a portion of the real environment through the camera (425).

[0149] In operation (1203), the wearable device (101) can display at least a part of an external image (575) in a virtual area (e.g., a virtual area (580) of FIG. 5) according to first depth information (e.g., the first depth information (641) of FIG. 6). The first depth information (641) can indicate a distance from a user (105) of the wearable device (101) to the virtual area (580). The virtual area (580) can indicate a space in which an external image (575), which is a real image captured by a camera (425), is displayed on a virtual image (570) corresponding to a virtual space through a VST function of the wearable device (101). The wearable device (101) can display a virtual area (580) for a pass-through operation at a location corresponding to the first depth information (641). The wearable device (101) can display at least a portion of an external image (575) so that at least a portion of a real environment is exposed within the virtual area (580) when the user looks at the virtual area (580). The wearable device (101) can display the external image (575) through the virtual area (580) while the virtual image (570) is displayed.

[0150] In operation (1205), the wearable device (101) can obtain a distance to an external object (e.g., an external object (520) of FIG. 5). For example, the external image (575) can include the external object (520). The wearable device (101) can obtain the distance between the external object (520) and the wearable device (101) through at least one sensor (e.g., a sensor (430) of FIG. 4). For example, the at least one sensor can include a depth sensor and / or a time of flight (ToF) sensor.

[0151] In operation (1207), the wearable device (101) may display a virtual object (e.g., the virtual object (720) of FIG. 7) corresponding to the external object (520) while displaying at least a portion of the virtual image (570) and the external image (575) based on whether a location corresponding to a distance from the external object (520) is within a space according to second depth information (e.g., the threshold value (741) of FIG. 7). According to one embodiment, the wearable device (101) may determine a distance according to second depth information. The second depth information may indicate a depth (e.g., a z-index) of an area that serves as a reference for determining whether to display the virtual object (720). The area may be referred to as a border area (780). The wearable device (101) can determine whether the distance between the wearable device (101) and the external object (520) is closer than the distance indicated by the threshold value (741) according to the second depth information. If the distance between the wearable device (101) and the external object (520) is shorter than the distance indicated by the threshold value (741), the wearable device (101) can determine that the location of the external object (520) is within the space according to the second depth information (e.g., the threshold value (741) of FIG. 7). In order to alleviate distortion in the distance perception of the user (105), the wearable device (101) can display a virtual object (720) corresponding to the external object (520). Meanwhile, as shown in FIGS. 13 to 15 described below, even if the boundary area is a three-dimensional figure having volume in a three-dimensional space, the wearable device (101) can set the boundary surface of the three-dimensional figure (e.g., which can be displayed in a three-dimensional space according to the surf() function) as the boundary area. For example, the second depth information can indicate the distance between each point of the boundary surface of the three-dimensional figure and the wearable device (101).The wearable device (101) can determine the distance between the boundary surface of the three-dimensional figure and the wearable device (101) based on the direction of the external object (520) from the wearable device (101). If the distance between the wearable device (101) and the external object (520) is shorter than the distance between the boundary surface of the three-dimensional figure and the wearable device (101), the wearable device (101) can determine that the location of the external object (520) is within the space according to the second depth information. The wearable device (101) can display the virtual object (720) while displaying at least a portion of the virtual image (570) and the external image (575). The virtual object (720) can be displayed at least partially overlapping the external image (575). For example, the wearable device (101) may display the virtual object (720) so that the virtual object (720) overlaps an area of ​​an external object (520) of an external image (575).

[0152] In FIGS. 5 to 13, a virtual area (580) corresponding to the first depth information (641) and a boundary area (780) corresponding to a threshold value (741) are defined to display a virtual object corresponding to an external object. Areas (e.g., the first area (731), the second area (732), and the third area (733) of FIG. 7) can be classified based on the distance between the external object (520) and the wearable device (101), the virtual area (580), and the boundary area (780). For example, the areas can be distinguished based on how far they are from the user (105). Meanwhile, in FIG. 7, the boundary area (780) corresponding to the threshold value (741) is illustrated as being a plane (e.g., an xy plane) perpendicular to one direction (e.g., a z-axis), but embodiments of the present disclosure are not limited thereto. The boundary area (780) that serves as a reference for determining whether to display a virtual object (720) may be defined as a fixed space instead of simply a surface. In the present disclosure, the boundary area (e.g., the boundary area (780) of FIG. 7, the boundary area (1315) of FIG. 13) may be referred to as a display area, a display boundary area, a display surface, a boundary surface, a display space, a boundary space, a boundary surface, an object display boundary, or terms having equivalent technical meanings thereto, in addition to the boundary area. Hereinafter, virtual areas occupying space in a virtual space are described in FIGS. 13 to 15.

[0153] Fig. 13 illustrates an example of a boundary area for displaying a virtual object (e.g., a virtual object (720) of Fig. 7). A method for displaying an external object (e.g., an external object (520) of Fig. 5) in a virtual space may include a method of displaying an external object (520) captured according to a pass-through operation through a virtual area (e.g., a virtual area (580) of Fig. 5) or a method of displaying a virtual object corresponding to the external object (520). Fig. 13 describes an example of displaying a virtual object corresponding to the external object (520) or displaying an external image (e.g., an external image (575) of Fig. 5) without displaying the virtual object, depending on the position of the external object (520).

[0154] Referring to FIG. 13, the wearable device (101) can obtain the location of an external object (520). Based on the location of the external object (520), the wearable device (101) can determine whether to display a virtual object (1320) corresponding to the external object (520). For the description of the virtual object (1320), a virtual object (e.g., 720 of FIG. 7) may be referenced. Even if the boundary area is a three-dimensional figure having a volume in a three-dimensional space, the wearable device (101) can set the boundary surface of the three-dimensional figure as the boundary area (1315). The distance between each point of the boundary area (1315) and the wearable device (101) can be referenced as second depth information. For example, the wearable device (101) may determine a virtual distance between the boundary surface of the three-dimensional figure and the wearable device (101) based on the direction of the external object (520) from the wearable device (101). If the actual distance between the wearable device (101) and the external object (520) is shorter than the virtual distance, the wearable device (101) may determine that the external object (520) is located within the boundary area (1315). The external object (520) located within the boundary area (1315) may be within the space according to the second depth information. The wearable device (101) may set a three-dimensional boundary area (1315) based on the virtual area (580) as a boundary for distinguishing between a real environment and a virtual environment. The wearable device (101) can display a virtual object (1320) corresponding to an external object (520) located within a boundary area (1315). The wearable device (101) can display an external image including the external object (520) for an external object (520) located outside the boundary area (1315).

[0155] The wearable device (101) can set a boundary area (1315) having a volume in a three-dimensional space (1313). A space (1313) can be formed between the boundary area (1315) and a virtual area (580). For example, if the location of an external object (520) is included in the space (1313), the wearable device (101) can display an external image through the space (1313) and / or the virtual area (580) instead of displaying a virtual object (1320) corresponding to the external object (520). For example, if the location of the external object (520) is outside the space (1313), the wearable device (101) can display a virtual object (1320) corresponding to the external object (520). The wearable device (101) can display a virtual object (1320) in an external space of a three-dimensional figure occupied by the boundary area (1315) and the virtual area (580). For example, even if the external object (520) exists at a position (e.g., the first position (1321), the third position (1325)) having a value greater than the z-axis value of a portion of the boundary area (1315), the wearable device (101) can display the virtual object (1320) while displaying an external image. For example, since the first position (1321) or the third position (1325) is outside the space (1313), the virtual object (1320) corresponding to the external object (520) can be displayed. For example, if the location of the external object (520) (e.g., the second location (1323)) is closer to the user (105) than the location of a portion of the boundary area (1315), the wearable device (101) may display the virtual object (1320) while displaying the external image.

[0156] FIG. 14 illustrates an example of a border area (e.g., border area (1315) of FIG. 13) for displaying a virtual object and a virtual environment. Through the border area (1315) and the virtual area (e.g., virtual area (580) of FIG. 5), a virtual space (1313) can be formed.

[0157] Referring to FIG. 14, the wearable device (101) can display a screen (1400) through a display (e.g., the display (420) of FIG. 4). The screen (1400) can include a virtual image (570) and an external image (575). The external image (575) can be displayed through a virtual space (1313). The wearable device (101) can display the virtual space (1313) while displaying the virtual image (570). The wearable device (101) can display the external image (575) through the virtual space (1313). The external image can include an external object (520). For example, the wearable device (101) can obtain information about a distance to an external object (520) and information about a direction to the external object (520). The wearable device (101) can determine the location of the external object (520) based on information about the distance to the external object (520) and information about the direction to the external object (520). The wearable device (101) can determine the location of the external object (520) within the virtual environment based on the user (105). For example, if the determined location of the external object (520) is outside the virtual space (1313), the wearable device (101) can display a virtual object (1320) corresponding to the external object (520). Meanwhile, unlike as shown in FIG. 13, if the location of the determined external object (520) is within the virtual space (1313), the wearable device (101) may display the external object (520) without a virtual object (1320) corresponding to the external object (520).

[0158] The wearable device (101) can display a screen (1450) through the display (420). The screen (1450) can include a virtual image (570), an external image (575), and a virtual object (1320). The external image (575) can be displayed through a virtual space (1313). The wearable device (101) can display the virtual space (1313) while displaying the virtual image (570). The wearable device (101) can display the external image (575) through the virtual space (1313). The external image can include an external object (520). For example, the wearable device (101) can determine the location of the external object (520) based on information about the distance to the external object (520) and information about the direction to the external object (520). The wearable device (101) can determine the location of an external object (520) within a virtual environment based on the user (105). For example, if the determined location of the external object (520) is outside the virtual space (1313), the wearable device (101) can display a virtual object (1320) corresponding to the external object (520).

[0159] Although one side of the virtual space (1313) is depicted in FIG. 14 as corresponding to the virtual area (580), embodiments of the present disclosure are not limited thereto. The virtual space (1313) may be formed solely by the space covered by the boundary area (1315), without a side such as the virtual area (580).

[0160] Figure 15 illustrates an example (1500) of a boundary area of ​​a curved surface.

[0161] Referring to FIG. 15, the wearable device (101) may include a curved boundary area (1315). According to one embodiment, the wearable device (101) may set the boundary area (1315) forming the curved surface within a three-dimensional space of a virtual environment. If the location of the external object (e.g., the external object (520)) is closer to the user (101) than the boundary area (1315), the wearable device (101) may display a virtual object (e.g., the virtual object (1320)) corresponding to the external object (520). For example, the wearable device (101) may display a virtual object corresponding to a hand as an overlay on an external image.

[0162] The distance between the user (105) and the wearable device (101) and the boundary area (1315) to be compared may be determined based on various criteria. In one embodiment, the boundary area (1315) may be determined based on the distance between the wearable device (101) and the virtual area (580) in the virtual space. For example, the closer the distance between the wearable device (101) and the virtual area (580), the narrower the boundary area (1315) may be formed. This is because the narrower the area of ​​the boundary area (1315), the higher the likelihood that a virtual object will be displayed. In one embodiment, the boundary area (1315) may be determined based on the distance between the wearable device (101) and the virtual area (580) in the real environment. For example, the closer the distance between the wearable device (101) and the virtual area (580), the narrower the boundary area (1315) may be formed. In one embodiment, the boundary area (1315) may be determined based on the angle formed by the normal vector (1531) of the virtual area (580) with the gaze direction (1532) of the user (105). For example, the narrower the angle, the narrower the boundary area (1315) may be formed. In one embodiment, the boundary area (1315) may be determined based on a threshold value of a weight set to the external object (520). In one embodiment, the boundary area (1315) may be determined based on a value specified according to a user input (e.g., a custom variable). In addition, as a non-limiting example, the boundary area (1315) may be determined through a combination of the above-described elements.

[0163] In embodiments, an electronic device (101) is provided. The electronic device (101) may include a display (420), at least one sensor (430), a memory (415) for storing instructions, and at least one processor (410). The instructions, when executed by the at least one processor (410), may cause the electronic device (101) to acquire an external image of the electronic device (101) while displaying a virtual image of a virtual environment through the display (420), display at least a portion of the external image through the display (420) in a virtual area according to first depth information in the virtual environment while displaying the virtual image, acquire a distance to an external object included in at least a portion of the external image through the at least one sensor (430), and display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image based on a position corresponding to the distance to the external object being within a space according to second depth information.

[0164] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to identify whether a distance corresponding to the first depth information is greater than or equal to a first threshold value, and if the distance corresponding to the first depth information is greater than or equal to the first threshold value, to display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image.

[0165] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to identify whether a normal vector for the virtual area is within a threshold distance from a viewing direction, and if the normal vector for the virtual area is within the threshold distance from the viewing direction, display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image.

[0166] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to identify whether an area of ​​the virtual area is greater than or equal to a second threshold value, and if the area of ​​the virtual area is greater than or equal to the second threshold value, to display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image.

[0167] In one embodiment, the first depth information may indicate a distance between the reference position and the virtual area in the virtual environment. The second depth information may indicate a distance between the reference position and a display boundary area in the virtual environment. The display boundary area may be used to identify whether to display the virtual object.

[0168] In one embodiment, the space between the virtual area and the display boundary area may represent a three-dimensional volume in the virtual environment. The second depth information for the display boundary area may represent a plurality of depth values ​​from the reference position.

[0169] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to obtain a weight of the external object based on at least one of a size of the external object, a distance to the external object, speed information of the external object, or color information of the external object, and, if the weight of the external object is equal to or greater than the object threshold value, display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image, and, if the weight of the external object is less than the object threshold value, display the virtual image and the external image through the display (420) without displaying the virtual object.

[0170] In one embodiment, when the size of the external object is a first size, the weight may be a first value. When the size of the external object is a second size that is greater than the first size, the weight may be a second value. The second value may be greater than the first value.

[0171] In one embodiment, when the velocity information of the external object is a first velocity value, the weight may be a first value. When the velocity information of the external object is a second velocity value greater than the first velocity value, the weight may be a second value. The second value may be greater than the first value.

[0172] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to display the virtual image and the external image through the display (420) without displaying the virtual object when the distance to the external object is outside the second distance.

[0173] According to one embodiment, the virtual object may be displayed superimposed on the external image according to depth information determined based on the distance from the external object.

[0174] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to obtain a plurality of locations corresponding to the external object in the virtual environment, generate a second virtual object corresponding to the external object if at least some of the plurality of locations corresponding to the external object are locations corresponding to second depth information, and display the generated second virtual object through the display (420) while displaying the virtual image and at least a portion of the external image. An opacity of the second virtual object may be greater than an opacity of the virtual object.

[0175] According to one embodiment, the at least one sensor (430) may include a depth sensor.

[0176] In one embodiment, the second depth information may indicate a distance closer than the distance corresponding to the first depth information from the reference position of the virtual environment.

[0177] In one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to stop displaying the virtual object if the location information of the external object is outside the space according to the second depth information and is located between the first depth information and the second depth information.

[0178] In embodiments, an electronic device (101) is provided. The electronic device (101) may include a display (420), at least one sensor (430), a memory (415) for storing instructions, and at least one processor (410). The instructions, when executed by the at least one processor (410), may cause the electronic device (101) to acquire an external image of the electronic device (101) while displaying a virtual image of a virtual environment through the display (420), display at least a portion of the external image through the display (420) in a virtual area according to first depth information in the virtual environment while displaying the virtual image, acquire location information of the external object based on a distance from the external object acquired through the at least one sensor (430), and, when the location information of the external object is within a space according to second depth information, display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image, and stop displaying the virtual object when the location information of the external object is outside the space according to the second depth information and is located between the first depth information and the second depth information. The second depth information may indicate a distance closer than the distance corresponding to the first depth information from the reference position of the virtual environment.

[0179] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to obtain velocity information of the external object through the at least one sensor (430), and to generate a virtual object corresponding to the external object based on the velocity information of the external object being less than a velocity threshold value.

[0180] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to identify whether a distance corresponding to the first depth information is greater than or equal to a first threshold, and, if the distance corresponding to the first depth information is greater than or equal to the first threshold, to generate a virtual object corresponding to the external object while displaying the virtual image and at least a portion of the external image.

[0181] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to identify whether a normal vector for the virtual area is within a threshold distance from a viewing direction, and if the normal vector for the virtual area is within the threshold distance from a viewing direction, generate a virtual object corresponding to the external object while displaying the virtual image and at least a portion of the external image.

[0182] According to one embodiment, the instructions, when executed by the at least one processor (410), may cause the electronic device (101) to identify whether an area of ​​the virtual area is greater than or equal to a second threshold, and, if the area of ​​the virtual area is greater than or equal to the second threshold, to generate a virtual object corresponding to the external object while displaying the virtual image and at least a portion of the external image.

[0183] In one embodiment, the first depth information may indicate a distance between the reference position and the virtual area in the virtual environment. The second depth information may indicate a distance between the reference position and a display boundary area in the virtual environment. The display boundary area may be used to identify whether to display the virtual object.

[0184] In embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium may include a memory including instructions. The instructions, when executed by at least one processor (410), may cause the electronic device (101) to acquire an external image of the electronic device (101) while displaying a virtual image of a virtual environment through a display (420), display at least a portion of the external image through the display (420) in a virtual area according to first depth information in the virtual environment while displaying the virtual image, acquire a distance to an external object included in at least a portion of the external image through at least one sensor (430), and display a virtual object corresponding to the external object through the display (420) while displaying the virtual image and at least a portion of the external image based on a location corresponding to the distance to the external object being within a space according to second depth information.

[0185] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0186] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0187] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0188] Various embodiments of the present document may be implemented as software including one or more instructions stored on a machine-readable storage medium. For example, a processor of the machine may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0189] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0190] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; At least one sensor; Memory for storing instructions; and comprising at least one processor, The above instructions, when executed by the at least one processor, cause the electronic device to: While displaying a virtual image of a virtual environment through the above display, an image outside the electronic device is acquired, While displaying the above virtual image, displaying at least a part of the external image through the display in a virtual area according to the first depth information in the virtual environment, Obtaining a distance to an external object included in at least a part of the external image through at least one sensor; Causing a virtual object corresponding to the external object to be displayed through the display while displaying the virtual image and at least a part of the external image, based on the location corresponding to the distance from the external object being within the space according to the second depth information. Electronic devices.

2. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: Identify whether the distance corresponding to the first depth information is greater than or equal to a first threshold value, If the distance corresponding to the first depth information is greater than or equal to the first threshold value, causing a virtual object corresponding to the external object to be displayed through the display while displaying at least a portion of the virtual image and the external image. Electronic devices.

3. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: Identify whether the normal vector for the above virtual area is within a critical distance from the line of sight, If the normal vector for the virtual area is within a threshold distance from the viewing direction, causing a virtual object corresponding to the external object to be displayed through the display while displaying at least a part of the virtual image and the external image. Electronic devices.

4. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: Identify whether the area of ​​the above virtual area is greater than or equal to a second threshold value, If the area of ​​the virtual area is greater than or equal to a second threshold value, causing a virtual object corresponding to the external object to be displayed through the display while displaying at least a portion of the virtual image and the external image. Electronic devices.

5. In claim 1, The first depth information represents the distance between the reference position and the virtual area in the virtual environment, The second depth information represents the distance between the reference position and the display boundary area in the virtual environment, The above display boundary area is used to identify whether the virtual object is to be displayed. Electronic devices.

6. In claim 5, The space between the above virtual area and the display boundary area represents a three-dimensional volume in the virtual environment, The second depth information for the above display boundary area represents a plurality of depth values ​​from the reference position. Electronic devices.

7. In claim 1, The above instructions, when executed by the at least one processor, cause the electronic device to: Obtaining a weight of the external object based on at least one of the size of the external object, the distance to the external object, the speed information of the external object, or the color information of the external object, If the weight of the external object is greater than or equal to the object threshold value, a virtual object corresponding to the external object is displayed through the display while displaying the virtual image and at least a portion of the external image, If the weight of the external object is less than the object threshold value, causing the virtual image and the external image to be displayed through the display without displaying the virtual object. Electronic devices.

8. In claim 7, When the size of the above external object is the first size, the weight is the first value, When the size of the external object is a second size greater than the first size, the weight is a second value, The second value is greater than the first value, Electronic devices.

9. In claim 7, When the velocity information of the above external object is the first velocity value, the weight is the first value, When the velocity information of the external object is a second velocity value greater than the first velocity value, the weight is the second value, The second value is greater than the first value, Electronic devices.

10. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: If the distance to the external object is outside the second distance, causing the virtual image and the external image to be displayed through the display without displaying the virtual object, Electronic devices.

11. In claim 1, The above virtual object is displayed superimposed on the external image according to depth information determined based on the distance from the external object. Electronic devices.

12. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: Obtaining multiple locations corresponding to the external object in the above virtual environment, If at least some of the plurality of positions corresponding to the external object are positions corresponding to the second depth information, a second virtual object corresponding to the external object is generated, Causing the generated second virtual object to be displayed through the display while displaying at least a portion of the virtual image and the external image; The opacity of the second virtual object is greater than the opacity of the virtual object. Electronic devices.

13. In claim 1, The second depth information indicates a distance from the reference position of the virtual environment that is closer than the distance corresponding to the first depth information. Electronic devices.

14. In claim 1, the instructions, when executed by the at least one processor, cause the electronic device to: If the position information of the external object is outside the space according to the second depth information and is located between the first depth information and the second depth information, causing the display of the virtual object to be stopped, Electronic devices.

15. In a non-transitory computer-readable medium, the non-transitory computer-readable medium includes a memory including instructions, The above instructions, when executed by at least one processor, cause the electronic device to: While displaying a virtual image of a virtual environment through a display, an external image of the electronic device is acquired, While displaying the above virtual image, displaying at least a part of the external image through the display in a virtual area according to the first depth information in the virtual environment, Obtaining a distance to an external object included in at least a part of said external image through at least one sensor, Causing a virtual object corresponding to the external object to be displayed through the display while displaying the virtual image and at least a part of the external image, based on the location corresponding to the distance from the external object being within the space according to the second depth information. Non-transitory computer-readable medium.

Images