Electronic device, method, and computer-readable medium for displaying virtual object

Abstract

The wearable device may include: a camera; at least one sensor; a display; a memory for storing instructions; and a processor, wherein the instructions, when executed by the processor, cause the wearable device to: obtain information on a boundary area for guiding a boundary within which a virtual environment is provided; obtain a position of the wearable device with respect to the boundary area on the basis of sensor data obtained from the camera and the at least one sensor; on the basis of identifying that the position of the wearable device in the boundary area is within a first threshold distance from the boundary area, display a pattern image for indicating the boundary area through the display while displaying the virtual environment through the display; and on the basis of identifying that the position of an input means for the wearable device is within a second threshold distance from the boundary area, display, through the display, at least a part of an external image obtained through the camera in a virtual area according to the position of the input means while displaying the virtual environment and the pattern image through the display.

Description

Electronic device, method, and computer-readable medium for displaying a virtual area

[0001] The present disclosure relates to an electronic device, method, and computer-readable medium for displaying a virtual area.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices can provide AR services to users by utilizing virtual objects corresponding to the user.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] In embodiments, a wearable device is provided. The wearable device may include a camera, at least one sensor, a display, a memory for storing instructions, and a processor. The instructions, when executed by the processor, may cause the wearable device to obtain information about a boundary zone for guiding a boundary within which a virtual environment is provided, obtain a location of the wearable device with respect to the boundary zone based on sensor data obtained from the camera or the at least one sensor, display a pattern image for representing the boundary zone through the display while displaying a virtual environment through the display based on identifying that a location of the wearable device within the boundary zone is within a first threshold distance from the boundary zone, and display at least a portion of an external image obtained through the camera through the display in a virtual area according to a location of the input means while displaying the virtual environment and the pattern image through the display based on identifying that a location of an input means for the wearable device is within a second threshold distance from the boundary zone.

[0005] In embodiments, a wearable device is provided. The wearable device may include a camera, at least one sensor, a display, a memory for storing instructions, and a processor. The instructions, when executed by the processor, cause the wearable device to obtain information about a boundary zone for providing a virtual environment based on scanning of a physical space, obtain a location of the wearable device with respect to the boundary zone based on sensor data obtained from the camera or the at least one sensor, and display an external image obtained through the camera through the display based on identifying that the location of the wearable device is located outside the boundary zone, and display a pattern image for indicating the boundary zone while displaying the external image obtained through the camera based on identifying that the location of the wearable device is within a first threshold distance from the boundary zone outside the boundary zone, and display a virtual image of the virtual environment in a boundary virtual zone according to the location of the wearable device or the location of the input means while displaying the external image and the pattern image through the display based on identifying that the location of the wearable device or the location of the input means for the wearable device is within a second threshold distance from the boundary zone. It may cause it to be displayed.

[0006] Figure 1 shows an example of a boundary area for providing a virtual environment.

[0007] Figure 2a shows an example of a perspective view of a wearable device.

[0008] FIG. 2b illustrates an example of one or more hardware devices arranged within a wearable device.

[0009] Figures 3a and 3b show an example of the appearance of a wearable device.

[0010] Figure 4 shows an example of a block diagram of a wearable device.

[0011] Figures 5a to 5c illustrate examples of interactions between a boundary area and a user within a boundary area.

[0012] Figure 6 shows an example of virtual area and sound control for external images.

[0013] Figure 7 shows an example of displaying a pattern image according to the user's status.

[0014] Figure 8 shows an example of a notification message for notifying a user of his / her status.

[0015] Figures 9a to 9e illustrate examples of visual objects for area adjustment of a virtual area.

[0016] Figures 10a and 10b illustrate examples of interaction between a boundary area and a user using multiple virtual areas.

[0017] Figure 11a illustrates the operation flow of a wearable device for interaction with a boundary zone within a boundary zone.

[0018] Figure 11b shows the operation flow of a wearable device for adjusting the area of ​​a virtual area within a boundary zone.

[0019] Figures 12a and 12b illustrate examples of interactions between a boundary area and a user outside the boundary area.

[0020] Figure 13 shows an example of displaying a boundary area according to an application outside the boundary area.

[0021] Figure 14a illustrates the operation flow of a wearable device for interaction with a boundary zone from outside the boundary zone.

[0022] Figure 14b shows the operation flow of the wearable device for adjusting the area of ​​the virtual area outside the boundary area.

[0023] Figure 15 shows an example of a virtual area for an external object entering the boundary zone.

[0024] Figure 16 shows an example of a notification of an external object entering a boundary area.

[0025] Figure 17 illustrates the operation flow of a wearable device for notifying an external object entering a boundary area.

[0026] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0027] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0028] In the following description, terms referring to space (e.g., plane, object, shape, surface, figure, solid figure, area, occupied area, position, depth, distance), terms referring to distance (e.g., position, distance, depth, distance information, distance value, depth value, position information, position data, depth data), terms referring to pass-through surfaces (e.g., pass-through area, virtual surface, pass-through surface, view surface, virtual area, view area, transition surface, transition space, transition area, pass surface, pass area), terms referring to values ​​(e.g., threshold value, reference value, reference area, reference range, level, threshold, range, value, area), terms for operation states (e.g., step, operation, procedure), terms referring to network entities, terms referring to components of a device, etc., are examples for convenience of description. Therefore, the present disclosure is not limited to the terms described below, and other terms having equivalent technical meanings may be used.

[0029] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0030] The processor (120) of the present disclosure may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be individually and / or collectively configured to perform the various functions(s) described in the present disclosure. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms may include, for example, without limitation, situations where a single processor performs some of the recited functions, situations where other processor(s) perform other of the recited functions, situations where a single processor can perform all of the recited functions, and / or combinations of processors performing in a distributed manner. Additionally, instructions (or program commands) for various functions in the present disclosure, when executed by a processor, may cause an electronic device (e.g., a wearable device (101)) to execute the various functions.

[0031] Figure 1 shows an example of a boundary area for providing a virtual environment.

[0032] Referring to FIG. 1, a wearable device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a video see-through (VST) or visible see-through device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 4. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIGS. 2A, 2B, 3A, and / or 3B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory (e.g., a strap) for attaching to the head of a user to form an HMD.

[0033] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0034] In one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0035] According to one embodiment, the wearable device (101) can identify or recognize a position (or location) and / or direction (or orientation) of the wearable device (101) based on images (and / or videos) obtained or acquired using a camera. Referring to FIG. 1, within an exemplary state (191) contained in an external space (e.g., a room) including external objects (121, 122, 123, 124, 125, 126), the wearable device (101) can obtain information about the external space using one or more cameras and / or one or more sensors. The information can include a geographic location (e.g., global positioning system (GPS) coordinates) of the external space identified from one or more sensors. The information can include images and / or videos of the external space identified from one or more cameras. The wearable device (101) can perform object recognition on an image and / or video to identify external objects (121, 122, 123, 124, 125, 126) included in the external space from the image and / or video.

[0036] The wearable device (101) may provide a user experience based on virtual reality and / or augmented reality while worn by a user (110). For example, the wearable device (101) may display a virtual object or imaginary object that covers at least a portion of the field-of-view (FoV) of the user (110). If a real object is occluded by the virtual object within the field of view, the user (110) who does not recognize the real object may collide with the real object. The wearable device (101) may define a boundary area (130) so that the user (110) does not collide with the distribution (or positional relationship) of the real object (e.g., external objects (121, 122, 123, 124, 125, 126)).

[0037] The wearable device (101) can set a boundary zone (130). The boundary zone (130) represents the range of the physical location of the wearable device (101) and / or an input means for the wearable device (101) (e.g., a controller connected to the wearable device (101), at least a part of the body of the user (110) (e.g., a hand)) within which the wearable device (101) is set to display a virtual environment. The boundary zone (130) can be used to guide the boundary within which the virtual environment is provided to the user (110). The boundary zone (130) may be referred to as a guardian area, a guardian region, a protected region, a protected area, a protected boundary area, a protected boundary region, a safety area, a safety region (safety region or safety zone), or a guardian region, in terms of providing virtual reality and / or augmented reality without collision.

[0038] According to one embodiment, the wearable device (101) may set a boundary zone (130) according to a user input of a user (110). The boundary zone (130) may include a boundary line (131) indicating a boundary on the floor and a boundary surface (132) formed above the boundary line (131). For example, the wearable device (101) may set a closed curve area set through a controller connected to the wearable device (101) as the boundary line (131) of the boundary zone (130). For example, the wearable device (101) may set a closed shape (e.g., a circle) of a specified size as the boundary line (131) of the boundary zone (130) in response to the user input. According to another embodiment, the wearable device (101) may set the boundary zone (130) using at least one sensor and / or camera. For example, the wearable device (101) can set a boundary zone (130) by scanning a physical space using at least one sensor and / or a camera. The wearable device (101) can obtain information about the distribution (or positional relationship) of real objects (e.g., external objects (121, 122, 123, 124, 125, 126)) located within a plurality of areas of a designated area. The wearable device (101) can set a boundary zone (130) based on the information about the real objects.

[0039] Below, an example of a hardware configuration of a wearable device (101) is described with reference to FIGS. 2a, 2b, 3a, 3b, and 4.

[0040] FIG. 2A illustrates an example of a perspective view of a wearable device. FIG. 2B illustrates an example of one or more hardware components arranged within the wearable device. According to one embodiment, the wearable device (101) may have a form of glasses that can be worn on a body part (e.g., head) of a user. The wearable device (101) of FIGS. 2A and 2B may be an example of the wearable device (101) of FIG. 1. The wearable device (101) may include a head-mounted display (HMD). For example, the housing of the wearable device (101) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (101) may include one or more straps capable of being twined around the user's head, and / or one or more temples attachable to the ears of the head.

[0041] Referring to FIG. 2A, according to one embodiment, a wearable device (101) may include at least one display (250) and a frame (200) supporting at least one display (250).

[0042] According to one embodiment, a wearable device (101) can be worn on a part of a user's body. The wearable device (101) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (101). For example, the wearable device (101) can display a virtual reality image provided from at least one optical device (282, 284) of FIG. 2B on at least one display (250) in response to a user's designated gesture acquired through the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0043] According to one embodiment, at least one display (250) may provide visual information to a user. For example, at least one display (250) may include a transparent or translucent lens. At least one display (250) may include a first display (250-1) and / or a second display (250-2) spaced apart from the first display (250-1). For example, the first display (250-1) and the second display (250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0044] Referring to FIG. 2B, at least one display (250) can provide visual information transmitted from external light to the user through a lens included in the at least one display (250), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (250) can include a first surface (231) and a second surface (232) opposite to the first surface (231). A display area can be formed on the second surface (232) of the at least one display (250). When a user wears the wearable device (101), external light can be transmitted to the user by being incident on the first surface (231) and transmitted through the second surface (232). As another example, at least one display (250) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (282, 284) on a real screen transmitted through external light, in a display area formed on the second surface (232).

[0045] In one embodiment, at least one display (250) may include at least one waveguide (233, 234) that diffracts light emitted from at least one optical device (282, 284) and transmits the diffracted light to a user. The at least one waveguide (233, 234) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (233, 234). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (233, 234) may be propagated to the other end of the at least one waveguide (233, 234) by the nano-pattern. At least one waveguide (233, 234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (233, 234) may be arranged within the wearable device (101) to guide a screen displayed by at least one display (250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (233, 234).

[0046] The wearable device (101) can analyze an object included in a real image collected through a shooting camera (260-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (250). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (101) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (101) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using a multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (101) can view an image displayed on at least one display (250).

[0047] According to one embodiment, the frame (200) may be formed as a physical structure that allows the wearable device (101) to be worn on the user's body. According to one embodiment, the frame (200) may be configured so that, when the user wears the wearable device (101), the first display (250-1) and the second display (250-2) can be positioned corresponding to the user's left and right eyes. The frame (200) may support at least one display (250). For example, the frame (200) may support the first display (250-1) and the second display (250-2) to be positioned corresponding to the user's left and right eyes.

[0048] Referring to FIG. 2A, the frame (200) may include a region (220) that is at least partially in contact with a part of the user's body when the user wears the wearable device (101). For example, the region (220) of the frame (200) that is in contact with a part of the user's body may include a region that is in contact with a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that the wearable device (101) makes contact with. According to one embodiment, the frame (200) may include a nose pad (210) that is in contact with a part of the user's body. When the wearable device (101) is worn by the user, the nose pad (210) may be in contact with a part of the user's nose. The frame (200) may include a first temple (204) and a second temple (205) that are in contact with another part of the user's body that is distinct from the part of the user's body.

[0049] For example, the frame (200) may include a first rim (201) that surrounds at least a portion of the first display (250-1), a second rim (202) that surrounds at least a portion of the second display (250-2), a bridge (203) that is disposed between the first rim (201) and the second rim (202), a first pad (211) that is disposed along a portion of the edge of the first rim (201) from one end of the bridge (203), a second pad (212) that is disposed along a portion of the edge of the second rim (202) from the other end of the bridge (203), a first temple (204) that extends from the first rim (201) and is fixed to a portion of the wearer's ear, and a second temple (205) that extends from the second rim (202) and is fixed to a portion of the ear opposite the ear. The first pad (211) and the second pad (212) may be in contact with a portion of the user's nose, and the first temple (204) and the second temple (205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (204, 205) may be rotatably connected to the rim through the hinge units (206, 207) of FIG. 2B. The first temple (204) may be rotatably connected to the first rim (201) through the first hinge unit (206) disposed between the first rim (201) and the first temple (204). The second temple (205) may be rotatably connected to the second rim (202) through the second hinge unit (207) disposed between the second rim (202) and the second temple (205). According to one embodiment, the wearable device (101) can identify an external object (e.g., a user's fingertip) touching the frame (200) and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (200).

[0050] According to one embodiment, the wearable device (101) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 4). For example, the hardwares may include a battery module (270), an antenna module (275), at least one optical device (282, 284), speakers (e.g., speakers 255-1, 255-2), a microphone (e.g., microphones 265-1, 265-2, 265-3), a light-emitting module (not shown), and / or a printed circuit board (PCB) (290) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (200).

[0051] According to one embodiment, the microphones (e.g., microphones 265-1, 265-2, 265-3) of the wearable device (101) may be disposed on at least a portion of the frame (200) to acquire sound signals. A first microphone (265-1) disposed on the bridge (203), a second microphone (265-2) disposed on the second rim (202), and a third microphone (265-3) disposed on the first rim (201) are illustrated in FIG. 2B, but the number and arrangement of the microphones (265) are not limited to the embodiment of FIG. 2B. When the number of microphones (265) included in the wearable device (101) is two or more, the wearable device (101) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame (200).

[0052] According to one embodiment, at least one optical device (282, 284) may project a virtual object onto at least one display (250) to provide various image information to a user. For example, at least one optical device (282, 284) may be a projector. At least one optical device (282, 284) may be disposed adjacent to at least one display (250) or may be included within at least one display (250) as a part of at least one display (250). According to one embodiment, the wearable device (101) may include a first optical device (282) corresponding to a first display (250-1) and a second optical device (284) corresponding to a second display (250-2). For example, at least one optical device (282, 284) may include a first optical device (282) disposed at an edge of a first display (250-1) and a second optical device (284) disposed at an edge of a second display (250-2). The first optical device (282) may transmit light to a first waveguide (233) disposed on the first display (250-1), and the second optical device (284) may transmit light to a second waveguide (234) disposed on the second display (250-2).

[0053] In one embodiment, the camera (260) may include a recording camera (260-4), an eye tracking camera (ET CAM) (260-1), and / or a motion recognition camera (260-2, 260-3). The recording camera (260-4), the eye tracking camera (260-1), and the motion recognition cameras (260-2, 260-3) may be positioned at different locations on the frame (200) and may perform different functions. The eye tracking camera (260-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (101). For example, the wearable device (101) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (260-1). The wearable device (101) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (101) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (260-1). The wearable device (101) can render an image (or screen) displayed on at least one display (250) based on the position of the user's eyes. For example, the visual quality of a first region related to the gaze within the image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinguished from the first region may be different from each other. The wearable device (101) can obtain an image having visual quality of a first area matching the user's gaze and visual quality of a second area using foveated rendering.For example, if the wearable device (101) supports an iris recognition function, user authentication can be performed based on iris information acquired using the gaze tracking camera (260-1). An example in which the gaze tracking camera (260-1) is positioned toward the user's right eye is illustrated in FIG. 2B, but the embodiment is not limited thereto, and the gaze tracking camera (260-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0054] In one embodiment, the capturing camera (260-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (260-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (260-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (250). The at least one display (250) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (260-4) and a virtual image provided through at least one optical device (282, 284) are superimposed. The wearable device (101) can compensate for depth information (e.g., the distance between the wearable device (101) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (260-4). The wearable device (101) can perform object recognition through an image acquired using the capturing camera (260-4). The wearable device (101) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (260-4). The wearable device (101) can perform a pass-through function to display an image acquired through the capturing camera (260-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (250). In one embodiment, the capturing camera (260-4) can be disposed on a bridge (203) disposed between the first rim (201) and the second rim (202).

[0055] The gaze tracking camera (260-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (101) and thereby matching the user's gaze with visual information provided to at least one display (250). For example, when the wearable device (101) looks straight ahead, the wearable device (101) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (250). The gaze tracking camera (260-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (260-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (260-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (260-1) may be positioned within the first rim (201) and / or the second rim (202) to face the direction in which the user wearing the wearable device (101) is positioned.

[0056] The motion recognition camera (260-2, 260-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (250). The motion recognition camera (260-2, 260-3) can recognize the user's motion (gesture recognition), obtain a signal corresponding to the motion, and provide a display corresponding to the signal on at least one display (250). The processor can identify the signal corresponding to the motion, and perform a designated function based on the identification. The motion recognition camera (260-2, 260-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the motion recognition camera (260-2, 260-3). In one embodiment, the motion recognition cameras (260-2, 260-3) may be positioned on the first rim (201) and / or the second rim (202).

[0057] The camera (260) included in the wearable device (101) is not limited to the above-described gaze tracking camera (260-1) and motion recognition cameras (260-2, 260-3). For example, the wearable device (101) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (101) can identify an external object based on a sensor for identifying the distance between the wearable device (101) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (260) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (101) may include a camera (260) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (101).

[0058] Although not shown, in one embodiment, the wearable device (101) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (260). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame (200) and the hinge units (206, 207).

[0059] According to one embodiment, the battery module (270) may supply power to electronic components of the wearable device (101). In one embodiment, the battery module (270) may be disposed within the first temple (204) and / or the second temple (205). For example, the battery module (270) may be a plurality of battery modules (270). The plurality of battery modules (270) may be disposed within each of the first temple (204) and the second temple (205). In one embodiment, the battery module (270) may be disposed at an end of the first temple (204) and / or the second temple (205).

[0060] The antenna module (275) can transmit signals or power to the outside of the wearable device (101), or receive signals or power from the outside. In one embodiment, the antenna module (275) can be positioned within the first temple (204) and / or the second temple (205). For example, the antenna module (275) can be positioned close to one surface of the first temple (204) and / or the second temple (205).

[0061] The speaker (255) can output an acoustic signal to the outside of the wearable device (101). The acoustic output module may be referred to as a speaker. In one embodiment, the speaker (255) may be positioned within the first temple (204) and / or the second temple (205) so as to be positioned adjacent to the ear of a user wearing the wearable device (101). For example, the speaker (255) may include a second speaker (255-2) positioned within the first temple (204) and thus positioned adjacent to the user's left ear, and a first speaker (255-1) positioned within the second temple (205) and thus positioned adjacent to the user's right ear.

[0062] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (101) to the user. For example, when the wearable device (101) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (201) and / or the second rim (202).

[0063] Referring to FIG. 2B, according to one embodiment, a wearable device (101) may include a printed circuit board (PCB) (290). The PCB (290) may be included in at least one of the first temple (204) or the second temple (205). The PCB (290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (101) (e.g., hardwares illustrated by different blocks in FIG. 4A) may be disposed on the PCB (290). The wearable device (101) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0064] According to one embodiment, a wearable device (101) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (101) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (101). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (101) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (101) based on the IMU.

[0065] FIGS. 3A and 3B illustrate an example of an exterior appearance of a wearable device (e.g., a wearable device (101)). The wearable device (101) of FIGS. 3A and 3B may be an example of the wearable device (101) of FIG. 1 . According to one embodiment, an example of an exterior appearance of a first side (310) of a housing of the wearable device (101) is illustrated in FIG. 3A , and an example of an exterior appearance of a second side (320) opposite to the first side (310) may be illustrated in FIG. 3B .

[0066] Referring to FIG. 3A, according to one embodiment, a first surface (310) of a wearable device (101) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (101) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (204) and / or the second temple (205) of FIGS. 2A and 2B). A first display (250-1) for outputting an image to a left eye among the user's two eyes, and a second display (250-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (310). The wearable device (101) is formed on the first surface (310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (250-1) and the second display (250-2).

[0067] According to one embodiment, the wearable device (101) may include cameras (260-1) for photographing and / or tracking both eyes of the user adjacent to each of the first display (250-1) and the second display (250-2). The cameras (260-1) may be referred to as the gaze tracking camera (260-1) of FIG. 2B. According to one embodiment, the wearable device (101) may include cameras (260-5, 260-6) for photographing and / or recognizing the face of the user. The cameras (260-5, 260-6) may be referred to as FT cameras. The wearable device (101) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (260-5, 260-6). For example, the wearable device (101) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar expressing a human face) using information obtained by cameras (260-5, 260-6) (e.g., FT cameras) and representing the facial expression of a user wearing the wearable device (101).

[0068] Referring to FIG. 3B, a camera (e.g., cameras 260-7, 260-8, 260-9, 260-10, 260-11, 260-12)) and / or a sensor (e.g., a depth sensor (330)) for obtaining information related to the external environment of the wearable device (101) may be disposed on a second surface (320) opposite to the first surface (310) of FIG. 3A. For example, the cameras (260-7, 260-8, 260-9, 260-10) may be disposed on the second surface (320) to recognize external objects. Cameras (260-7, 260-8, 260-9, 260-10) may be referenced to the motion recognition cameras (260-2, 260-3) of FIG. 2B.

[0069] For example, using cameras (260-11, 260-12), the wearable device (101) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (260-11) can be placed on the second face (320) of the wearable device (101) to obtain an image to be displayed through the second display (250-2) corresponding to the right eye among the two eyes. The camera (260-12) can be placed on the second face (320) of the wearable device (101) to obtain an image to be displayed through the first display (250-1) corresponding to the left eye among the two eyes. The cameras (260-11, 260-12) can be referred to as the shooting camera (260-4) of FIG. 2B.

[0070] According to one embodiment, the wearable device (101) may include a depth sensor (330) disposed on the second face (320) to identify a distance between the wearable device (101) and an external object. Using the depth sensor (330), the wearable device (101) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (101). Although not illustrated, a microphone may be disposed on the second face (320) of the wearable device (101) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0071] Hereinafter, with reference to FIG. 4, the hardware or software configuration of the wearable device (101) is described.

[0072] Fig. 4 illustrates an example of a block diagram of a wearable device (e.g., a wearable device (101)). The wearable device (101) of Fig. 4 may be an example of the wearable device (101) of Fig. 1 or the wearable devices (101) of Figs. 2A to 3B.

[0073] Referring to FIG. 4, a wearable device (101) according to one embodiment may include at least one of a processor (410), a memory (415), a display (420), a camera (425), a sensor (430), or a communication circuit (480). The processor (410), the memory (415), the display (420), the camera (425), the sensor (430), and the communication circuit (480) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (402). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 4. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 4. The elements within the memory described below (e.g., layers and / or modules) may be logically distinct, but are not limited thereto.

[0074] According to one embodiment, the processor (410) of the wearable device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (410) may be one or more. For example, the processor (410) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0075] The memory (415) of the wearable device (101) according to one embodiment may include a hardware component for storing data and / or instructions input and / or output to the processor (410). The memory (415) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi-media card (eMMC).

[0076] In one embodiment, the display (420) of the wearable device (101) can output visualized information to a user of the wearable device (101). For example, the display (420) can be controlled by a processor (410) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (420) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0077] In one embodiment, the camera (425) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (425) may be arranged in the form of a two-dimensional array. The camera (425) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (425) may mean one (a) two-dimensional frame data acquired from the camera (425). For example, video data captured using the camera (425) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (425) according to a frame rate. The camera (425) may further include a flash light that is positioned toward the direction in which the camera (425) receives light and outputs light toward the direction.

[0078] According to one embodiment, the wearable device (101) may include a plurality of cameras (not shown) arranged in different directions, as an example of a camera (425). A first camera among the plurality of cameras may be referred to as a motion recognition camera, and a second camera may be referred to as an eye tracking camera. The wearable device (101) may identify a position, shape, and / or gesture of a hand using an image of the first camera (e.g., image (110) of FIG. 1). The wearable device (101) may identify a direction of a user's (e.g., user (105) of FIG. 1) gaze using an image of the second camera (not shown). For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0079] According to one embodiment, a sensor (430) of a wearable device (101) may generate electrical information that may be processed by a processor (410) and / or a memory (415) of the wearable device (101) from non-electronic information related to the wearable device (101). The information may be referred to as sensor data. The sensor (430) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (101).

[0080] In one embodiment, the communication circuit (480) of the wearable device (101) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device (e.g., the server (405) of FIG. 4). The communication circuit (480) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (480) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0081] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (410) of the wearable device (101) may be stored in the memory (415) of the wearable device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (101) and / or the processor (410) may perform at least one of the operations described below when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, and / or an application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (415), and that the one or more applications are stored in a format executable by the processor (410) (e.g., a file having an extension specified by the operating system of the wearable device (101)). As an example, the application may include a program and / or a library related to a service provided to a user.

[0082] Referring to FIG. 4, programs installed in the wearable device (101) may be classified into any one of different layers, including an application layer (440), a framework layer (450), and / or a hardware abstraction layer (HAL) (460), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware of the wearable device (101) (e.g., a display (420), a camera (425), and / or a sensor (430)) may be classified within the hardware abstraction layer (460). For example, although FIG. 4 illustrates the layers being divided within the memory (415), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (415).

[0083] For example, within the framework layer (450), programs designed to target at least one of the hardware abstraction layer (460) and / or the application layer (440) (e.g., a position tracker (453-1), a space recognizer (453-2), a gesture tracker (453-3), an eye tracker (453-4), a controller tracker (453-5), a virtual space analyzer (455), and / or a renderer (454)) may be classified. Programs classified into the framework layer (450) may provide an executable API (application programming interface) based on other programs.

[0084] For example, within the application layer (440), programs designed to target a user (e.g., user (105) of FIG. 1) controlling a wearable device (101) may be classified. As an example of programs classified into the application layer (440), the first software application (441) is exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) classified into the application layer (440) may call an API to cause the execution of functions supported by programs classified into the framework layer (450). For example, the wearable device (101) may display a screen representing a virtual environment on the display (420) based on the execution of the first software application (441).

[0085] According to one embodiment, the virtual environment manager (451) may be configured for a virtual environment service. For example, the virtual environment manager (451) may include a platform (e.g., an Android platform) for supporting the virtual environment service. The virtual environment manager (451) may display the posture of a virtual object representing the user's posture, rendered using data acquired through the sensor (430), on the display. The virtual environment manager (451) may be referred to as a composition presentation manager.

[0086] For example, the virtual environment manager (451) may include a runtime service (451-1). As an example, the runtime service (451-1) may be referred to as an OpenXR runtime module. The runtime service (451-1) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (101). As an example, the runtime service (451-1) may be used to perform rendering for a virtual environment service to the user. For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the runtime module (451-1).

[0087] For example, the perception abstract layer (452) can be used for data exchange between the virtual environment manager (451) and the perception service layer (453). From the perspective of being used for data exchange between the virtual environment manager (451) and the perception service layer (453), the perception abstract layer (452) can be referred to as an interface. For example, the perception abstract layer (452) can be referenced as OpenPX. The perception abstract layer (452) can be used for a perception client and a perception service.

[0088] According to one embodiment, the recognition service layer (453) may include one or more programs for processing data acquired from a sensor (430) (or camera (425)). The one or more programs may include at least one of a position tracker (453-1), a spatial recognizer (453-2), a gesture tracker (453-3), a gaze tracker (453-4), and a controller tracker (453-5).

[0089] For example, the position tracker (453-1) can identify the posture of the wearable device (101) by using at least one sensor of the wearable device (101). As an example, the position tracker (453-1) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) based on data acquired using a camera (e.g., camera (425) of FIG. 4) and an IMU. The position tracker (453-1) may be referred to as a head tracking (HeT) module.

[0090] For example, the space recognizer (453-2) may be used to construct a three-dimensional virtual space surrounding the wearable device (101) (or a user of the wearable device (101)). The space recognizer (453-2) may be used to reconstruct the three-dimensional surroundings of the wearable device (101) based on data acquired using a camera (e.g., the camera (425) of FIG. 4). The space recognizer (453-2) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (101). The space recognizer (453-2) may be referred to as a scene understanding (SU) module.

[0091] For example, the gesture tracker (453-3) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of a wearable device (101). As an example, the gesture tracker (453-3) may identify a pose and / or gesture of a hand of a user based on data acquired from at least one sensor. As an example, the gesture tracker (453-3) may identify a pose and / or gesture of a hand of a user based on data acquired using a camera (e.g., an image (110)). The gesture tracker (453-3) may be referred to as a hand tracking (HaT) module.

[0092] For example, the gaze tracker (453-4) may be used to identify (or track) eye movements of a user of the wearable device (101). For example, the gaze tracker (453-4) may identify eye movements of the user based on data acquired from at least one sensor. For example, the gaze tracker (453-4) may identify eye movements of the user based on data acquired using a camera (e.g., the gaze tracking camera (260-1) of FIGS. 2A and 2B) and / or an infrared light emitting diode (IR LED). The gaze tracker (453-4) may be referred to as an eye tracking (ET) module.

[0093] For example, the controller tracker (453-4) may be used to identify (or track) the movement of a controller connected to a wearable device (101). As an example, the controller tracker (453-4) may identify the movement of the controller based on receiving sensor data included in the controller through a communication circuit.

[0094] For example, the recognition service layer (453) of the wearable device (101) may further include a face tracking module (not shown) for tracking the user's face. For example, the face tracking module may be used to identify (or track) the user's facial movements and / or the user's facial expressions. The face tracking module may estimate the user's facial expressions based on the user's facial movements. For example, the face tracking module may identify the user's facial movements and / or the user's facial expressions based on data (e.g., images) acquired using a camera.

[0095] For example, the wearable device (101) may obtain data representing the user's posture and / or movement from one or more programs included in the recognition service layer (453) based on the execution of the renderer (454), thereby rendering a screen including a virtual object representing the user's posture within a virtual environment. The wearable device (101) may render one or more images (e.g., virtual object animation) for changing the posture of the virtual object using the posture information (470) based on the execution of the renderer (454). The wearable device (101) may change the posture of the entire virtual object based on inverse kinematics by identifying the posture of at least a portion of the user. When changing a portion of the virtual object, the wearable device (101) may change the posture of the entire virtual object by inferring the position at which another portion of the virtual object will change based on position information within the virtual environment using the inverse kinematics.

[0096] For example, the wearable device (101) can identify virtual environment information based on the execution of the virtual environment analyzer (455). As an example, the virtual environment information can be received from the server (405) to the wearable device (101).

[0097] For example, the posture information (470) may include one or more images representing the posture of the virtual object. As an example, the wearable device (101) may receive the posture information (470) from the server (405). The wearable device (101) may use the posture information (470) to identify the posture of the virtual object corresponding to the user's posture, thereby controlling the posture of the virtual object displayed on the display to be changed.

[0098] Referring to FIG. 4, the wearable device (101) and the server (405) may be connected to each other based on a wired network and / or a wireless network. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the wearable device (101) and the server (405) are illustrated as being directly connected, the wearable device (101) and the server (405) may be indirectly connected via an intermediate node within the network. The server (405) may be referred to as an external electronic device in that it is located outside the wearable device (101).

[0099] Referring to FIG. 4, in one embodiment, the server (405) may include at least one of a processor (410-1), a memory (415-1), and a communication circuit (480-1). Within the server (405), the processor (410-1), the memory (415-1), and the communication circuit (480-1) may be electrically and / or operatively coupled via a communication bus. The processor (410-1), the memory (415-1), and the communication circuit (480-1) included within the server (405) may include hardware components and / or circuits corresponding to the processor (410), the memory (415), and the communication circuit (480) of the wearable device (101). Hereinafter, to reduce repetition, descriptions of hardware and / or software included within the server (405) may be omitted to the extent that they overlap with the wearable device (101).

[0100] According to one embodiment, the server (405) can transmit a screen representing a virtual environment to the wearable device (101) using a virtual environment plug-in (490). The server (405) can provide a virtual environment service to the wearable device (101) (or the user (105)) using the virtual environment plug-in (490). The server (405) can store a program including instructions for interaction with the wearable device (101) in the memory (415-1). As an example of the program, the first software application (441-1) is exemplified, but the embodiment is not limited thereto.

[0101] For example, the server (405) may receive data (e.g., sensor data) acquired from one or more programs included in the recognition service layer (453) of the wearable device (101) from the wearable device (101). The server (405) may use the received data to generate rendering data for changing the posture of a virtual object using the renderer (454-1). For example, the server (405) may generate rendering data for changing the posture of a virtual object to the wearable device (101) based on the execution of the first software application (441-1). The server (405) may transmit the rendering data to the wearable device (101) to display a screen including a virtual object whose posture has been changed using the rendering data on the display (420) of the wearable device (101). However, the present invention is not limited thereto.

[0102] Figures 5a to 5c illustrate examples of interactions between a boundary zone (e.g., boundary zone (130)) and a user (e.g., user (110)) within a boundary zone. Figures 5a to 5c illustrate a situation in which a user (110) is positioned within a boundary zone set by a wearable device (101). The boundary zone may be set to guide the user (110) to the boundaries of a virtual environment. The same reference numbers may indicate the same description in different drawings.

[0103] Referring to FIG. 5A, the wearable device (101) may provide visual information about the boundary zone (130) to the user (110) of the wearable device (101) when the user (110) of the wearable device (101) is located within a first threshold distance from the boundary zone (130). For example, the wearable device (101) may display the boundary zone (130) when the location of the wearable device (101) is located within the first threshold distance from the boundary zone (130). The boundary zone (130) may include a boundary line (131) and a boundary surface (132). The wearable device (101) may display the boundary line (131) through a display (e.g., display (420)) so that the user (110) may recognize the boundary zone (130) on the floor. The wearable device (101) can display a pattern image (e.g., pattern image (560a), pattern image (560b)) on the boundary surface (132) through the display (420) so that the user (110) can recognize the boundary area (130) in a three-dimensional space. The wearable device (101) can display the pattern image while displaying a virtual environment through the display (420).

[0104] According to one embodiment, the wearable device (101) may display a pattern image according to the location of the wearable device (101). For example, the first screen (511) represents an avatar of a user (110) of the wearable device (101) located at a first point (518a) within the boundary zone (130). At the first point (518a), the distance between the boundary zone (130) and the wearable device (101) may be a first distance (519a). The wearable device (101) may display a pattern image (560a) based on the first distance (519a). The second screen (513) represents an avatar of a user (110) of the wearable device (101) located at a second point (519b) within the boundary zone (130). At the second point (518b), the distance between the boundary area (130) and the wearable device (101) may be a second distance (519b). The wearable device (101) may display a pattern image (560b) based on the second distance (519b).

[0105] The pattern image (e.g., pattern image (560a), pattern image (560b)) displayed on the boundary surface (132) of the boundary area (130) may include a pattern (e.g., grid) in which a specified shape is repeated. As the user (110) gets closer to the boundary area (130), the wearable device (101) may set the density of the specified shape within the grid to be higher. For example, as the user (110) gets closer to the boundary area (130), the wearable device (101) may generate the pattern image so that the density of the specified shape becomes higher. For example, the second point (518b) may be closer to the boundary area (130) than the first point (518a). The first distance (519a) may be longer than the second distance (519b). The pattern image (560a) may include a first grid in which circles are repeated. The pattern image (560b) may include a second grid with repeating circles. The density of the circles in the second grid may be higher than the density of the circles in the first grid. As the user (110) approaches the boundary area (130), the shapes (e.g., circles) on the pattern image are displayed more densely, so the user (110) of the wearable device (101) may recognize that the wearable device (101) is closer to the boundary area (130).

[0106] The user (110) may bring at least a part of the user's (110) body close to the boundary zone (130). For example, the user (110) may wear a wearable device (101), and the user's (110) hand may hold a controller (590) connected to the wearable device (101). Hereinafter, as an example, the controller (590) is described as a device that can be gripped by the user's (110) hand, but the controller (590) may be another type of device. For example, the controller (590) may be a wearable device (e.g., a ring, a bracelet, a glove, an armband, a wristband) that can be worn on at least a part of the user's (110) body. The wearable device (101) may obtain information about the movement of the user's (110) hand through the movement of the controller (590). According to one embodiment, the wearable device (101) may display a pattern image according to the position of the controller (590). When the position of the controller (590) is within a second threshold distance from the boundary zone (130), the wearable device (101) may display a pattern image to indicate an interaction between the controller (590) and the boundary zone (130). As the position of the controller (590) gets closer to the boundary zone (130), the wearable device (101) may merge shapes within the pattern image (e.g., pattern image (560b)) and display the merged shapes to more effectively indicate the interaction. Hereinafter, in describing embodiments of the present disclosure, the controller (590) is exemplified as an input means for indicating an interaction between the boundary zone (130) and the user (110), but embodiments of the present disclosure are not limited thereto. Not only the controller (590) connected to the wearable device (101), but also at least a part of the user's (110) body can be used as an input means for the wearable device (101).For example, the wearable device (101) can detect user input by detecting a part of the user's (110) body (e.g., left hand, right hand, arm, gesture) or movement of the part through a camera (e.g., camera (425)) or sensor (430)) of the wearable device (101).

[0107] For example, the third screen (515) represents at least a part (570) (e.g., a hand) of the avatar of the user (110) located within a second threshold distance from the boundary zone (130). The wearable device (101) may display a pattern image (560c) based on identifying that the location of the controller (590) is located within the second threshold distance from the boundary zone (130). The wearable device (101) may display a merged figure by connecting figures located at a certain distance from a point indicated by the direction of the controller (590) among figures of the pattern image being displayed (e.g., pattern image (560b)). The wearable device (101) may display a pattern image (560c) including the merged figure. For example, the fourth screen (517) represents at least a portion (570) (e.g., a hand) of the avatar of the user (110) touching the boundary area (130). The wearable device (101) may display a pattern image (560d) based on identifying that the controller (590) is positioned over the boundary area (130). The wearable device (101) may display a merged shape by connecting shapes located at a certain distance from a point indicated by the direction of the controller (590) among the shapes of the pattern image being displayed (e.g., pattern image (560b, pattern image (560c)). In the fourth screen (517), the wearable device (101) may merge more shapes than in the third screen (517) in order to express the interaction more strongly. The wearable device (101) can display a pattern image (560d) including one shape based on the above merging.

[0108] In FIG. 5a, a circle is exemplified as a basic shape forming a pattern image (e.g., pattern image (560a), pattern image (560b), pattern image (560c), pattern image (560d)), but embodiments of the present disclosure are not limited thereto. In addition to a circle, shapes such as a '+' sign, a square, a polygon, a '*', a '#', and an 'X' may be used to represent a boundary surface (132) within a pattern image. In FIG. 5b, a pattern image according to a straight line pattern (e.g., a rectangle) is exemplified instead of a circle.

[0109] Referring to FIG. 5B, the wearable device (101) may provide visual information about the boundary zone (130) to the user (110) of the wearable device (101) when the user (110) of the wearable device (101) is located within a first threshold distance from the boundary zone (130). For example, the wearable device (101) may display the boundary zone (130) when the location of the wearable device (101) is located within the first threshold distance from the boundary zone (130). The boundary zone (130) may include a boundary line (131) and a boundary surface (132). The wearable device (101) may display the boundary line (131) through a display (e.g., display (420)) so that the user (110) may recognize the boundary zone (130) on the floor. The wearable device (101) can display a pattern image (e.g., pattern image (561a), pattern image (561b)) on the boundary surface (132) through the display (420) so that the user (110) can recognize the boundary area (130) in a three-dimensional space. The wearable device (101) can display the pattern image while displaying a virtual environment through the display (420).

[0110] According to one embodiment, the wearable device (101) may display a pattern image according to the location of the wearable device (101). For example, the first screen (521) represents an avatar of a user (110) of the wearable device (101) located at a first point (528a) within the boundary zone (130). At the first point (528a), the distance between the boundary zone (130) and the wearable device (101) may be a first distance (529a). The wearable device (101) may display a pattern image (561a) based on the first distance (529a). The second screen (523) represents an avatar of a user (110) of the wearable device (101) located at a second point (529b) within the boundary zone (130). At the second point (528b), the distance between the boundary area (130) and the wearable device (101) may be a second distance (529b). The wearable device (101) may display a pattern image (561b) based on the second distance (529b).

[0111] The pattern image (e.g., pattern image (561a), pattern image (561b)) displayed on the boundary surface (132) of the boundary area (130) may include a pattern in which a designated pattern (e.g., stripes) is repeated. As the user (110) approaches the boundary area (130), the wearable device (101) may set the density of the designated pattern to be higher. For example, as the user (110) approaches the boundary area (130), the wearable device (101) may generate the pattern image so that the density of the designated pattern becomes higher. For example, the first point (528a) may be closer to the boundary area (130) than the second point (529b). The first distance (529a) may be longer than the second distance (529b). The pattern image (561a) may include a first stripe. The pattern image (561b) may include a second stripe. The spacing between lines in the second stripe may be smaller than the spacing between lines in the first stripe. As the user (110) approaches the boundary area (130), the spacing between lines in the stripes becomes more densely displayed, so that the user (110) of the wearable device (101) can recognize that the wearable device (101) is closer to the boundary area (130).

[0112] A user (110) may bring at least a part of the user's (110) body close to a boundary zone (130). For example, the user (110) may wear a wearable device (101), and the user's (110) hand may hold a controller (590) connected to the wearable device (101). The wearable device (101) may obtain information about the movement of the user's (110) hand through the movement of the controller (590). According to one embodiment, the wearable device (101) may display a pattern image according to the position of the controller (590). When the position of the controller (590) is within a second threshold distance from the boundary zone (130), the wearable device (101) may display a pattern image to indicate an interaction between the controller (590) and the boundary zone (130). As the position of the controller (590) gets closer to the boundary area (130), the wearable device (101) can connect the lines within the stripes of the pattern image (e.g., pattern image (561b)) to more effectively represent the interaction. The wearable device (101) can display a shape (e.g., a rectangle) that includes the lines.

[0113] For example, the third screen (525) represents at least a part (570) (e.g., a hand) of the avatar of the user (110) located within a second threshold distance from the boundary zone (130). The wearable device (101) may display a pattern image (561c) based on identifying that the location of the controller (590) is within the second threshold distance from the boundary zone (130). The wearable device (101) may display a shape (e.g., a square) including lines by connecting lines located at a certain distance from a point indicated by the direction of the controller (590) among the shapes of the pattern image being displayed (e.g., the pattern image (561b)). The wearable device (101) may display a pattern image (561c) including a shape displayed in bolder color than individual lines based on the connection. For example, the fourth screen (527) represents at least a portion (570) (e.g., a hand) of the avatar of the user (110) touching the boundary area (130). The wearable device (101) may display a pattern image (561d) based on identifying that the controller (590) is positioned over the boundary area (130). The wearable device (101) may connect lines of the pattern images being displayed (e.g., pattern images (561b, pattern images (561c))) that are positioned at a certain distance from a point indicated by the direction of the controller (590). As the lines are connected, a shape (e.g., a square) may be formed. The wearable device (101) may display a pattern image (561d) that includes a shape (e.g., a square) that is displayed in bolder color than individual lines based on the connection. In the fourth screen (527), the wearable device (101) can connect more lines than in the third screen (527) in order to express the interaction more strongly.Accordingly, the size of the shape displayed on the area corresponding to the position of the controller (590) on the fourth screen (527) may be larger than the size of the shape displayed on the area corresponding to the position of the controller (590) on the third screen (527).

[0114] In FIGS. 5A and 5B , examples are described in which a pattern image is displayed differently in a boundary zone (130) depending on the position of the wearable device (101) and / or the position of the controller (590). A user (110) may want to know about the actual environment outside the boundary zone (130) by bringing at least a part of the user's (110) body (e.g., a hand) close to the boundary zone (130). In order to display an external image corresponding to the actual environment according to the intention of the user (110), the wearable device (101) may perform a pass-through operation. Hereinafter, an example of performing the pass-through operation is described through FIG. 5C .

[0115] Referring to FIG. 5C, the wearable device (101) may display a pattern image to indicate a boundary zone (130). The user (110) of the wearable device (101) may move. According to one embodiment, as the position of the wearable device (101) approaches the boundary zone (130) (e.g., the boundary surface (132) of the boundary zone (130), the pattern image displayed on the boundary surface (132) may change. According to one embodiment, as the position of not only the wearable device (101) but also the controller (590) approaches the boundary zone (130) (e.g., the boundary surface (132) of the boundary zone (130), not only the pattern image but also an external image (585) according to a pass-through operation may be displayed on the boundary surface (132). For example, a user (110) of a wearable device (101) may be moving closer to a boundary area (130). A pattern image (562a) of a first screen (531), a pattern image (562b) of a second screen (533), a pattern image (562c) of a third screen (535), and a pattern image (562d) of a fourth screen (537) may be sequentially displayed on the boundary area (132).

[0116] In the third screen (535) and the fourth screen (537), the controller (590) connected to the wearable device (101) may be within a second threshold distance from the boundary zone (130). As a non-limiting example, if the position of the controller (590) is within the second threshold distance from the boundary zone (130), regardless of whether the position of the controller (590) is beyond the boundary surface (132) or within the boundary surface (132), the wearable device (101) may display at least a portion of the external image (585) through a virtual area corresponding to the position of the controller (590). The wearable device (101) may perform a pass-through operation based on the position of the controller (590). The pass-through operation may include an operation of displaying at least a portion of the external image (585) corresponding to the FOV of the user (110) while displaying a virtual image of a virtual environment. The external image (585) can be acquired through a camera (e.g., camera (425)) of the wearable device (101). The wearable device (101) can determine a virtual area on which to display at least a portion of the external image (585) acquired through the camera. By displaying at least a portion of the external image (585) on the virtual area through the display (e.g., display (420)), the wearable device (101) can provide the user (110) with not only a VR experience but also an AR (augmented-reality) and / or MR (mixed reality) experience.

[0117] The wearable device (101) may perform rendering to obtain a screen to be stored in a display buffer and displayed through a display. For example, the wearable device (101) may place an image layer corresponding to an external image (585) on an image layer corresponding to the virtual area (580) to display at least a portion of the external image (585) in the virtual area (580). The wearable device (101) may perform rendering to display the virtual area (580) and the external image (585) in an overlapping manner on a virtual image of a virtual environment, thereby obtaining a third screen (535) and a fourth screen (537). The external image (585) may be referred to as a pass-through image from the perspective that the surrounding environment of the wearable device (101) is shown to the user (110) by passing through the virtual image and pattern image (e.g., pattern image (562c), pattern image (562d)) of the virtual environment. The external image (585) may also be referred to as a punch-through image from the perspective that at least a portion of the external image (585) is shown to the user (110) by penetrating through at least a portion of the virtual image and pattern image (e.g., pattern image (562c), pattern image (562d)) of the virtual environment. In addition to the virtual area, the virtual area (580) may also be referred to as a virtual surface, a punch-through area, a virtual surface, a punch-through surface, a pass-through area, a pass-through surface, a viewing surface, a viewing area, a transition surface, a transition space, a transition area, a passing surface, a passing area, or terms having equivalent technical meanings thereto.

[0118] According to one embodiment, the wearable device (101) can determine the size of the virtual area (580) based on the position of the controller (590). While the virtual image and pattern image of the virtual environment are displayed, the wearable device (101) can display at least a portion of an external image in the virtual area (580) according to the determined size through the display (420). For example, in the third screen (535), the wearable device (101) can display at least a portion of the external image (585) on the virtual area (580a). In the fourth screen (537), the wearable device (101) can display at least a portion of the external image (585) on the virtual area (580b). In the fourth screen (537), the controller (590) is closer to the boundary area (130) than the controller (590) in the third screen (535), so the virtual area (580b) may be wider than the virtual area (580a). The size of the virtual area (580b) may be larger than the size of the virtual area (580a).

[0119] According to one embodiment, the wearable device (101) can determine the location of the virtual area (580) based on the location of the controller (590). The wearable device (101) can obtain information about the location of the controller (590) based on the user's (110) intention to obtain information about the actual environment. The wearable device (101) can create a virtual area (580) at a location corresponding to the controller (590) on the boundary surface (132) of the boundary zone (130) (e.g., a location including a point where a straight line from the controller (590) touches the boundary surface (132). While displaying the virtual environment and pattern image through the display (420), the wearable device (101) can display at least a portion of an external image in the virtual area.

[0120] In FIGS. 5A to 5C, only one controller (590) is illustrated as being connected to the wearable device (101), but embodiments of the present disclosure are not limited thereto. The number of controllers connected to the wearable device (101) may be two or more. For example, the controllers connected to the wearable device may include a first controller for the left hand and a second controller for the right hand. If at least one controller among the controllers is within a second threshold distance from the boundary area (130), at least a portion of an external image according to a pass-through operation may be displayed in a virtual area according to the position of the at least one controller.

[0121] Figure 6 illustrates an example of a virtual area and sound control for an external image. The same reference numbers may indicate the same description in other drawings.

[0122] Referring to FIG. 6, the wearable device (101) can display a pattern image (660) according to the location of the wearable device (101). For example, the first screen (610) shows an avatar of a user (110) of the wearable device (101) located at a first point (618a) within the boundary zone (130). Based on the distance between the wearable device (101) and the boundary zone (130) at the first point (618a), the wearable device (101) can display a pattern image (660a). The pattern image (660a) can include a first grid composed of circles. For example, the second screen (620) shows an avatar of a user (110) of the wearable device (101) located at a second point (618b) within the boundary zone (130). Based on the distance between the wearable device (101) and the boundary area (130) at the second point (618b), the wearable device (101) may display a pattern image (660b). The pattern image (660b) may include a second grid composed of circles. The spacing between the circles of the second grid may be narrower than the spacing between the circles of the first grid.

[0123] The wearable device (101) may display at least a portion (685) of an external image (585) on the virtual area (580) based on the position of the wearable device (101) and / or the position of the controller (590). According to one embodiment, the wearable device (101) may display a virtual image of a virtual environment. The virtual environment may provide sound effects. The wearable device (101) may output sound signals to provide a sense of immersion in the virtual environment while an application corresponding to the virtual environment is running. While the pattern image (660a) is displayed, the virtual area (580) is not generated, so the user (110) may feel that it is a closed virtual space. The wearable device (101) at the first point (618a) may output a first sound signal having a first volume (611). On the other hand, while the pattern image (660b) is displayed, since at least a part (685) of the external image (585) is displayed on the virtual area (580), the user (110) can recognize that he is getting closer to the real environment. The wearable device (101) at the second point (618b) can output a second sound signal having a second volume (612). As at least a part (685) of the external image (585) corresponding to the real environment is exposed, the sense of immersion in the virtual environment can be lowered. The wearable device (101) can reduce the volume to inform the user (110) that he is getting closer to the real environment. For example, the second volume (612) of the second sound signal can be lower than the first volume (611) of the first sound signal. The wearable device (101) can set a second volume (612) lower than the first volume (611) of the first sound signal while at least a portion (685) of an external image (585) is displayed through the virtual area (580), and output the second sound signal according to the second volume (612).As another example, the wearable device (101) may turn off the output of sound signals while at least a portion (685) of an external image (585) is displayed through the virtual area (580). The size of the second volume (612) may be '0'.

[0124] In FIG. 6, an example of outputting sounds related to a virtual environment in an application for providing a virtual environment is described. However, in addition to simply providing sounds of a virtual environment, it may also be understood as an embodiment of the present disclosure to output sounds related to a real environment by combining them with sounds related to the virtual environment. The wearable device (101) may mix a sound signal related to the virtual environment (hereinafter, referred to as a first sound signal) and a sound signal related to the real environment (hereinafter, referred to as a second sound signal) to output a synthesized sound signal. For example, as the user (110) approaches closer to the boundary area (130), the wearable device (101) may change the synthesis ratio of the first sound signal and the second sound signal. In order to inform the user (110) that he / she is getting closer to the real environment, the wearable device (101) may increase a weight to be applied to the second sound signal and decrease a weight to be applied to the first sound signal. As another example, as the user (110) moves away from the boundary area (130), the wearable device (101) may change the synthesis ratio of the first sound signal and the second sound signal. To inform the user (110) that he or she is moving away from the actual environment, the wearable device (101) may lower the weight applied to the second sound signal and increase the weight applied to the first sound signal.

[0125] Figure 7 illustrates an example of displaying a pattern image according to the status of a user (e.g., user (110)). The same reference numbers may indicate the same description in other drawings.

[0126] Referring to FIG. 7, the wearable device (101) can obtain information about the status of the user (110). For example, the wearable device (101) can obtain information about the status of the user (110) (e.g., heart rate, body temperature) through one or more sensors. For example, the wearable device (101) can obtain information about the status of the user (110) through data from an external electronic device. For example, the wearable device (101) can obtain information about the status of the user (110) through a camera. For example, the wearable device (101) can obtain information about the status of the user (110) (e.g., wearing time) based on an internal timer.

[0127] The wearable device (101) can provide information about the status of the user (110) through a visual effect on the boundary zone (130). For example, the wearable device (101) can determine the color of the boundary zone (130) based on information about the status of the user (110). For example, the wearable device (101) can determine the color (e.g., green, yellow, red) of the boundary zone (130) based on the heart rate or body temperature of the user (110). According to one embodiment, the wearable device (101) can obtain the heart rate over a certain period of time. For example, the wearable device (101) can apply a designated visual effect (e.g., green) to the boundary zone (130) to indicate that the heart rate of the user (110) is below the specific heart rate. For example, the wearable device (101) may apply a designated visual effect (e.g., red) to the boundary area (130) to indicate that the heart rate of the user (110) is above the specified heart rate. Through the designated visual effect (e.g., red), the wearable device (101) may inform the user to pause use of the wearable device (101) and take a rest.

[0128] For example, the wearable device (101) may determine the thickness of the boundary line (131) of the boundary zone (130) based on information about the state of the user (110). For example, the wearable device (101) may determine the thickness of the boundary line (131) (e.g., indicating the thickness level in a range from 1 to 10) based on the heart rate or body temperature of the user (110). According to one embodiment, the wearable device (101) may provide information about the physical state of the user (110) based on the thickness of the boundary zone (130). For example, the wearable device (101) may display the boundary zone (130) through a first visual effect on the first screen (710). For example, the thickness of the boundary zone (130) may be 10. On the second screen (720), the boundary area (130) can be displayed using a second visual effect. For example, the thickness of the boundary area (130) can be 5. The wearable device (101) can display the boundary area (130) using a first visual effect on the third screen (730). For example, the thickness of the boundary area (130) can be 1.

[0129] For example, the wearable device (101) may determine an animation effect of the boundary zone (130) based on information about the state of the user (110). For example, the wearable device (101) may determine an animation effect (e.g., a blinking frequency) of the boundary zone (130) based on the time that the user (110) wears the wearable device (101). According to one embodiment, the wearable device (101) may obtain information about body temperature. For example, the wearable device (101) may apply a designated visual effect (e.g., 2 blinks per second) to the boundary zone (130) to indicate that the body temperature of the user (110) is within a stable range. The wearable device (101) may apply a designated visual effect (e.g., 5 blinks per second) to the boundary zone (130) to indicate that the body temperature of the user (110) is above a threshold. Through the above-mentioned visual effect, the wearable device (101) can guide the user to pause use of the wearable device (101) and take a break.

[0130] FIG. 8 illustrates an example of a notification message for notifying a user (e.g., user (110)) of a status. Prolonged use of a wearable device (101) may cause excessive fatigue to the user (110). Below, FIG. 8 illustrates an example of a message for delivering an alert to the wearable device (101). The same reference numerals may indicate the same description in other drawings.

[0131] Referring to FIG. 8, on the screen (800), the wearable device (101) can display a boundary line (131) of a boundary area (130) and display a pattern image (860) on the boundary surface (132).

[0132] The wearable device (101) can obtain information about the status of the user (110). The wearable device (101) can obtain information about the usage time of the user (110). For example, the wearable device (101) can obtain information about the time the user (110) wears the wearable device (101) based on an internal timer. For example, the wearable device (101) can obtain information about the time the user (110) experiences the virtual environment based on the execution time of an application providing the virtual environment.

[0133] The wearable device (101) may display a notification message (815) (e.g., display (420)) based on identifying that the usage time is greater than a threshold. For example, the notification message (815) may include text (801) (e.g., "Take a rest") to inform that a rest is needed. Additionally, the notification message (815) may include a button (803) for closing the notification message (815). In one embodiment, in response to a user input (e.g., a button input on the wearable device (101) or an input on the controller (590)) for the button (803), the wearable device (101) may end the display of the notification message (815). The wearable device (101) may continue to display a virtual image and a pattern image (860) of a virtual environment. Additionally, the notification message (815) may include a button (805) for displaying a pass-through screen. In one embodiment, in response to a user input for the button (805) (e.g., a button input on the wearable device (101) or an input on the controller (590), the wearable device (101) may terminate the display of the virtual image of the virtual environment and the display of the pattern image (860). The wearable device (101) may perform a pass-through operation. The wearable device (101) may display an external image acquired through a camera (e.g., camera (425)).

[0134] Figures 9a to 9e illustrate examples of visual objects for adjusting the size of a virtual area (e.g., virtual area (580)). The wearable device (101) may display at least a portion of an external image (585) on the virtual area (580) based on the position of the wearable device (101) and / or the position of a controller (590) connected to the wearable device (101). Examples for controlling the size of the virtual area (580) in a situation where the external image (585) is displayed are described. The same reference numerals may indicate the same description in different drawings.

[0135] Referring to FIG. 9a, the wearable device (101) can create a virtual area (580) at a boundary surface (132) of a boundary zone (130). The wearable device (101) can display at least a portion (975a) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101) through the virtual area (580). A user of the wearable device (101) may want to increase or decrease the size of the virtual area (580). On the first screen (910a), the wearable device (101) can display at least a portion (975a) of the external image (585) through the virtual area (580) while displaying a pattern image (960a) including circles and a virtual environment.

[0136] According to one embodiment, the wearable device (101) may receive a first user input for the virtual area (580). The first user input may be used to display a visual object (e.g., a first visual object (971a), a second visual object (972a), a third visual object (973a), a fourth visual object (974a)) for adjusting an area of ​​the virtual area (580). For example, when the gaze information of the user (110) is directed toward the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. For another example, when the position of the controller (590) remains in the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. For another example, the wearable device (101) may initiate display of the visual object through a user input using a controller other than the controller (590) used to display the virtual area (580).

[0137] According to one embodiment, the wearable device (101) may display visual objects (e.g., a first visual object (971a), a second visual object (972a), a third visual object (973a), a fourth visual object (974a)) for adjusting the area of ​​the virtual area (580) (e.g., adjusting the size, adjusting the position) in response to the first user input. The visual objects may be used as soft handlers for adjusting the area of ​​the virtual area (580). For example, the wearable device (101) may display visual objects (e.g., a first visual object (971a), a second visual object (972a), a third visual object (973a), a fourth visual object (974a)) for controlling the size of the virtual area (580) around the virtual area (580).

[0138] In one embodiment, the wearable device (101) may adjust the size of the virtual area (580) based on a second user input on the visual object. For example, the user (110) may select a visual object (e.g., a first visual object (971a), a second visual object (972a), a third visual object (973a), and a fourth visual object (974a)) through the controller (590) and then move the controller (590) away from the selected visual object in the virtual area (580). The wearable device (101) may obtain an input for the selection and track the direction of the movement, thereby recognizing that the user's (110) intention is to expand the size of the virtual area (580). The wearable device (101) may change the size of the virtual area (580) based on the input of the visual object through the controller (590) and the movement of the controller (590). For example, the size of the virtual area (580) can be changed from a first size to a second size. The second size can be larger than the first size. In this way, the user (110) of the wearable device (101) can adjust the size of the virtual area (580) not only through one visual object but also through multiple visual objects. For example, the wearable device (101) can increase the size of the virtual area (580) through a gesture of spreading the first visual object (971a) and the third visual object (973a) in different directions. For example, the wearable device (101) can also decrease the size of the virtual area (580) through a gesture of pinching the second visual object (972a) and the fourth visual object (974a) in different directions.

[0139] Referring to FIG. 9b, the wearable device (101) can create a virtual area (580) at the boundary surface (132) of the boundary zone (130). The wearable device (101) can display at least a portion (975b) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101) through the virtual area (580). A user of the wearable device (101) may want to increase or decrease the size of the virtual area (580). On the second screen (910b), the wearable device (101) can display at least a portion (975b) of the external image (585) through the virtual area (580) while displaying a pattern image (960b) including lines and a virtual environment.

[0140] According to one embodiment, the wearable device (101) may receive a first user input for the virtual area (580). The first user input may be used to display a visual object (e.g., a first visual object (971b), a second visual object (972b)) for adjusting the area of ​​the virtual area (580). For example, when the gaze information of the user (110) is directed toward the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, when the position of the controller (590) remains in the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, the wearable device (101) may initiate display of the visual object through user input using a controller other than the controller (590) used to display the virtual area (580).

[0141] According to one embodiment, the wearable device (101) may display a visual object for area adjustment (e.g., size adjustment, position adjustment) of the virtual area (580) in response to the first user input. The visual object may be used as a handler for area adjustment of the virtual area (580). For example, the wearable device (101) may display a visual object (e.g., a first visual object (971b), a second visual object (972b)) for controlling the size of the virtual area (580) around the virtual area (580).

[0142] In one embodiment, the wearable device (101) may adjust the size of the virtual area (580) based on a second user input on the visual object. For example, the user (110) may select a visual object (e.g., a first visual object (971b), a second visual object (972b)) through the controller (590) and then move the controller (590) away from the selected visual object in the virtual area (580). The wearable device (101) may obtain an input for the selection and track the direction of movement, thereby recognizing that the user's (110) intention is to expand the size of the virtual area (580). The wearable device (101) may change the size of the virtual area (580) based on the input of the visual object through the controller (590) and the movement of the controller (590). For example, the size of the virtual area (580) may be changed from a first size to a second size. The second size may be larger than the first size. In this way, the user (110) of the wearable device (101) can adjust the size of the virtual area (580) not only through one visual object but also through multiple visual objects. For example, the wearable device (101) can increase the size of the virtual area (580) through a gesture that spreads the first visual object (971b) and the second visual object (972b) in different directions. For example, the wearable device (101) can also decrease the size of the virtual area (580) through a gesture that pinches the first visual object (971b) and the second visual object (972b) in different directions.

[0143] Referring to FIG. 9c, the wearable device (101) can create a virtual area (580) at the boundary surface (132) of the boundary zone (130). The wearable device (101) can display at least a portion (975c) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101) through the virtual area (580). In the third screen (910c), the wearable device (101) can display at least a portion (975c) of the external image (585) through the virtual area (580) while displaying a pattern image (960c) including lines and a virtual environment. The wearable device (101) may display visual objects in the form of a fence (e.g., a first portion (987a), a second portion (987b), and a third portion (987c)) to provide a more intuitive and realistic user experience to the user (110). For example, the wearable device (101) may display the first portion (987a) and the third portion (987c) corresponding to the fence on the boundary surface (132), and display the second portion (987b) corresponding to the gate of the fence. A portion of the external image (585) may be hidden from the user (110) through the second portion (987b).

[0144] According to one embodiment, the wearable device (101) may receive a first user input for the virtual area (580). The first user input may be used to display a visual object (e.g., a visual object (976)) for adjusting the area of ​​the virtual area (580). For example, when the gaze information of the user (110) is directed toward the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, when the position of the controller (590) remains in the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, the wearable device (101) may initiate display of the visual object through user input using a controller other than the controller (590) used to display the virtual area (580).

[0145] In one embodiment, the wearable device (101) may display a visual object (976) for adjusting the area (e.g., resizing, positioning) of the virtual area (580) in response to the first user input. The wearable device (101) may further display a visual object (976) for guiding the user input of the user (110). For example, the wearable device (101) may display an arrow object indicating the direction of opening the fence gate as the visual object (976).

[0146] In one embodiment, the wearable device (101) may adjust the size of the virtual area (580) based on a second user input on the visual object. For example, the user (110) may select a visual object (e.g., the visual object (976) or the second portion (987b)) through the controller (590) and then move the controller (590) away from the selected visual object in the virtual area (580). The wearable device (101) may obtain an input for the selection and track the direction of the movement, thereby recognizing that the user's (110) intention is to expand the size of the virtual area (580). For example, a portion of the external image (585) that is obscured by the second portion (987b) may be displayed in response to the second user input.

[0147] Referring to FIG. 9d, the wearable device (101) can create a virtual area (580) at the boundary surface (132) of the boundary zone (130). The wearable device (101) can display at least a portion (975d) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101) through the virtual area (580). In the fourth screen (910d), the wearable device (101) can display at least a portion (975d) of the external image (585) through the virtual area (580) while displaying a pattern image (960d) including lines and a virtual environment. The wearable device (101) may display visual objects (e.g., a first portion (988a), a second portion (988b), and a third portion (988c)) for a sliding door to provide a more intuitive and realistic user experience to the user (110). For example, the wearable device (101) may display a first portion (987a) and a third portion (987c) corresponding to a wall on the boundary surface (132), and a second portion (988b) corresponding to a sliding door mounted on the wall. At least a portion (975d) of an external image (585) may be displayed to the user (110) through the second portion (988b) and the third portion (988c).

[0148] According to one embodiment, the wearable device (101) may receive a first user input for the virtual area (580). The first user input may be used to display a visual object (e.g., a visual object (977)) for adjusting the area of ​​the virtual area (580). For example, when the gaze information of the user (110) is directed toward the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, when the position of the controller (590) remains in the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, the wearable device (101) may initiate display of the visual object through user input using a controller other than the controller (590) used to display the virtual area (580).

[0149] According to one embodiment, the wearable device (101) may display a visual object (977) for adjusting the area (e.g., adjusting the size, adjusting the position) of the virtual area (580) in response to the first user input. The wearable device (101) may further display a visual object (977) for guiding the user input of the user (110). For example, the wearable device (101) may display an arrow object indicating the direction in which to open a sliding door as the visual object (977).

[0150] In one embodiment, the wearable device (101) may adjust the size of the virtual area (580) based on a second user input on the visual object. For example, the user (110) may select a visual object (e.g., the visual object (977) or the second portion (988b)) through the controller (590) and then move the controller (590) away from the selected visual object in the virtual area (580). The wearable device (101) may obtain an input for the selection and track the direction of the movement, thereby recognizing that the user's (110) intention is to expand the size of the virtual area (580). For example, in response to the second user input, as the second portion (988b) moves further to the left with respect to the user (110), the size of a portion of the external image (585) displayed to the user (110) may increase. In response to the second user input, the size of the virtual area (580) can be changed from a first size to a second size. The second size can be larger than the first size.

[0151] Although FIG. 9d illustrates an example of a user input for increasing the size of a virtual area (580) by pushing a sliding door, embodiments of the present disclosure are not limited thereto. The wearable device (101) may acquire a gesture for pulling a sliding door (e.g., the second portion (988b)) through an input means (e.g., a controller (590) or a body part (e.g., a hand)). In response to the gesture, the wearable device (101) may display a virtual area (580) having a reduced size.

[0152] Referring to FIG. 9e, the wearable device (101) can create a virtual area (580a) on the boundary surface (132) of the boundary zone (130). The wearable device (101) can display at least a portion (995a) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101) through the virtual area (580a). On the first screen (920a), the wearable device (101) can display a visual pattern (996a) to indicate the boundary surface (132). In order to provide a more intuitive and real-world user experience to the user (110), the visual pattern (996a) can include a curtain-shaped image. The wearable device (101) can display at least a portion (995a) of an external image (585) through a virtual area (580a) while displaying a visual pattern (996a).

[0153] According to one embodiment, the wearable device (101) may display visual objects (e.g., a first visual object (997a), a second visual object (997b)) for area adjustment (e.g., size adjustment, position adjustment) of a virtual area (580a). The visual objects may be used to guide user input of a user (110). For example, the visual objects may include an arrow object representing a curtain-pulling gesture.

[0154] According to one embodiment, the wearable device (101) can adjust the size of the virtual area (580a) based on a second user input on the visual object. For example, the user (110) can select a visual object (e.g., a first visual object (997a), a second visual object (997b)) through the controller (590) and then move the controller (590) away from the selected visual object in the virtual area (580a). The wearable device (101) can recognize that the user's (110) intention is to expand the size of the virtual area (580a) by obtaining the input for the selection and tracking the direction of the movement. The size of the virtual area (580a) can be expanded.

[0155] In response to the second user input, the wearable device (101) may display a second screen (920b). In the second screen (920b), the wearable device (101) may create a virtual area (580b) on the boundary surface (132) of the boundary zone (130). The wearable device (101) may display, through the virtual area (580b), at least a portion (995b) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101). In the second screen (920b), the wearable device (101) may display a visual pattern (996b) to indicate the boundary surface (132). Like the first screen (920a), the visual pattern (996b) may include a curtain-shaped image. The wearable device (101) may display at least a portion (995b) of an external image (585) through a virtual area (580b) while displaying a visual pattern (996b). The size of the virtual area (580b) may be larger than the size of the virtual area (580a).

[0156] According to one embodiment, the wearable device (101) can adjust the size of the virtual area (580b). For example, the user (110) can select a visual object (e.g., a first visual object (997a), a second visual object (997b)) through the controller (590) and then move the controller (590) away from the selected visual object to the virtual area (580b). The wearable device (101) can recognize that the user's (110) intention is to expand the size of the virtual area (580b) by obtaining an input for the selection and tracking the direction of the movement. The size of the virtual area (580b) can be expanded.

[0157] The wearable device (101) can display a third screen (920c). In the third screen (920c), the wearable device (101) can create a virtual area (580c) on the boundary surface (132) of the boundary zone (130). The wearable device (101) can display at least a portion (995c) of an external image (585) acquired through a camera (e.g., camera (425)) of the wearable device (101) through the virtual area (580c). In the third screen (920c), the wearable device (101) can display a visual pattern (996b) to indicate the boundary surface (132). Like the first screen (920a) and the second screen (920b), the visual pattern (996c) can include a curtain-shaped image. The wearable device (101) may display at least a portion (995c) of an external image (585) through a virtual area (580c) while displaying a visual pattern (996c). The size of the virtual area (580b) may be larger than the size of the virtual area (580c).

[0158] To provide a feeling of continuous expansion to the user (110), in response to a single gesture, the wearable device (101) may sequentially display virtual areas of various sizes. The wearable device (101) may gradually increase the size of the virtual area. For example, the wearable device (101) may be connected to two controllers (e.g., a first controller for the left hand and a second controller for the right hand). The wearable device (101) may select a first visual object (997a) through the first controller and a second visual object (997a) through the second controller. After the selection, the user (110) may move the first controller to the left and the second controller to the right. The wearable device (101) may sequentially adjust the size of the virtual area based on the movements of the first controller and the second controller. The sizes of the virtual area (580a), the virtual area (580b), and the virtual area (580c) may gradually increase in that order. Based on the increasing size of the virtual area, the size of the visual pattern representing the boundary surface (132) may decrease. The wearable device (101) may reduce the size of the visual pattern so that the user (110) sees the shape of a curtain being opened. The wearable device (101) may display the visual pattern (996a), the visual pattern (996b), and the visual pattern (996c) in that order.

[0159] Although examples of adjusting the size of the virtual area (580) are described in FIGS. 9A and 9E , embodiments of the present disclosure are not limited thereto. According to one embodiment, the wearable device (101) can change the location of the virtual area (580). For example, the wearable device (101) can move the location of the virtual area (580) through user input for visual objects located around the virtual area (580). For example, as illustrated in FIG. 9B , the location of the virtual area (580) can be changed through a gesture of moving the first visual object (971b) and the second visual object (972b) in the same direction (e.g., to the right).

[0160] Figures 10a and 10b illustrate examples of interactions between a boundary zone (e.g., boundary zone (130)) and a user (e.g., user (110)) using multiple virtual areas. A wearable device (101) may be connected to multiple controllers (e.g., a first controller, a second controller). The same reference numbers may indicate the same description in different drawings.

[0161] Referring to FIG. 10A, the first screen (1010a) illustrates an example in which multiple virtual areas are displayed. The wearable device (101) can display a virtual environment. While the virtual environment is displayed, the wearable device (101) can display a pattern image (1060a) (e.g., a grid in which circles are repeatedly arranged). The wearable device (101) can display a boundary line (131) of a boundary area (130) and display a pattern image (1060a) on a boundary surface (132). While displaying the virtual environment and pattern image (1060a), the wearable device (101) can display a first virtual area (1071a) and a second virtual area (1072a). A first virtual area (1071a) may be generated by a first controller connected to a wearable device (101) located within a second threshold distance from a boundary area (130). A second virtual area (1072a) may be generated by a second controller connected to a wearable device (101) located within a second threshold distance from the boundary area (130). The location of the second virtual area (1072a) may be different from the location of the first virtual area (1071a). While the virtual environment and pattern image (1060a) is displayed, the wearable device (101) may display at least a portion of an external image (585) through the first virtual area (1071a) and at least another portion of an external image (585) through the second virtual area (1072a).

[0162] The second screen (1020a) illustrates an example in which an extended virtual area (1073a) including a plurality of virtual areas is displayed. According to one embodiment, the wearable device (101) may determine whether to display the extended virtual area (1073a) based on the distance between the virtual areas. For example, the wearable device (101) may obtain information about the location of the first virtual area (1071a). The wearable device (101) may obtain information about the location of the second virtual area (1072a). The wearable device (101) may determine the distance between the location of the first virtual area (1071a) and the location of the second virtual area (1072a). The wearable device (101) may determine whether the distance is less than or equal to a threshold distance. The wearable device (101) may maintain the display of the first virtual area (1071a) and the second virtual area (1072a) when the distance is greater than or equal to the threshold distance. The wearable device (101) may display the extended virtual area (1073a) when the distance is less than the threshold distance. The extended virtual area (1073a) may include the first virtual area (1071a) and the second virtual area (1072a). As a non-limiting example, when the distance is less than the threshold distance, the wearable device (101) may generate the extended virtual area (1073a) after a certain period of time (e.g., N seconds, where N is a natural number) has elapsed from the time at which at least a portion of an external image is displayed through the first virtual area (1071a) and the second virtual area (1072a). The wearable device (101) can display at least a portion of an external image (585) through an extended virtual area (1073a) while a virtual environment and pattern image (1060a) is displayed.

[0163] Referring to FIG. 10b, the first screen (1010b) illustrates an example in which multiple virtual areas are displayed. The wearable device (101) can display a virtual environment. While the virtual environment is being displayed, the wearable device (101) can display a pattern image (1060b) (e.g., a stripe in which lines are repeatedly arranged). The wearable device (101) can display a boundary line (131) of a boundary area (130) and display a pattern image (1060b) on a boundary surface (132). While displaying the virtual environment and the pattern image (1060b), the wearable device (101) can display a first virtual area (1071b) and a second virtual area (1072b). The first virtual area (1071b) may be generated by a first controller connected to a wearable device (101) located within a second threshold distance from a boundary area (130). The second virtual area (1072b) may be generated by a second controller connected to a wearable device (101) located within a second threshold distance from the boundary area (130). The location of the second virtual area (1072b) may be different from the location of the first virtual area (1071b). The wearable device (101) may display at least a portion of the external image (585) through the first virtual area (1071b) and at least another portion of the external image (585) through the second virtual area (1072b) while the virtual environment and pattern image (1060b) is displayed.

[0164] The second screen (1020b) illustrates an example in which an extended virtual area (1073b) including a plurality of virtual areas is displayed. According to one embodiment, the wearable device (101) may determine whether to display the extended virtual area (1073b) based on the distance between the virtual areas. For example, the wearable device (101) may obtain information about the location of the first virtual area (1071b). The wearable device (101) may obtain information about the location of the second virtual area (1072b). The wearable device (101) may determine the distance between the location of the first virtual area (1071b) and the location of the second virtual area (1072b). The wearable device (101) may determine whether the distance is less than or equal to a threshold distance. The wearable device (101) may maintain the display of the first virtual area (1071b) and the second virtual area (1072b) when the distance is greater than or equal to the threshold distance. The wearable device (101) may display the extended virtual area (1073b) when the distance is less than the threshold distance. The extended virtual area (1073b) may include the first virtual area (1071b) and the second virtual area (1072b). As a non-limiting example, when the distance is less than the threshold distance, the wearable device (101) may generate the extended virtual area (1073b) after a certain period of time (e.g., N seconds, where N is a natural number) has elapsed from the time at which at least a portion of an external image is displayed through the first virtual area (1071b) and the second virtual area (1072b). The wearable device (101) can display at least a portion of an external image (585) through an extended virtual area (1073b) while a virtual environment and pattern image (1060b) are displayed. As if the user (110) were pulling back a curtain with both hands, the wearable device (101) can form the extended virtual area (1073b) through the first controller and the second controller.The wearable device (101) can display at least a portion of an external image (585) through an extended virtual area (1073b).

[0165] FIG. 11a illustrates an operation flow of a wearable device (e.g., wearable device (101)) for interaction with a boundary zone within a boundary zone (e.g., boundary zone (130)).

[0166] Referring to FIG. 11A, in operation (1101), a wearable device (101) (e.g., processor (410)) may obtain information about a boundary zone (130). The boundary zone (130) may be defined to guide a boundary within which a virtual environment is displayed to a user (110) of the wearable device (101). The wearable device (101) may set the boundary zone (130) based on user input and / or specified settings.

[0167] In operation (1103), the wearable device (101) can obtain the location of the wearable device (101). The wearable device (101) can obtain the location of the wearable device (101) with respect to the boundary zone (130) based on sensor data obtained from at least one sensor (e.g., sensor (430)) of the wearable device (101).

[0168] In operation (1105), the wearable device (101) may determine whether the location of the wearable device (101) is within a first threshold distance from the boundary zone (130) within the boundary zone (130). The wearable device (101) may be located within the boundary zone. The wearable device (101) may determine whether the location of the wearable device (101) is within the first threshold distance from the boundary zone (130). If the location of the wearable device (101) is within the first threshold distance from the boundary zone (130), the wearable device (101) may perform operation (1107). The wearable device (101) may perform operation (1103) again if the location of the wearable device (101) is not within the first threshold distance from the boundary area (130) (i.e., outside the first threshold distance). The wearable device (101) may continue displaying the virtual environment.

[0169] In operation (1107), the wearable device (101) may display a pattern image to indicate a boundary zone (130) while displaying a virtual environment. The wearable device (101) may display the pattern image on a boundary surface (132) of the boundary zone (130) to inform the user (110) of proximity to the boundary zone. The pattern image may include a visual pattern to indicate an area occupied by the boundary surface (132). For example, the pattern image may include a designated pattern (e.g., stripes). The spacing between lines in the stripes may be determined according to the position of the wearable device (101). The closer the position of the wearable device (101) is to the boundary zone (130), the shorter the spacing between the lines may be. In other words, the closer the position of the wearable device (101) is to the boundary area (130), the more densely the lines can be displayed on the boundary surface (132). For example, the pattern image may include a grid in which a designated shape is repeated. The density of the designated shape on the grid may be determined according to the position of the wearable device (101). The closer the position of the wearable device (101) is to the boundary area (130), the more densely the repeated shape can be arranged on the grid. For example, the closer the position of the wearable device (101) is to the boundary area (130), the larger the size of the repeated shape may be.

[0170] In operation (1109), the wearable device (101) can determine whether the position of an input means (e.g., a controller (590), at least a part of the body of the user (110) (e.g., a hand)) is within a second threshold distance from the boundary zone (130). For example, the input means may include the controller (590). As an example, the controller (590) may be a device that can be gripped by the hand of the user (110), or may be a device that can be worn on at least a part of the body of the user (110) (e.g., a ring, a bracelet, a glove, an armband, a wristband). The wearable device (101) can obtain information about the position of the controller (590). Based on the information about the position of the controller (590), the wearable device (101) can determine whether the controller (590) is located within the second threshold distance from the boundary surface (132) of the boundary zone (130). A controller (590) positioned within the second threshold distance from the boundary surface (132) may indicate an intention of a user (110) to interact with a real environment beyond the boundary zone (130). As another example, the input means may include at least a part of the body (e.g., a hand) of a user (110) of a wearable device (101). The wearable device (101) may acquire an image through a camera (e.g., camera (425)) and determine whether a designated object is included in the image. The designated object may correspond to a part of the body of the user (110). As an example, the wearable device (101) may detect an object corresponding to the hand of the user (110). The wearable device (101) can determine, based on information about a body part (e.g., a hand) of the user (110), whether the body part is located within the second threshold distance from the boundary surface (132) of the boundary zone (130).A part of the user's (110) body located within the second threshold distance from the boundary surface (132) may indicate the user's (110) intention to interact with the actual environment beyond the boundary zone (130).

[0171] If the position of the input means is within the second threshold distance from the boundary area (130), the wearable device (101) can perform operation (1111). If the position of the input means is not within the second threshold distance from the boundary area (130) (outside the second threshold distance), the wearable device (101) can perform operation (1109) again.

[0172] In operation (1111), the wearable device (101) may display at least a part of an external image (e.g., an external image (585)) in a virtual area (e.g., a virtual area (580)) according to the position of an input means (e.g., a controller (590), at least a part of the body of the user (110) (e.g., a hand)) while displaying a virtual environment. For example, the wearable device (101) may determine that the intention of the user (110) is to interact with the real environment beyond the boundary area (130) by moving the controller (590). For example, the wearable device (101) may determine that the intention of the user (110) is to interact with the real environment beyond the boundary area (130) by confirming that a part of the body of the user (110) is captured in the external image captured through the camera (425). The wearable device (101) can acquire an external image (585) corresponding to an actual environment through a camera (e.g., camera (425)). The wearable device (101) can determine a virtual area (580) corresponding to the position of the input means. The position of the input means can indicate an area of ​​the actual environment that the user (110) wants to see. The wearable device (101) can display at least a part of the external image (585) through the virtual area (580) while displaying the virtual environment and the pattern image.

[0173] FIG. 11B illustrates an operation flow of a wearable device (e.g., wearable device (101)) for adjusting a virtual area within a boundary area (e.g., boundary area (130)). The wearable device (101) may display a virtual area (e.g., virtual area (580)) when the position of the controller (590) and / or the position of the wearable device (101) is within a second threshold distance from the boundary area (130). In FIG. 11B, a situation is described in which at least a portion of an external image (585) is displayed through the virtual area (580) together with a virtual image of a virtual environment by the wearable device (101) positioned within the boundary area (130).

[0174] Referring to FIG. 11B, in operation (1151), the wearable device (101) (e.g., the processor (410)) may obtain a first user input for the virtual area (580). The first user input may be used to display a visual object (e.g., the first visual object (971a), the second visual object (972a), the third visual object (973a), the fourth visual object (974a) of FIG. 9A, and the first visual object (971b), the second visual object (972b) of FIG. 9B) for area adjustment of the virtual area (580). For example, when the gaze information of the user (110) is directed toward the virtual area (580) for a specified time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. In other words, if the gaze input of the user (110) is maintained for the specified time, the wearable device (101) can initiate display of the visual object. For another example, if the position of the controller (590) remains in the virtual area (580) for a specified time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) can initiate display of the visual object. For another example, the wearable device (101) can initiate display of the visual object through user input using a controller other than the controller (590) used to display the virtual area (580).

[0175] In operation (1153), the wearable device (101) may display a visual object for area adjustment (e.g., the first visual object (971a), the second visual object (972a), the third visual object (973a), and the fourth visual object (974a) of FIG. 9A, and the first visual object (971b) and the second visual object (972b) of FIG. 9B). The wearable device (101) may display the visual object in response to the first user input. The visual object may be used as a soft handler for area adjustment of the virtual area (580). The visual object may be arranged adjacent to the periphery of the virtual area (580). For example, the wearable device (101) may display a plurality of visual objects as a user interface for area adjustment. The plurality of visual objects may be displayed adjacent to the virtual area (580) to guide a user input.

[0176] In operation (1155), the wearable device (101) can adjust the size of the virtual area based on a second user input on the visual object. The wearable device (101) can obtain the second user input on the visual object. According to one embodiment, the wearable device (101) can obtain the second user input on the visual object based on the movement of the controller (590). For example, the wearable device (101) can select the plurality of visual objects through the controller (590) and expand the size of the virtual area (580) through a gesture of pulling the selected plurality of visual objects outward from the center. As another example, the wearable device (101) can expand the size of the virtual area (580) by moving one of the plurality of visual objects away from the virtual area (580) through the controller (590).

[0177] Although not illustrated in FIG. 11B, as the size of the virtual area (580) is adjusted, additional effects may be provided to the user (110). For example, as the size of the virtual area (580) increases, the volume of the sound signal for the virtual environment may decrease. The wearable device (101) may decrease the volume of the sound signal to increase the user's (110's) ability to focus on the external environment. In another example, as the size of the virtual area (580) decreases, the volume of the sound signal for the virtual environment may increase. The wearable device (101) may increase the volume of the sound signal to increase the user's (110's) ability to perceive the virtual environment.

[0178] In addition to simply adjusting the volume of a sound signal for a virtual environment, the wearable device (101) can output a sound signal for a real environment by synthesizing it with a sound signal for the virtual environment. The wearable device (101) can adaptively set a synthesis ratio of a sound signal for a real environment (hereinafter, a second sound signal) and a sound signal for a virtual environment (hereinafter, a first sound signal). For example, in order to inform the user (110) that he / she is getting closer to the real environment, the wearable device (101) can increase a weight to be applied to the second sound signal and decrease a weight to be applied to the first sound signal. In another example, in order to inform the user (110) that he / she is getting farther from the real environment, the wearable device (101) can decrease a weight to be applied to the second sound signal and increase a weight to be applied to the first sound signal.

[0179] Figures 12a and 12b illustrate examples of interactions between a boundary zone and a user (e.g., a user (110)) outside a boundary zone (e.g., a boundary zone (130)). Like reference numerals may indicate like descriptions in other drawings.

[0180] Referring to FIG. 12A, the first screen (1210) represents a situation in which a user (110) outside the boundary zone (130) is positioned within a first threshold distance from the boundary zone (130). When the wearable device (101) is positioned within the first threshold distance, the wearable device (101) may display a pattern image (1260) to indicate the boundary zone (130). The wearable device (101) may provide the boundary zone (130) to the user (110) while displaying an external image acquired through a camera (e.g., camera (425)). For example, the wearable device (101) may display a boundary line (131) while displaying the external image, and may display the pattern image (1260) on an external boundary surface (i.e., the opposite surface of the boundary surface (132)).

[0181] The second screen (1220) represents a situation in which a user (110) outside the boundary zone (130) is positioned within a first threshold distance from the boundary zone (130). The user (110) of the wearable device (101) may move the controller (e.g., the controller (590)) closer to the boundary zone (130) by moving the head of the user (110) or a part of the body (e.g., a hand) of the user (110). For example, the position of the controller (590) may be within the second threshold distance from the boundary zone (130). The wearable device (101) may generate a virtual area (1270) as the controller (590) approaches the boundary zone (130). The virtual area (1270) may be generated in an area corresponding to a direction in which the controller (590) faces on the external boundary surface (e.g., the opposite surface of the boundary surface (132)). The wearable device (101) can display at least a portion of an image inside the boundary zone (130) through the virtual area (1270). The inside of the boundary zone (130) may be a space for a virtual environment. The wearable device (101) can display at least a portion of a virtual image of the virtual environment through the virtual area (1270). While displaying the external image and the pattern image (1260), the wearable device (101) can display at least a portion of a virtual image of the virtual environment through the virtual area (1270).

[0182] Referring to FIG. 12B, the first screen (1251) represents a situation in which a user (110) outside the boundary zone (130) is positioned within a threshold distance from the boundary zone (130). For example, the user (110) of the wearable device (101) may move the head of the user (110) or a part of the body (e.g., a hand) of the user (110) to move a controller (e.g., a controller (590)) closer to the boundary zone (130). For example, the position of the controller (590) may be within a second threshold distance from the boundary zone (130). The wearable device (101) may display at least a portion of an image inside the boundary zone (130) through the virtual area (1270). The wearable device (101) may display at least a portion of a virtual image of a virtual environment through the virtual area (1270). In one embodiment, the application providing the virtual environment may not provide sounds for the virtual environment. The wearable device (101) may provide information about the virtual environment to the user (110) only through the virtual area (1270).

[0183] The second screen (1252) represents a situation where a user (110) outside the boundary zone (130) is located within a threshold distance from the boundary zone (130). The wearable device (101) can display at least a portion of a virtual image of a virtual environment through the virtual area (1270). Unlike the first screen (1251), the application providing the virtual environment can provide sound for the virtual environment. According to one embodiment, the wearable device (101) can not only display at least a portion of a virtual image of the virtual environment through the virtual area (1270), but also output a sound signal (1280) for the virtual environment. To create an effect of the sound signal (1280) flowing out through the virtual area (1270), the wearable device (101) can control the volume of the sound signal (1280). For example, the wearable device (101) can determine the volume of the sound signal (1280) based on the distance between the boundary zone (130) and the wearable device (101). The shorter the distance between the boundary zone (130) and the wearable device (101), the greater the volume of the sound signal (1280). For example, the wearable device (101) can determine the volume of the sound signal (1280) based on the size of the virtual area (1270). The larger the size of the virtual area (1270), the greater the volume of the sound signal (1280). In a virtual area (1270) of a first size, a sound signal according to a first volume may be output, and in a virtual area (1270) of a second size greater than the first size, a sound signal according to a second volume greater than the first volume may be output.

[0184] Figure 13 illustrates an example of displaying a boundary zone (130) according to an application outside a boundary zone (e.g., boundary zone (130)). The same reference numbers may indicate the same description in other drawings.

[0185] In FIG. 13, the first screen (1311) represents a situation in which a user (110) of the wearable device (101) is located outside a first threshold distance from a boundary zone (130). The wearable device (101) can acquire an external image through a camera (e.g., camera (425)) of the wearable device (101). The wearable device (101) can display the external image. While displaying the external image, the wearable device (101) can display an icon (e.g., a first icon (1331), a second icon (1332)) for at least one application (e.g., a video application, a music application). For example, the wearable device (101) can run an application (e.g., a video application) that was running for a virtual environment within the boundary zone (130) in the background outside the boundary zone (130). Outside the boundary zone (130), the wearable device (101) does not need to display the virtual environment of the application, so the wearable device (101) can display an icon corresponding to the application (e.g., the first icon (1331)) together with the external image. As a non-limiting example, the status of the application corresponding to the icon can be indicated through the border of the icon. For example, if the border of the icon is green, it can indicate that the application is running in the background. For example, if the border of the icon is red, it can indicate that the application is not running in the background.

[0186] The second screen (1312) represents a situation in which the user (110) of the wearable device (101) is closer to the boundary zone (130) than the position in the first screen (1311). The wearable device (101) can display an external image. While displaying the external image, the wearable device (101) can display an icon (e.g., the first icon (1331)) for an application (e.g., a video application) located in the field of view of the user (110). The wearable device (101) can display at least a part of the boundary zone (130) for a virtual environment provided through the application. For example, the wearable device (101) can display a boundary line (131) of the boundary zone (130). While displaying the external image, the wearable device (101) can display the boundary line (131). The wearable device (101) may preferentially display a boundary line (131) among the boundary zones (130) to gradually display the boundary zone (130) to the user (110).

[0187] The third screen (1313) represents a situation in which the user (110) of the wearable device (101) is closer to the boundary zone (130) than the position in the second screen (1312). For example, the wearable device (101) may be positioned within a first threshold distance from the boundary zone (130). The wearable device (101) may display the bottom of the boundary zone (130) through a boundary line (131). In addition, the wearable device (101) may display an outer boundary surface (1320) of the boundary zone (130). As a non-limiting example, the wearable device (101) may display the boundary zone (130) with a certain level of opacity, thereby displaying the boundary zone (130) together with an external image. As another example, the wearable device (101) may display a pattern image on the outer boundary surface (1320), as described with reference to FIGS. 12A and 12B .

[0188] FIG. 14a illustrates an operation flow of a wearable device (e.g., wearable device (101)) for interaction with a boundary zone from outside the boundary zone.

[0189] Referring to FIG. 14A, in operation (1401), a wearable device (101) (e.g., processor (410)) may obtain information about a boundary zone. The boundary zone (130) may be defined to guide a boundary within which a virtual environment is displayed to a user (110) of the wearable device (101). The wearable device (101) may set the boundary zone (130) based on user input and / or specified settings.

[0190] In operation (1403), the wearable device (101) can display an external image outside the boundary zone (130). The wearable device (101) can obtain the location of the wearable device (101) with respect to the boundary zone (130) based on sensor data obtained from at least one sensor (e.g., sensor (430)) of the wearable device (101). The wearable device (101) can be located outside the boundary zone (130). Therefore, since the wearable device (101) is outside the boundary zone (130), it can obtain an external image corresponding to an actual environment instead of a virtual environment through a camera (e.g., camera (425)). The wearable device (101) can display the obtained external image.

[0191] In operation (1405), the wearable device (101) can determine whether the location of the wearable device is within a first threshold distance from the boundary zone. The wearable device (101) can obtain the location of the wearable device (101) with respect to the boundary zone (130) based on sensor data obtained from at least one sensor (e.g., sensor (430)) of the wearable device (101). If the location of the wearable device (101) is within the first threshold distance from the boundary zone (130), the wearable device (101) can perform operation (1407). If the location of the wearable device (101) is not within the first threshold distance from the boundary zone (130) (i.e., outside the first threshold distance), the wearable device (101) can perform operation (1403) again.

[0192] In operation (1407), the wearable device (101) may display a pattern image to indicate a boundary area while displaying an external image. The wearable device (101) may display a pattern image to indicate a boundary area (130) while displaying a virtual environment. The wearable device (101) may display the pattern image on an outer boundary surface of the boundary area (130) to notify the user (110) of proximity to the boundary area. The pattern image may include a visual pattern to indicate an area occupied by the outer boundary surface. For example, the pattern image may include a designated pattern (e.g., stripes). The spacing between lines in the stripes may be determined according to the position of the wearable device (101). The closer the position of the wearable device (101) is to the boundary area (130), the shorter the spacing between the lines may be. In other words, the closer the position of the wearable device (101) is to the boundary zone (130), the denser the lines can be displayed on the outer boundary surface. For example, the pattern image may include a grid in which a designated shape is repeated. The density of the designated shape on the grid may be determined according to the position of the wearable device (101). The closer the position of the wearable device (101) is to the boundary zone (130), the denser the repeated shape can be arranged on the grid. For example, the closer the position of the wearable device (101) is to the boundary zone (130), the larger the size of the repeated shape can be.

[0193] In operation (1409), the wearable device (101) can determine whether the position of an input means (e.g., a controller (590), at least a part of the body of the user (110) (e.g., a hand)) is within a second threshold distance from the boundary zone. For example, the input means can include the controller (590). The wearable device (101) can obtain information about the position of the controller (590). Based on the information about the position of the controller (590), the wearable device (101) can determine whether the controller (590) is located within the second threshold distance from a boundary surface (132) of the boundary zone (130). A controller (590) located within the second threshold distance from an outer boundary surface of the boundary zone (130) can indicate an intention of the user (110) to interact with the virtual environment beyond the boundary zone (130). For another example, the input means may include at least a part of the body (e.g., a hand) of a user (110) of the wearable device (101). The wearable device (101) may acquire an image through a camera (e.g., camera (425)) and determine whether a specified object is included in the image. The specified object may correspond to a part of the body of the user (110). As an example, the wearable device (101) may detect an object corresponding to the hand of the user (110). Based on information about the part of the body (e.g., the hand) of the user (110), the wearable device (101) may determine whether the part is located within the second threshold distance from the boundary surface (132) of the boundary zone (130). A part of the user's (110) body located within the second threshold distance from the boundary surface (132) may indicate the user's (110) intention to interact with the actual environment beyond the boundary zone (130).

[0194] If the position of the input means is within the second threshold distance from the boundary area (130), the wearable device (101) can perform operation (1411). If the position of the input means is not within the second threshold distance from the boundary area (130) (outside the second threshold distance), the wearable device (101) can perform operation (1409) again.

[0195] In operation (1411), the wearable device (101) may display at least a portion of a virtual image of a virtual environment in a virtual area (e.g., virtual area (1270)) according to the position of the input means while displaying an external image (e.g., external image (585)). The wearable device (101) may determine, through movement of the controller (590), that the user's (110) intention is to interact with the virtual environment beyond the boundary area (130). The wearable device (101) may determine a virtual area (1270) corresponding to the position of the controller (590). The wearable device (101) may display at least a portion of a virtual image of the virtual environment through the virtual area (1270) while displaying the external image and the pattern image.

[0196] FIG. 14B illustrates an operation flow of a wearable device (e.g., wearable device (101)) for adjusting a virtual area outside a boundary area. The wearable device (101) may display a virtual area (e.g., virtual area (580)) when the position of the controller (590) and / or the position of the wearable device (101) is within a second threshold distance from the boundary area (130). In FIG. 14B, a situation is described in which at least a portion of a virtual image of a virtual environment is displayed through the virtual area (580) together with an external image (585) corresponding to a real environment by the wearable device (101) located outside the boundary area (130).

[0197] Referring to FIG. 14B, in operation (1451), the wearable device (101) (e.g., the processor (410)) may obtain a first user input for the virtual area. The first user input may be used to display a visual object for area adjustment of the virtual area (580). For example, when the gaze information of the user (110) is directed toward the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, when the position of the controller (590) remains in the virtual area (580) for a specified period of time (e.g., N seconds, where N is a predetermined natural number), the wearable device (101) may initiate display of the visual object. As another example, the wearable device (101) may initiate display of the visual object through user input using a controller other than the controller (590) used to display the virtual area (580).

[0198] In operation (1453), the wearable device (101) may display a visual object for region adjustment. In FIGS. 9A and 9B , examples of displaying the visual objects for region adjustment within the boundary zone (130) are illustrated, but the visual objects may also be displayed outside the boundary zone (130). The wearable device (101) may display the visual object for region adjustment on a boundary surface outside the boundary zone (130). The wearable device (101) may display the visual object in response to the first user input. The visual object may be used as a soft handler for region adjustment of the virtual region (580). The visual object may be arranged adjacent to the periphery of the virtual region (580). For example, the wearable device (101) may display a plurality of visual objects as a user interface for region adjustment. The above plurality of visual objects may be displayed adjacent to a virtual area (580) to guide user input.

[0199] In operation (1455), the wearable device (101) can adjust the size of the virtual area based on a second user input on the visual object. The wearable device (101) can obtain the second user input on the visual object. According to one embodiment, the wearable device (101) can obtain the second user input on the visual object based on the movement of the controller (590). For example, the wearable device (101) can select the plurality of visual objects through the controller (590) and expand the size of the virtual area (580) through a gesture of pulling the selected plurality of visual objects outward from the center. As another example, the wearable device (101) can expand the size of the virtual area (580) by moving one of the plurality of visual objects away from the virtual area (580) through the controller (590).

[0200] Although not illustrated in FIG. 14b, as the size of the virtual area (580) is adjusted, additional effects may be provided to the user (110). For example, as the size of the virtual area (580) expands, the volume of the sound signal for the virtual environment may increase. The wearable device (101) may increase the volume of the sound signal to enhance the user's (110) perception of the virtual environment. In another example, as the size of the virtual area (580) decreases, the volume of the sound signal for the virtual environment may decrease. The wearable device (101) may decrease the volume of the sound signal to enhance the user's (110) perception of the actual environment instead of the virtual environment.

[0201] In addition to simply adjusting the volume of a sound signal for a virtual environment, the wearable device (101) can output a sound signal for a real environment by synthesizing it with a sound signal for the virtual environment. The wearable device (101) can adaptively set a synthesis ratio of a sound signal for a real environment (hereinafter, a second sound signal) and a sound signal for a virtual environment (hereinafter, a first sound signal). For example, in order to inform the user (110) that he / she is getting closer to the virtual environment, the wearable device (101) can increase a weight to be applied to the first sound signal and decrease a weight to be applied to the second sound signal. In another example, in order to inform the user (110) that he / she is getting farther away from the virtual environment, the wearable device (101) can decrease a weight to be applied to the first sound signal and increase a weight to be applied to the second sound signal.

[0202] Figure 15 illustrates an example of a virtual area for an external object entering a boundary zone (130). The same reference numbers may indicate the same description in other drawings.

[0203] Referring to FIG. 15, on the first screen (1511), the wearable device (101) can display a virtual image (1510) of a virtual environment. An external object can approach the wearable device (101). For example, the external object can enter within a certain distance from a boundary area (130) set in the wearable device (101) (e.g., boundary line (131), boundary surface (132)). As the external object enters within the certain distance from the boundary area (130), the wearable device (101) can display a pattern image (1560).

[0204] On the second screen (1512), the wearable device (101) may display a pattern image (1560) along with a boundary line (131) while displaying a virtual image (1510). The pattern image (1560) may be displayed on the boundary surface (132) of the boundary area (130). An external object may approach closer to the wearable device (101).

[0205] In the third screen (1513), the wearable device (101) may generate a virtual area (1570). According to one embodiment, the wearable device (101) may generate the virtual area (1570) based on the gaze input of the user (110). For example, if the gaze input of the user (110) is directed to a specific location for a certain period of time, the wearable device (101) may generate a virtual area (1570) at the specific location. According to one embodiment, if it is detected that the external object is located within a second threshold distance from the boundary area (130), the wearable device (101) may generate a virtual area (1570) in an area corresponding to the external object. The wearable device (101) may display at least a portion of an image corresponding to the external object through the virtual area (1570).

[0206] In the fourth screen (1514), the wearable device (101) can create a virtual area (1573). According to one embodiment, the wearable device (101) can enlarge the size of the virtual area (1570). For example, the wearable device (101) can obtain a gaze input that gazes at the virtual area (1570) for a specified period of time (e.g., N seconds or more, where N is a natural number). In response to the gaze input, the wearable device (101) can expand the virtual area (1570) to create the virtual area (1573). The wearable device (101) can display at least a portion of an image corresponding to an external object through the virtual area (1573). The user's (110) gaze being directed to a specific external object can be understood as the user's (110) intention being interested in the specific external object. The wearable device (101) can provide interaction with external objects outside the boundary area (130) by expanding the size of the virtual area based on the above-mentioned gaze. For another example, the wearable device (101) can obtain user input for a separate visual object (e.g., a soft handler, a resizing handler) for expanding the virtual area (1570). In response to the user input, the wearable device (101) can expand the virtual area (1570) to create a virtual area (1573).

[0207] While an example of enlarging the size of a virtual area is described in FIG. 15, embodiments of the present disclosure are not limited thereto. Reducing the size of a displayed virtual area due to an external object may also be understood as an embodiment of the present disclosure.

[0208] Figure 16 shows an example of a notification of an external object entering a boundary area.

[0209] Referring to FIG. 16, in the first screen (1611), the wearable device (101) can display a virtual image (1510) of a virtual environment. While the wearable device (101) displays the virtual image (1510), an external object can approach the wearable device (101). For example, in the second screen (1612), an external object (1630) can enter within a certain distance from a boundary area (130) set in the wearable device (101) (e.g., boundary line (131), boundary surface (132)). As the external object (1630) enters within the certain distance from the boundary area (130), the wearable device (101) can display a pattern image (1660).

[0210] The wearable device (101) may create a virtual area (1670). In one embodiment, the wearable device (101) may create the virtual area (1670) based on identifying that an external object (1630) enters within a certain distance (e.g., a second threshold distance) from the boundary area (130). Meanwhile, even if at least a part of an external image is displayed through the virtual area (1670), the external object (1630) may not be located in the field of view (e.g., FoV) of the user (110). Accordingly, the wearable device (101) may generate a notification (1673) to inform the user (110) that the external object (1630) is located adjacent to the boundary area (130). As in the third screen (1613), the wearable device (101) may display the notification (1673) within the virtual environment, thereby indicating to the user (110) that the external object (1630) has entered within a certain distance from the boundary area (130). For example, the notification (1673) may be displayed in an edge area of ​​the field of view of the user (110).

[0211] Although a visual notification (1673) is illustrated in the third screen (1613), embodiments of the present disclosure are not limited thereto. For example, the wearable device (101) may indicate to the user (110) through a separate audio notification that an external object (1630) has entered within a certain distance from the boundary area (130). The fourth screen (1614) displays the user's (110) field of view when the user (110) turns his / her gaze in the direction in which the external object (1630) approaches. The wearable device (101) may display the screen (1614) in response to the rotational movement of the user (110). While displaying the virtual environment and pattern image (1660), the wearable device (101) may display at least a portion of the external image through the virtual area (1675). At least a portion of the displayed external image may include the external object (1630).

[0212] In FIG. 16, examples of displaying a notification to a user (110) when an external object (1630) enters within a certain distance from a boundary zone (130) (e.g., boundary line (131), boundary surface (132)) set in a wearable device (101) are described. However, it may also be understood as an embodiment of the present disclosure to notify the user (110) not only when a physical approach of an external object (1630) is detected from the outside. For example, the wearable device (101) may detect a specified sound (e.g., a sound above a certain intensity, a voice signal corresponding to a specific name, a voice signal corresponding to a specific command) from an external space through at least one microphone (e.g., multiple microphones, a directional microphone). When the specified sound is detected, the wearable device (101) may display a notification corresponding to the specified sound in the boundary zone (130). The wearable device (101) can determine the position of the boundary surface (132) corresponding to the direction in which the specified sound is detected. The wearable device (101) can determine that the position of the boundary surface (132) is within the FoV of the user (110). The wearable device (101) can provide a specific visual effect (e.g., a water wave, a transparency effect) starting from the position. Meanwhile, as another example, the wearable device (101) can determine that the position of the boundary surface (132) is outside the FoV of the user (110). The wearable device (101) can notify the user (110) of the input of the specified sound through the speaker, or display a notification indicating that the specified sound has been detected at a specific point to the user (110) through the display.

[0213] FIG. 17 illustrates an operation flow of a wearable device (e.g., wearable device (101)) for notifying an external object entering a boundary area (e.g., boundary area (130)).

[0214] Referring to FIG. 17, in operation (1701), a wearable device (101) (e.g., processor (410)) may obtain information about a boundary zone. The boundary zone (130) may be defined to guide a boundary within which a virtual environment is displayed to a user (110) of the wearable device (101). The wearable device (101) may set the boundary zone (130) based on user input and / or specified settings.

[0215] In operation (1703), the wearable device (101) can display a virtual environment within a boundary zone (130). The wearable device (101) can be located within the boundary zone (130). The wearable device (101) can display a virtual image of the virtual environment on a boundary surface (132) of the boundary zone (130).

[0216] In operation (1705), the wearable device (101) can determine whether an external object approaches. The wearable device (101) can determine whether the external object enters within a certain distance (e.g., a second threshold distance) from the boundary zone (130). For example, the wearable device (101) can determine whether the external object enters within the certain distance from the boundary zone (130) through a camera (e.g., camera (425), a 360-degree camera). As another example, the wearable device (101) can determine whether the external object enters within a certain distance from the boundary zone (130) through a microphone. The wearable device (101) can determine whether the external object enters within a certain distance from the boundary zone (130) through whether the intensity of a sound signal related to the external object received through the microphone is greater than or equal to a threshold intensity. For example, if the intensity of the sound signal related to the external object is greater than or equal to the threshold intensity, the wearable device (101) may determine that the external object has entered within a certain distance from the boundary zone (130). For example, if the intensity of the sound signal related to the external object is less than the threshold intensity, the wearable device (101) may determine that the external object has not entered within a certain distance from the boundary zone (130). In addition, for example, the wearable device (101) may detect a distance to the external object and a change in the distance through at least one sensor (e.g., sensor (430), a distance sensor, an ultrasonic sensor), and, based on a result of the detection, determine whether the external object has entered within the certain distance from the boundary zone (130).

[0217] The wearable device (101) may perform operation (1707) when the external object enters within the predetermined distance from the boundary zone (130). The wearable device (101) may perform operation (1705) again when the external object does not enter within the predetermined distance from the boundary zone (130).

[0218] In operation (1707), the wearable device (101) can determine whether an external object is located within the field of view screen. The field of view screen is a screen shown by a user (110) of the wearable device (101) and represents FoV. If the external object is located within the field of view screen, the wearable device (101) can perform operation (1709). If the external object is not located within the field of view screen, the wearable device (101) can perform operation (1711).

[0219] In operation (1709), the wearable device (101) may display at least a portion of an external image in a virtual area according to the location of an external object while displaying a virtual environment. The external image may be acquired through a camera (e.g., camera (425), 360-degree camera)) of the wearable device (101). At least a portion of the external image may include the external object. The wearable device (101) may display an external image including an external object through the virtual area while displaying a virtual environment and a pattern image within a boundary area (130).

[0220] In operation (1711), the wearable device (101) may display a notification at the edge of the visual screen. By displaying the notification within a virtual environment, the wearable device (101) may indicate to the user (110) that the external object has entered within a certain distance from the boundary area (130).

[0221] Although the above notification is shown in FIG. 17, embodiments of the present disclosure are not limited thereto. As a non-limiting example, the wearable device (101) may display a guidance message indicating that an external object has entered the boundary zone (130). As a non-limiting example, the wearable device (101) may notify the user (110) that an external object has entered the boundary zone (130) through vibration or audio signal output from the controller (590).

[0222] Although not shown in FIG. 17, the wearable device (101) displaying the virtual area may further perform additional operations to adjust the size of the virtual area. For example, the wearable device (101) may display a visual object for adjusting the area of ​​the virtual area (e.g., the first visual object (971a), the second visual object (972a), the third visual object (973a), and the fourth visual object (974a) of FIG. 9A, and the first visual object (971b) and the second visual object (972b) of FIG. 9B). The wearable device (101) may receive a user input for the visual object (e.g., the second user input of FIGS. 9A and 9B, an input for pulling a soft handler outward, an input for pulling the soft handler inward). The wearable device (101) may adjust the size of the virtual area in response to the user input. For another example, the wearable device (101) can obtain an input of a designated gesture using the controller (590). For example, if the designated gesture is a gesture of holding both sides and opening, the wearable device (101) can expand the size of the virtual area. For another example, if the designated gesture is a gesture of holding both sides and closing, the wearable device (101) can reduce the size of the virtual area. For another example, the wearable device (101) can adjust the size of the virtual area by moving the controller (590) in the direction of the gaze of the user (110) (e.g., a gaze direction maintained for N seconds or more, where N is a natural number).

[0223] In embodiments, a wearable device is provided. The wearable device may include a camera, at least one sensor, a display, a memory for storing instructions, and a processor. The instructions, when executed by the processor, may cause the wearable device to obtain information about a boundary zone for guiding a boundary within which a virtual environment is provided, obtain a location of the wearable device with respect to the boundary zone based on sensor data obtained from the camera or the at least one sensor, display a pattern image for representing the boundary zone through the display while displaying a virtual environment through the display based on identifying that a location of the wearable device within the boundary zone is within a first threshold distance from the boundary zone, and display at least a portion of an external image obtained through the camera through the display in a virtual area according to a location of the input means while displaying the virtual environment and the pattern image through the display based on identifying that a location of an input means connected to the wearable device is within a second threshold distance from the boundary zone.

[0224] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to obtain a first user input for the virtual area having a first size, display at least one visual object for adjusting a size of the virtual area in response to the first user input, and display at least a portion of the external image in the virtual area having a size changed from the first size to a second size in response to a second user input for the at least one visual object. The second size of the virtual area may be greater than the first size of the virtual area.

[0225] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to output sound information for the virtual environment according to a first volume while displaying at least a portion of the external image in the virtual area having the first size, and to output sound information for the virtual environment according to a second volume while displaying at least a portion of the external image in the virtual area having the second size. The second volume may be smaller than the first volume.

[0226] In one embodiment, the first user input may be obtained based on a user's gaze input maintained in the virtual area for a specified period of time. The second user input may be obtained based on a movement of the input means away from the at least one visual object in the virtual area.

[0227] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to generate the pattern image based on identifying that a location of the wearable device is within a first threshold distance from the boundary zone. The pattern image may include a first grid in which a designated shape is repeated when a distance between the wearable device and the boundary zone is a first distance. The pattern image may include a second grid in which the designated shape is repeated when a distance between the wearable device and the boundary zone is a second distance less than the first distance. A density of the designated shape in the second grid may be narrower than a density of the designated shape in the first grid.

[0228] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to display the pattern image including a designated figure of a first size based on identifying that an input means connected to the wearable device is located at a first location, and to display the pattern image including a designated figure of a second size based on identifying that an input means connected to the wearable device is located at a second location. A distance between the second location and the boundary area may be shorter than a distance between the first location and the boundary area. The second size may be greater than the first size.

[0229] According to one embodiment, the pattern image includes shapes of different sizes while at least a portion of the external image is displayed in a virtual area according to the position of the input means, and a size of a first shape among the shapes that is relatively close to the virtual area may be smaller than a size of a second shape among the shapes that is relatively far from the virtual area.

[0230] According to one embodiment, the instructions, when executed by the processor, may cause the wearable device to obtain user status information based on sensor data obtained from the at least one sensor, determine a color based on the user status information, and display the boundary area and the pattern image through the display based on the determined color.

[0231] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to display the virtual environment and the pattern image through the display based on identifying that a position of a second input means connected to the wearable device is within the second threshold distance from the boundary zone, while displaying at least a portion of the external image in a second virtual area according to a position of the second input means through the display, and to display at least a portion of the external image in a third virtual area including the virtual area and the second virtual area through the display based on identifying that a separation distance between the virtual area and the second virtual area is less than the threshold distance.

[0232] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to display an external image acquired through the camera through the display based on identifying that a location of the wearable device is outside the boundary zone, and to display a pattern image representing the boundary zone through the display while displaying the external image based on identifying that a location of the wearable device is outside the boundary zone and within the first threshold distance from the boundary zone.

[0233] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to display a virtual image of the virtual environment through the display in a boundary virtual area according to a position of the input means, while displaying the external image and the pattern image through the display, based on identifying that the position of the input means is within the second threshold distance from the boundary area.

[0234] In one embodiment, the instructions, when executed by the processor, cause the wearable device to obtain a first user input for the border virtual region having a first size, display at least one visual object for adjusting a size of the border virtual region in response to the first user input, and display at least a portion of the external image in the border virtual region having a size changed from the first size to a second size in response to a second user input for the at least one visual object, wherein the second size of the border virtual region may be greater than the first size of the border virtual region.

[0235] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to output sound information for the virtual environment according to a first volume while displaying at least a portion of the external image in the bounding virtual area having the first size, and to output sound information for the virtual environment according to a second volume while displaying at least a portion of the external image in the bounding virtual area having the second size, wherein the second volume may be larger than the first volume.

[0236] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to obtain information about an external object based on at least one of the camera or the at least one sensor, and to display an external image including the external object on the display in a virtual area corresponding to the location of the external object while displaying the virtual environment on the display based on identifying that the location of the external object is within a field of view of the wearable device.

[0237] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to display a notification about the external object within the field of view through the display while displaying the virtual environment based on identifying that the location of the external object is outside the field of view of the wearable device.

[0238] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to display, through the display, at least a portion of an external image acquired through the camera in a virtual area according to a location of the wearable device while displaying the virtual environment and the pattern image through the display, based on identifying that a location of the wearable device is within a second threshold distance from the boundary zone. The second threshold distance may be less than the first threshold distance.

[0239] In embodiments, a wearable device is provided. The wearable device may include a camera, at least one sensor, a display, a memory for storing instructions, and a processor. The instructions, when executed by the processor, cause the wearable device to obtain information about a boundary zone for providing a virtual environment based on scanning of a physical space, obtain a location of the wearable device with respect to the boundary zone based on sensor data obtained from the camera or the at least one sensor, and display an external image obtained through the camera through the display based on identifying that the location of the wearable device is located outside the boundary zone, and display a pattern image for indicating the boundary zone while displaying the external image obtained through the camera based on identifying that the location of the wearable device is within a first threshold distance from the boundary zone outside the boundary zone, and display a virtual image of the virtual environment in a boundary virtual zone according to the location of the wearable device or the location of the input means while displaying the external image and the pattern image through the display based on identifying that the location of the wearable device or the location of an input means connected to the wearable device is within a second threshold distance from the boundary zone. It can cause it to be displayed through.

[0240] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to obtain a first user input for the border virtual region having a first size, display at least one visual object for adjusting a size of the border virtual region in response to the first user input, and display at least a portion of the external image in the border virtual region having a size changed from the first size to a second size in response to a second user input for the at least one visual object. The second size of the border virtual region may be greater than the first size of the border virtual region.

[0241] In one embodiment, the instructions, when executed by the processor, may cause the wearable device to output sound information for the virtual environment according to a first volume while displaying at least a portion of the external image in the bounding virtual area having the first size, and to output sound information for the virtual environment according to a second volume while displaying at least a portion of the external image in the bounding virtual area having the second size, wherein the second volume may be larger than the first volume.

[0242] In one embodiment, the first user input may be obtained based on a user's gaze input maintained within the boundary virtual area for a specified period of time. The second user input may be obtained based on a movement of the input means away from the at least one visual object in a direction away from the boundary virtual area.

[0243] Although a square pillar is illustrated as an example of a boundary area in the present disclosure, the shape of the boundary area is merely an example for describing embodiments and is not to be construed as limiting the embodiments of the present disclosure. For example, the boundary line (131) of the boundary area (130) may be a circle, and the boundary area (130) may have a cylinder shape. As another example, the boundary line (131) of the boundary area (130) may be a triangle, and the boundary area (130) may have a triangular pillar shape.

[0244] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0245] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0246] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0247] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., a wearable device (101)). For example, a processor (e.g., a processor (410)) of the machine (e.g., a wearable device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0248] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0249] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In wearable devices; camera; At least one sensor; display; Memory for storing instructions; Contains a processor, The above instructions, when executed by the processor, cause the wearable device to: Obtain information about the boundary area to guide the boundary within which the virtual environment is provided; Based on sensor data obtained from the camera or at least one sensor, the location of the wearable device with respect to the boundary area is obtained, Based on identifying that the position of the wearable device within the boundary zone is within a first threshold distance from the boundary zone, while displaying a virtual environment through the display, displaying a pattern image for indicating the boundary zone through the display; Based on identifying that the position of the input means for the wearable device is within a second threshold distance from the boundary area, while displaying the virtual environment and the pattern image through the display, causing at least a part of an external image acquired through the camera to be displayed through the display in a virtual area according to the position of the input means. Wearable devices.

2. In claim 1, The above instructions, when executed by the processor, cause the wearable device to: Obtaining a first user input for the virtual area having a first size, In response to said first user input, displaying at least one visual object for adjusting the size of said virtual area; In response to a second user input for said at least one visual object, cause at least a portion of said external image to be displayed in said virtual area having a size changed from said first size to said second size; The second size of the virtual area is larger than the first size of the virtual area. Wearable devices.

3. In claim 2, The above instructions, when executed by the processor, cause the wearable device to: While displaying at least a part of the external image in the virtual area having the first size, outputting sound information for the virtual environment according to the first volume, while displaying at least a part of the external image in the virtual area having the second size, cause sound information for the virtual environment to be output according to the second volume, The second volume is smaller than the first volume. Wearable devices.

4. In claim 2, The above first user input is obtained based on the user's gaze input maintained in the virtual area for a specified period of time, The second user input is obtained based on movement of the input means away from the virtual area from the at least one visual object. Wearable devices.

5. In claim 1, The above instructions, when executed by the processor, cause the wearable device to: causing the pattern image to be generated based on identifying that the location of the wearable device is within a first threshold distance from the boundary area; The above pattern image includes a first grid in which a specified shape is repeated when the distance between the wearable device and the boundary area is a first distance, The above pattern image includes a second grid in which the specified shape is repeated when the distance between the wearable device and the boundary area is a second distance shorter than the first distance, The density of the specified shape in the second grid is narrower than the density of the specified shape in the first grid. Wearable devices.

6. In claim 1, The above instructions, when executed by the processor, cause the wearable device to: Based on identifying that the input means for the wearable device is located at the first location, displaying the pattern image including the designated shape of the first size, Based on identifying that the input means for the wearable device is located at the second location, causing the pattern image including the designated shape of the second size to be displayed, The distance between the second position and the boundary zone is shorter than the distance between the first position and the boundary zone, The above second size is larger than the above first size, Wearable devices.

7. In claim 1, The above pattern image includes shapes of different sizes while at least a part of the external image is displayed in a virtual area according to the position of the input means, Among the above shapes, the size of the first shape that is relatively close to the virtual area is smaller than the size of the second shape that is relatively far from the virtual area among the above shapes. Wearable devices.

8. In claim 1, the instructions, when executed by the processor, cause the wearable device to: Based on sensor data obtained from the above camera or at least one sensor, user status information is obtained, Based on the above user's status information, determine the color, Causing the boundary area and the pattern image to be displayed through the display according to the determined color. Wearable devices.

9. In claim 1, the instructions, when executed by the processor, cause the wearable device to: Based on identifying that the position of the second input means for the wearable device is within the second threshold distance from the boundary area, while displaying the virtual environment and the pattern image through the display, displaying at least a part of the external image in a second virtual area according to the position of the second input means through the display, Based on identifying that a distance between the virtual area and the second virtual area is less than a threshold distance, causing at least a portion of the external image to be displayed through the display in a third virtual area including the virtual area and the second virtual area. Wearable devices.

10. In claim 1, The above instructions, when executed by the processor, cause the wearable device to: Based on identifying that the location of the wearable device is outside the boundary area, an external image acquired through the camera is displayed through the display, Based on identifying that the location of the wearable device is within the first threshold distance from the boundary zone outside the boundary zone, causing a pattern image to be displayed through the display to represent the boundary zone while displaying the external image. Wearable devices.

11. In claim 10, The above instructions, when executed by the processor, cause the wearable device to: Based on identifying that the position of the input means is within the second threshold distance from the boundary area, while displaying the external image and the pattern image through the display, causing a virtual image of the virtual environment to be displayed through the display in a boundary virtual area according to the position of the input means. Electronic devices.

12. In claim 11, The above instructions, when executed by the processor, cause the wearable device to: Obtaining a first user input for the above-described boundary virtual area having a first size, In response to said first user input, displaying at least one visual object for adjusting the size of said border virtual area; In response to a second user input for said at least one visual object, cause at least a portion of said external image to be displayed in said border virtual area having a size changed from said first size to said second size; The second size of the above boundary virtual area is larger than the first size of the above boundary virtual area. Wearable devices.

13. In claim 12, The above instructions, when executed by the processor, cause the wearable device to: While displaying at least a part of the external image in the boundary virtual area having the first size, output sound information for the virtual environment according to the first volume, while displaying at least a part of the external image in the boundary virtual area having the second size, cause sound information for the virtual environment to be output according to the second volume, The above second volume is larger than the above first volume, Wearable devices.

14. In claim 1, The above instructions, when executed by the processor, cause the wearable device to: Based on at least one of the above cameras or at least one of the above sensors, information about an external object is obtained, While displaying the virtual environment through the display based on identifying that the location of the external object is within the field of view of the wearable device, causing an external image including the external object to be displayed through the display in a virtual area corresponding to the location of the external object. Wearable devices.

15. In claim 14, The above instructions, when executed by the processor, cause the wearable device to: While displaying the virtual environment through the display based on identifying that the location of the external object is outside the field of view of the wearable device, causing a notification about the external object within the field of view to be displayed through the display. Wearable devices.

Images