Wearable device for displaying visual object for adjusting visibility of virtual object, and method thereof

The wearable device addresses the issue of obscured virtual objects by using visual objects to enhance visibility and user interaction, improving the experience in mixed and augmented reality through its processing and display capabilities.

WO2024228464A1PCT designated stage expired Publication Date: 2024-11-07SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/003098
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-02
Filing Date
2024-03-11
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Wearable devices struggle to effectively display virtual objects that are obscured by other virtual or real objects, leading to reduced visibility and user experience issues in augmented reality applications.

Method used

A wearable device with a processing circuit and display capabilities that identifies obscured virtual objects and displays visual objects to enhance visibility, allowing users to interact with and move virtual objects within the virtual space based on user inputs, such as gestures or gaze direction.

Benefits of technology

Improves the visibility and controllability of virtual objects by moving them to non-obscured positions, enhancing the user experience in mixed and augmented reality environments by using visual objects to represent obscured virtual objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024003098_07112024_PF_FP_ABST
    Figure KR2024003098_07112024_PF_FP_ABST
Patent Text Reader

Abstract

A processor of a wearable device, according to an embodiment, may display, on a display, a first virtual object included in a field-of-view (FoV) formed on the basis of a reference point within a virtual space. On the basis of identifying a second virtual object within the virtual space, the second virtual object arranged behind the first virtual object, in a direction from the reference point to the first virtual object, the processor may display a visual object related to the second virtual object according to the position of the first virtual object within the display. The processor may display, on the basis of an input representing selection of the visual object, the second virtual object moved to a portion of the virtual space, the portion being formed between the reference point and the first visual object.
Need to check novelty before this filing date? Find Prior Art

Description

Wearable device for displaying a visual object that controls the visibility of a virtual object and method thereof

[0001] The present disclosure relates to a wearable device and method for displaying a visual object that controls the visibility of a virtual object.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] According to an embodiment, a wearable device may include at least one display, a memory including one or more storage media storing instructions, and a processing circuit. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display, on the display, a first virtual object included in a field-of-view (FoV) formed based on a reference point in a virtual space. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to identify a second virtual object in the virtual space, positioned behind the first virtual object along a direction from the reference point toward the first virtual object, and to display a visual object associated with the second virtual object based on a location of the first virtual object in the display. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display the second virtual object, moved to a portion of the virtual space formed between the reference point and the first virtual object, based on an input indicating selection of the visual object.

[0004] In one embodiment, a method of a wearable device may include an operation of displaying a first virtual object included in a FoV formed based on a reference point in a virtual space on a display of the wearable device. The method may include an operation of displaying a visual object related to the second virtual object based on a position of the first virtual object in the display, based on identifying a second virtual object in the virtual space, the second virtual object being positioned behind the first virtual object along a direction from the reference point to the first virtual object. The method may include an operation of displaying the second virtual object, which has been moved to a portion of the virtual space between the reference point and the first virtual object, based on an input indicating a selection of the visual object.

[0005] In one embodiment, a wearable device may include a camera, a display, a memory including one or more storage media storing instructions, and at least one processor including a processing circuit. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to identify, using the camera, an external object visible through the display. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display, within the display, a visual object associated with the virtual object based on identifying, within a virtual space mapped with an external space including the external object, a virtual object positioned behind a second location within the virtual space corresponding to the camera in a direction from the first location within the virtual space corresponding to the camera to the second location within the virtual space corresponding to the external object. The instructions, when individually and / or collectively executed by the at least one processor, may cause the wearable device to display the virtual object, based on an input indicating selection of the visual object, moved to a portion of the virtual space defined between the first location and the second location.

[0006] In one embodiment, a method of a wearable device may include an operation of identifying an external object visible through a display of the wearable device using a camera of the wearable device. The method may include an operation of displaying a visual object related to the virtual object within the display based on identifying a virtual object located behind a second location within the virtual space corresponding to the external object in a direction from a first location within the virtual space corresponding to the camera to a second location within the virtual space corresponding to the external object, within a virtual space mapped with the external space. The method may include an operation of displaying the virtual object moved to a portion of the virtual space formed between the first location and the second location, based on an input indicating selection of the visual object.

[0007] FIG. 1 is a drawing for explaining an example of a screen displayed through a display of a wearable device according to one embodiment.

[0008] FIG. 2 illustrates an example block diagram of a wearable device according to one embodiment.

[0009] FIG. 3a, FIG. 3b, FIG. 3c, FIG. 3d, and FIG. 3e are drawings for explaining a screen displayed by a wearable device according to the positional relationship of a plurality of virtual objects in a virtual space.

[0010] FIGS. 4A and 4B are drawings for explaining the operation of a wearable device that displays a virtual object corresponding to a visual object based on an input related to the visual object.

[0011] FIGS. 5A, 5B, 5C, 5D, and 5E are drawings for explaining the operation of a wearable device that adjusts the position of a virtual object corresponding to a visual object in a virtual space based on an input related to the visual object.

[0012] FIG. 6 is a diagram illustrating the operation of a wearable device that displays a visual object corresponding to another virtual object obscured by a specific virtual object within a display.

[0013] FIG. 7 is an exemplary diagram illustrating the operation of a wearable device for identifying a virtual space mapped to an external space.

[0014] FIGS. 8A, 8B, and 8C are exemplary drawings for explaining the operation of a wearable device in relation to the positional relationship between an external object and a virtual object.

[0015] FIG. 9 is a flowchart of operations of a wearable device according to one embodiment.

[0016] FIG. 10 is a flowchart of operations of a wearable device according to one embodiment.

[0017] FIG. 11 is a flowchart of operations of a wearable device according to one embodiment.

[0018] FIG. 12A illustrates an example of a perspective view of a wearable device, according to one embodiment.

[0019] FIG. 12B illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0020] FIGS. 13A and 13B illustrate an example of an appearance of a wearable device according to one embodiment.

[0021] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0022] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0023] The term "module" as used in this document includes a unit composed of hardware, and may be used interchangeably with terms such as "component" or "circuit." A module may be an integrally composed component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0024] FIG. 1 is a diagram illustrating an example of a screen (140) displayed through a display of a wearable device (101) according to one embodiment. The wearable device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). Although the appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of one or more hardware components included in the wearable device (101) is exemplarily described with reference to FIG. 2. An example of the structure of the wearable device (101) that is wearable on the head of a user (110) is described with reference to FIGS. 12A to 12B and / or 13A to 13B. The wearable device (101) may be referred to as an electronic device. For example, an electronic device may be combined with an accessory to be attached to a user's head to form an HMD.

[0025] According to one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). Referring to FIG. 1, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). In the state, the wearable device (101) may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or active area) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0026] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). As shown in FIG. 1, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the eye of the user (110). The wearable device (101) may combine light emitted from a display of the wearable device (101) with ambient light passing through the lens. A display area of ​​the display may be formed within the lens through which the ambient light passes. Since the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0027] In one embodiment, the wearable device (101) can use the virtual space (120) to determine the location of a virtual object (e.g., virtual objects (131, 132)) within the display. The wearable device (101) can support access to the virtual object regardless of the visibility (or state) of the virtual object shown to the user (110) through the display. For example, if a specific virtual object is occluded by another virtual object within the display and / or is positioned in a different part from the part of the virtual space (120) to be displayed through the display, the visibility of the specific virtual object may be reduced because the specific virtual object is not displayed within the display. In one embodiment, the wearable device (101) can provide the user (110) with an option to display the specific virtual object on the display using a visual object to compensate for the visibility. Hereinafter, the visibility of a virtual object may refer to a state of the virtual object related to the number of pixels corresponding to the virtual object among the pixels of the display of the wearable device (101). Reducing the visibility of the virtual object may include a state in which the number of pixels corresponding to the virtual object is reduced, and / or a state in which the number of pixels corresponding to the virtual object is substantially zero.

[0028] Referring to FIG. 1, an exemplary state of a virtual space (120) managed by a wearable device (101) is illustrated. The virtual space (120) is a coordinate space for providing a three-dimensional user interface (UI) provided by the wearable device (101) and may be generated by information stored by the wearable device (101). The wearable device (101) may generate and / or change information about the virtual space (120) based on one or more software applications executed by a processor of the wearable device (101). In one embodiment, the wearable device (101) may connect at least a portion of the virtual space (120) to an external space including the wearable device (101) in order to provide a user experience in which virtual objects and real objects are mixed. The UI included in the virtual space (120) may be referred to as a virtual object. The virtual object may include a two-dimensional panel (e.g., a window) and / or a three-dimensional graphical object provided by an application executed by the wearable device (101). The virtual object may include one or more visual objects (e.g., a button, a menu, a text box, an image, and / or a video) for interacting with a user (110) of the wearable device (101).

[0029] Referring to FIG. 1, an exemplary state of a wearable device (101) displaying a screen (140) based on rendering of a portion of a virtual space (120) is illustrated. The wearable device (101) can display a virtual object (e.g., virtual objects (131, 132)) included in a field-of-view (FoV) formed based on a reference point (pv) in the virtual space (120) on the display. The portion of the virtual space (120) intersected with one side (122) of the virtual space (120) can have a fan shape with the reference point (pv) as the center and the FoV as the central angle. The screen (140) displayed through the display can correspond to a view of the virtual space (120) viewed from the reference point (pv) along a direction (dv). The FoV can correspond to a horizontal observation angle of the screen (140). In terms of forming a view of a virtual space (120) displayed through a display, a reference point (pv) may be referred to as a view point.

[0030] According to one embodiment, the wearable device (101) may provide a user experience of browsing a virtual space (120) based on the motion of the head, similar to a user (110) browsing an external space by moving the head. Referring to FIG. 1, while the user (110) is wearing the wearable device (101), the wearable device (101) may determine and / or move a reference point (pv) within the virtual space (120) based on a position (pr) corresponding to the physical position of the wearable device (101). While the user (110) is wearing the wearable device (101), the wearable device (101) may determine and / or move a direction (dv) within the virtual space (120) based on a direction (dr) toward which the wearable device (101) is facing. A direction (dr) parallel to the front direction of the wearable device (101) is shown, but the embodiment is not limited thereto.

[0031] In one embodiment, when a user (110) wearing a wearable device (101) moves and / or moves the head of the user (110), the wearable device (101) can identify a change in position (pr) and / or direction (dr) using one or more sensors. Based on the changed position (pr) and / or direction (dr), the wearable device (101) can change a reference point (pv) and / or direction (dv) within the virtual space (120). Based on the changed reference point (pv) and / or changed direction (dv), the wearable device (101) can change a portion of the virtual space (120) displayed through the screen (140). As a portion of the virtual space (120) displayed through the screen (140) changes, the wearable device (101) can provide the user (110) with a user experience of viewing the virtual space (120) according to the movement and / or head motion of the user (110).

[0032] In one embodiment, the virtual space (120) and / or one or more virtual objects included in the virtual space (120) may be created, modified, and / or deleted by a software application executed by the wearable device (101). For example, the virtual space (120) may be created by a software application installed in the wearable device (101) to provide VR. For example, the virtual space (120) may be provided by a system application of the wearable device (101) to accommodate windows (e.g., activities of the Android operating system), two-dimensional widgets, and / or three-dimensional widgets provided from software applications of the wearable device (101). In the above example, within a state where a plurality of applications are executed, the wearable device (101) may place a plurality of virtual objects provided from each of the plurality of applications within the virtual space (120).

[0033] Referring to the exemplary virtual space (120) of FIG. 1, virtual objects (131, 132) may be arranged along a direction (dv) within the virtual space (120). For example, the virtual object (132) may be positioned behind the virtual object (131) along a direction (dv) from a reference point (pv) to the virtual object (131). Since the virtual object (132) is positioned behind the virtual object (131) along the direction (dv), the virtual object (132) may be obscured by the virtual object (131) within a screen (140) corresponding to a view of the virtual space (120) from the reference point (pv). For example, the visibility of the virtual object (132) within the screen (140) may be reduced by the virtual object (131). For example, the wearable device (101) may display a virtual object (131) as an overlay on a virtual object (132) within the screen (140). For example, the wearable device (101) may not perform rendering for, and / or bypass, a virtual object (132) positioned behind the virtual object (131). For example, the wearable device (101) may stop displaying at least a portion of the virtual object (132) that is obscured by the virtual object (131) within the screen (140). An operation of the wearable device (101) to identify at least one virtual object that is not visible through the screen (140), such as the virtual object (132), is described with reference to FIGS. 3A to 3E.

[0034] According to one embodiment, the wearable device (101) may display a visual object (150) related to the virtual object (132) within the screen (140) based on identifying a virtual object (132) within the virtual space (120) placed behind the virtual object (131). The wearable device (101) may display the visual object (150) based on the location of the virtual object (131) displayed on the screen (140) displayed through the display. The visual object (150) may include an icon, an image, and / or a video representing the virtual object (132) covered by the virtual object (131). The image included in the visual object (150) may include a two-dimensional image (e.g., a screen shot and / or a snapshot) representing the shape of the virtual object (132). The icon included in the visual object (150) may include an icon of a software application of the wearable device (101) corresponding to the virtual object (132). The operation of the wearable device (101) displaying the visual object (150) is described with reference to FIGS. 2 to 11.

[0035] Referring to FIG. 1, in an exemplary state where a visual object (150) corresponding to a virtual object (132) covered by a virtual object (131) is displayed within a screen (140), the wearable device (101) may change the position of the virtual object (132) within the virtual space (120) based on an input related to the visual object (150). Based on the input, the wearable device (101) may move the position of the virtual object (132) from a first position covered by the virtual object (131) to a second position not covered by the virtual object (131) within the screen (140). In one embodiment, the wearable device (101) may move the virtual object (132) to a part of the virtual space (120) formed between the reference point (pv) and the virtual object (131). Since the wearable device (101) displays the virtual object (132) moved to the above part, the virtual object (132) can be displayed in front of the virtual object (131) within the screen (140). An exemplary operation of the wearable device (101) moving at least one virtual object based on the input related to the visual object (150) is described with reference to FIGS. 4A to 4B and / or FIGS. 5A to 5E.

[0036] As described above, according to one embodiment, the wearable device (101) may display a visual object (150) to enhance the visibility of a virtual object (132) within a state of displaying a screen (140) based on a virtual space (120). When a user (110) places a plurality of virtual objects within the virtual space (120), the visibility of at least one virtual object may be reduced due to overlap between virtual objects, such as the virtual objects (131, 132) of FIG. 1. The wearable device (101) may display a visual object (150) corresponding to the at least one virtual object with reduced visibility, thereby improving the controllability of the user (110) over the at least one virtual object. For example, using a visual object (150), the wearable device (101) can receive an input to move a virtual object (132) that is not visible to the user (110) to a part of the virtual space (120) that is visible to the user (110).

[0037] Although the overlap between virtual objects (131, 132) has been described as an example, the embodiment is not limited thereto. When the wearable device (101) provides a user experience based on a mixture of real objects and virtual objects, the virtual space (120) may be mapped to an external space recognized by the wearable device (101). Within the virtual space (120) mapped to the external space, the wearable device (101) may determine whether to display a virtual object based on the location of the real object with respect to the wearable device (101). For example, if a virtual object is placed behind a location in the virtual space (120) corresponding to a real object, the wearable device (101) may stop displaying the virtual object on the screen (140). In the example, the wearable device (101) may use a visual object corresponding to the virtual object to execute a function for improving and / or enhancing the visibility of the virtual object. According to one embodiment, the operation of a wearable device (101) based on a virtual space (120) mapped to an external space is described with reference to FIG. 7 and / or FIG. 8a to FIG. 8c.

[0038] Hereinafter, with reference to FIG. 2, an example of one or more hardwares included in a wearable device (101) and / or a software application installed in the wearable device (101) according to one embodiment is described.

[0039] FIG. 2 illustrates an example of a block diagram of a wearable device (101) according to one embodiment. The wearable device (101) of FIG. 2 may be an example of the wearable device (101) of FIG. 1.

[0040] According to one embodiment, a wearable device (101) may include at least one of a processor (210), a memory (220), a display (230), a sensor (240), and a communication circuit (250). The processor (210), the memory (220), the display (230), the sensor (240), and the communication circuit (250) may be electrically and / or operably coupled with each other by electronic components such as a communication bus (202). Hereinafter, operably coupled electronic components may mean that a direct connection or an indirect connection is established between the electronic components, either wired or wireless, such that a second electronic component is controlled by a first electronic component among the electronic components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least a portion of the processor (210), memory (220), and communication circuit (250)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware included in the wearable device (101) is not limited to that illustrated in FIG. 2. For example, the wearable device (101) may include only some of the hardware illustrated in FIG. 2.

[0041] In one embodiment, the processor (210) of the wearable device (101) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor (210) may have a single-core processor structure, or a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0042] In one embodiment, the memory (220) of the wearable device (101) may include a hardware component for storing data and / or instructions input to or output from the processor (210) of the wearable device (101). The memory (220) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC).

[0043] In one embodiment, the display (230) of the wearable device (101) can output visualized information (e.g., the screens of FIGS. 1, 3A to 3E, 4A to 4B, 5A to 5E, 6, 7, and / or 8A to 8C) to a user (e.g., the user (110) of FIG. 1). For example, the display (230) can visualize information generated by a processor (210) including a circuit such as a graphic processing unit (GPU). The display (230) can include a flexible display, a flexible display, a flat panel display (FPD), and / or electronic paper. The flexible display can be deformable by an external force or have a shape that is bent by the external force. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The display (230) of FIG. 2 may include at least one display (1250, 1350), which will be described later with reference to FIGS. 12A to 12B and / or FIGS. 13A to 13B.

[0044] According to one embodiment, a sensor (240) of a wearable device (101) may generate electrical information that may be processed by a processor (210) and / or a memory (220) of the wearable device (101) from non-electronic information related to the wearable device (101). The information may be referred to as sensor data. The sensor (240) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (101). The IMU may include at least one of an acceleration sensor, a gravity sensor, and a gyro sensor. For example, based on the data of the IMU, the processor (210) of the wearable device (101) may identify and / or measure the motion of the wearable device (101). The sensor (240) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal representing the color and / or brightness of light. The one or more optical sensors may be referred to as an image sensor and / or a camera.

[0045] In one embodiment, the communication circuit (250) of the wearable device (101) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device (252). The communication circuit (250) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (250) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, near field communication (NFC), long term evolution (LTE), 5G new radio (NR), and / or 6G.

[0046] Referring to FIG. 2, according to one embodiment, a wearable device (101) may receive a signal for executing at least one function supported by the wearable device (101) from an external electronic device (252) connected via a communication circuit (250). For example, the external electronic device (252) may include one or more buttons, a joystick, and / or a touch sensor for receiving a user input. From the perspective of controlling the wearable device (101) based on a user input, the external electronic device (252) may be referred to as a remote controller. Based on the signal, the processor (210) of the wearable device (101) may execute the at least one function.

[0047] Although not illustrated, according to one embodiment, the wearable device (101) may include an output means for outputting information in a form other than a visualized form. For example, the wearable device (101) may include a speaker for outputting an acoustic signal. For example, the wearable device (101) may include a motor for providing haptic feedback based on vibration.

[0048] Referring to FIG. 2, one or more instructions (or commands) representing operations and / or actions to be performed on data by the processor (210) of the wearable device (101) may be stored in the memory (220) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in an electronic device (e.g., a wearable device (101)), it may mean that one or more instructions provided in the form of an application are stored in the memory (220), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the wearable device (101)). According to one embodiment, the wearable device (101) may perform the operations of FIGS. 9 to 11 by executing one or more instructions stored in the memory (220).

[0049] Referring to FIG. 2, a program stored in the memory (220) of the wearable device (101) for displaying one or more virtual objects (e.g., virtual objects (131, 132) of FIG. 1) is illustrated by different blocks. For example, the programs stored in the memory (220) of the wearable device (101) may be divided into a device information providing unit (261), a user view determining unit (262), a 3D object information collecting unit (263), a 3D object relationship determining unit (264), a 3D object information display unit (265), a 3D object display processing unit (266), a 3D object information management unit (267), and / or a 3D object output processing unit (268). Exchange of information and / or data between the above programs may occur based on an API (application programming interface). The embodiment is not limited thereto. For example, the processor (210) of the wearable device (101) may identify the position and / or direction of the wearable device (101) by processing and / or analyzing data from the sensor (240) based on a program stored in the memory (220). For example, the processor (210) of the wearable device (101) may identify user input by recognizing gestures performed by the user's gaze and / or hands.

[0050] Referring to FIG. 2, the processor (210) of the wearable device (101) can obtain information on the location, direction, and / or field of view of a user (e.g., the user (110) of FIG. 1) wearing the wearable device (101) based on the device information providing unit (261). The processor (210) can identify an external space visible to the user from the information using the user view determining unit (262). The processor (210) can obtain information on a virtual object and / or an external object visible through the display (230) based on the 3D object information management unit (267). For example, the processor (210) can obtain information on a virtual object provided from an application and / or obtain information on one or more external objects included in the external space.

[0051] Referring to FIG. 2, the processor (210) of the wearable device (101) may perform rendering of one or more virtual objects represented by information acquired by the 3D object information management unit (267) based on the 3D object display processing unit (266). The processor (210) may identify a 3D object (e.g., a virtual object and / or an external object) included in the user's field of view from the information acquired by the 3D object information management unit (267) based on the 3D object information collection unit (263). The processor (210) may identify a positional relationship between virtual objects and / or external objects in the user's field of view based on the 3D object relationship determination unit (264). The processor (210) can display a visual object (e.g., a visual object (150) of FIG. 1) corresponding to a specific virtual object, which is displayed to be covered by a virtual object and / or an external object based on the positional relationship, based on the 3D object information display unit (265). The processor (210) can display, on the display (230), a screen (e.g., a screen (140) of FIG. 1) including one or more virtual objects rendered by the 3D object display processing unit (266) and a visual object provided from the 3D object information display unit (265), based on the 3D object output processing unit (268).

[0052] As described above, according to one embodiment, the processor (210) of the wearable device (101) may perform an operation to identify a virtual object that is not displayed through the display (230) based on the 3D object information display unit (265), the 3D object relationship determination unit (264), and the 3D object information collection unit (263), and display a visual object corresponding to the virtual object. Information for displaying the visual object may be provided to the 3D object output processing unit (268) executed by the processor (210), and used to display the visual object within the display (230). Using the visual object displayed through the display (230), the processor (210) may receive an input for improving the visibility of the virtual object.

[0053] Hereinafter, with reference to FIGS. 3a to 3e, the operation of a wearable device (101) that displays visual objects based on the positional relationship of virtual objects is exemplarily described.

[0054] FIGS. 3A to 3E are drawings for explaining a screen displayed by a wearable device according to the positional relationship of a plurality of virtual objects in a virtual space. The wearable device (101) of FIGS. 1 to 2 can perform the operation of the wearable device described with reference to FIGS. 3A to 3E. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device described with reference to FIGS. 3A to 3E.

[0055] Referring to FIGS. 3A to 3E , different states (301, 302, 303, 304, 305) of a virtual space (120) managed by a wearable device are illustrated. According to one embodiment, the wearable device can place virtual objects (310, 320) provided from one or more applications within the virtual space (120). The wearable device can identify a portion of the virtual space (120) to be displayed via the display. In the exemplary case of FIGS. 3A to 3E in which the virtual space (120) includes virtual objects (310, 320), the wearable device can display each of the virtual objects (310, 320) on the display based on whether each of the virtual objects (310, 320) is included in the portion of the virtual space (120).

[0056] Referring to FIG. 3A, in a state (301) in which an FoV is formed along a direction (dv) from a reference point (pv) in a virtual space (120), the wearable device can display at least one of the virtual objects (310, 320) on the screen (330) based on the positional relationship of the virtual objects (310, 320) included in the FoV. On the xy plane of the virtual space (120), a part (321-2) of the virtual object (320) may be placed behind the virtual object (310) with respect to the reference point (pv), which is a viewpoint of the screen (330). For example, the distance between the reference point (pv) and the virtual object (310) may be shorter than the distance between the reference point (pv) and the virtual object (320). According to one embodiment, the wearable device can identify a positional relationship between virtual objects (310, 320) based on the coordinates of each of the virtual objects (310, 320) within the virtual space (120).

[0057] Referring to FIG. 3A, the wearable device can display a virtual object (310) having a two-dimensional plane shape within a screen (330). The screen (330) can correspond to a view of the virtual space (120) viewed from a reference point (pv) along a direction (dv). In an exemplary state (301) of FIG. 3A, the screen (330) can include an image of the virtual space (120) projected onto the xz plane of the reference point (pv). The virtual object (310) displayed through the screen (330) can have a form of a window provided by an application. The virtual object (320) within the virtual space (120) can have a form of a three-dimensional widget provided by the application. Although a virtual object (320) having a pyramid shape is illustrated as an example, the embodiment is not limited thereto.

[0058] In state (301) of FIG. 3A, the wearable device can display a virtual object (310) as an overlay on a virtual object (320) based on the positional relationship of the virtual objects (310, 320) described above. As the virtual object (320) is placed behind the virtual object (310), a portion (321-2) of the virtual object (320) can be covered by the virtual object (310) within the screen (330). In state (301) of FIG. 3A, a portion (321-1) of the virtual object (320) can be displayed within the screen (330).

[0059] In state (301) of FIG. 3a, according to an embodiment, the wearable device may change, based on a user input, virtual objects (310, 320) displayed on the screen (330) and / or a portion of the virtual space (120) to be displayed on the screen (330). For example, the wearable device may identify an input of moving the virtual object (310) along a direction (ds). The input may be identified by a gesture of the user performed by a hand, a signal transmitted from an external electronic device (e.g., the external electronic device (252) of FIG. 2), a gaze of the user, and / or a speech of the user. In state (301) of FIG. 3a, based on the input, the wearable device may switch from state (301) of FIG. 3a to state (302) of FIG. 3b. For example, the wearable device can rotate the FoV along the direction (df) about the reference point (pv) based on the rotation of the wearable device identified through a sensor (e.g., sensor (240) of FIG. 2). The rotation of the wearable device can be caused by the rotation of the head of a user wearing the wearable device. In state (301) of FIG. 3a, based on the rotation of the wearable device, the wearable device can switch from state (301) of FIG. 3a to state (303) of FIG. 3c.

[0060] Referring to FIG. 3B, based on the movement of the virtual object (310), within the view of the virtual space (120) viewed from the reference point (pv) along the direction (dv), the virtual object (320) may be completely occluded by the virtual object (310). In the state (302) of FIG. 3B, the wearable device according to one embodiment may identify that the virtual object (320) is occluded by the virtual object (310) within the screen (330) displayed through the display. For example, the wearable device may identify that the size of a portion within the screen (330) on which the virtual object (320) is displayed is reduced due to the movement of the virtual object (310). If the size is reduced below a specified threshold and / or becomes substantially zero, the wearable device may display a visual object (322) to improve, enhance, and / or restore the visibility of the virtual object (320). For example, if the ratio of a portion (e.g., portion (321-2) of FIG. 3a) where a virtual object (320) is displayed to another portion (e.g., portion (321-1) of FIG. 3a) exceeds a specified ratio (e.g., about 95%), the wearable device may display a visual object (322). In the exemplary state (302) of FIG. 3b, the wearable device may display a visual object (322) based on identifying a virtual object (320) included in an area where a virtual object (310) is displayed within the screen (330).

[0061] Within state (302) of FIG. 3B, the wearable device may display a visual object (322) corresponding to a virtual object (320) that is not visible through the screen (330) within the screen (330). In one embodiment, the wearable device may display the visual object (322) based on a period of time during which the virtual object (320) is not visible within the screen (330) as the virtual object (320) is obscured by the virtual object (310). For example, if the period of time during which the virtual object (320) is included within an area within the screen (330) where the virtual object (310) is displayed exceeds a specified period of time (e.g., several seconds), the wearable device may display the visual object (322).

[0062] Referring to FIG. 3B, in a state (302) of displaying a visual object (322) related to a virtual object (320) obscured by a virtual object (310), the wearable device may display the visual object (322) based on the position of the virtual object (310) obscuring the virtual object (320) within the virtual space (120). Referring to the xy plane of the virtual space (120) of FIG. 3B, the distance between the reference point (pv), which is a viewpoint for displaying the screen (330), and the visual object (322) may correspond to the distance between the reference point (pv) and the virtual object (310). In one embodiment, the wearable device may display the visual object (322) using binocular parallax based on the distance between the reference point (pv) and the virtual object (310). By utilizing the above binocular disparity, the wearable device can display a visual object (322) having a sense of distance from the virtual object (310) to a user wearing the wearable device (e.g., the user (110) of FIG. 1). For example, a visual object (322) in a first image displayed to the user's left eye by the wearable device can have a position shifted by the binocular disparity from a position of a visual object (322) in a second image displayed to the user's right eye. For example, the user can perceive that the visual object (322) is spaced apart from the user by the same distance as the virtual object (310).

[0063] Referring to FIG. 3B, in an exemplary state (302) of displaying a visual object (322) in the form of an icon representing a virtual object (320) covered by a virtual object (310), the wearable device may display the visual object (322) on a portion of the screen (330) adjacent to an upper edge of the virtual object (310) along the z-axis. The embodiment is not limited thereto, and the wearable device may display the visual object (322) on a portion of the screen (330) adjacent to another edge of the virtual object (310) that is different from the upper edge. The position of the visual object (322) with respect to the virtual object (310) may be adjusted by a parameter set by a user of the wearable device.

[0064] Referring to FIG. 3c, based on the rotation of the FoV within the virtual space (120), a virtual object (320) can be placed outside the FoV within the view of the virtual space (120) viewed from the reference point (pv) along the direction (dv'). The rotation of the FoV within the virtual space (120) can be related to a motion measured by a sensor of the wearable device (e.g., sensor (240) of FIG. 2). For example, when a user wearing the wearable device turns his or her head, the wearable device can identify the motion of the wearable device rotated by the head using the sensor. For example, the wearable device can change the FoV within the virtual space (120) based on the motion measured by the sensor. In one embodiment, the wearable device may display a visual object (e.g., a visual object (322)) related to the identified virtual object on the display based on identifying a virtual object (e.g., a virtual object (320)) that has moved out of the FoV due to a change in the FoV.

[0065] Within the state (303) of FIG. 3c, the wearable device according to one embodiment may identify a virtual object (320) that is not displayed within the screen (330) corresponding to the FoV as it is positioned outside the FoV. For example, the wearable device may identify that the size of the virtual object (320) displayed within the screen (330) decreases as it moves outside the FoV while switching from the state (301) of FIG. 3a to the state (303) of FIG. 3c. If the size decreases below a specified threshold and / or becomes substantially zero, the wearable device may display a visual object (322) corresponding to the virtual object (320).

[0066] Within the state (303) of FIG. 3c, the wearable device can display a visual object (322) within the screen (330) to control the movement of an invisible virtual object (320) through the screen (330). The wearable device can display the visual object (322) based on the position of the virtual object (310) displayed through the screen (330) within the virtual space (120). Within the exemplary state (303) of FIG. 3c, the distance between the visual object (322) displayed by the wearable device and the reference point (pv) can correspond to the distance between the virtual object (310) and the reference point (pv). For example, the wearable device can display the visual object (322) based on the binocular disparity of the virtual object (310). Referring to the screen (330) of FIG. 3c, the wearable device may display a visual object (322) in a different part of the screen (330) than the part of the screen (330) where the virtual object (310) is displayed. For example, the visual object (322) may be displayed in a location that does not overlap with other virtual objects within the screen (330).

[0067] According to one embodiment, the wearable device can determine the location of a visual object (322) within the screen (330) based on a direction from a reference point (pv) of the FoV to a virtual object (320) positioned outside the FoV. In the exemplary state (303) of FIG. 3C, the virtual object (320) can be positioned in the direction of the -x axis with respect to the direction (dv') of the central axis of the FoV. Based on identifying the virtual object (320) positioned outside the FoV along the direction of the -x axis, the wearable device can display a visual object (322) corresponding to the virtual object (320) at an edge (e.g., a left edge) of the screen (330) corresponding to the direction of the -x axis.

[0068] Referring to FIG. 3D, an exemplary state (304) is illustrated in which a plurality of virtual objects (310, 340) are arranged between a virtual object (320) that is not displayed on a screen (330) and a reference point (pv) corresponding to the screen (330). In the state (304), according to one embodiment, the wearable device can identify the visibility of the virtual object (320) with respect to the screen (330) based on the virtual object (310) having the largest size among the plurality of virtual objects (310, 340) arranged in front of the virtual object (320) with respect to the reference point (pv). For example, if the entire width and height of a first region within the screen (330) occupied by a virtual object (310) are greater than the width and height of a second region within the screen (330) corresponding to the virtual object (320), the wearable device may determine that the virtual object (320) is obscured by the virtual object (310) within the screen (330). For example, if the second region is included within the first region, the wearable device may determine that the virtual object (320) is obscured by the virtual object (310) within the screen (330).

[0069] According to one embodiment, the wearable device may, based on identifying a virtual object (320) covered by another virtual object (virtual object (310) in the exemplary state (304) of FIG. 3D) within the screen (330), display a visual object (322) corresponding to the virtual object (320) based on a position of the virtual object (340) closest to the reference point (pv) among virtual objects (310, 340) positioned in front of the virtual object (320) with respect to the reference point (pv). For example, the wearable device may identify, among the virtual objects (310, 340), a virtual object (340) having a minimum distance from the reference point (pv). The wearable device may display, on the screen (330), a visual object (322) positioned in the virtual space (120) based on the minimum distance. The wearable device may display a visual object (322) at a location different from the area within the screen (330) occupied by the virtual object (340). For example, the wearable device may display a visual object (322) at a location within the screen (330) that is separate from a virtual object (e.g., virtual objects (310, 340)) and / or a real object visible through the screen (330).

[0070] In one embodiment, when a plurality of virtual objects are covered by a specific virtual object within the screen (330), the wearable device may display a visual object for at least one of the plurality of virtual objects. Referring to an exemplary state (305) of FIG. 3E, in a state (305) where a plurality of virtual objects (310, 320, 340) are arranged within the virtual space (120), the wearable device according to one embodiment may identify the plurality of virtual objects (310, 320) covered by the virtual object (340) within the screen (330). When a plurality of virtual objects (310, 320) are covered by a single virtual object (340), the wearable device may display a visual object (312) corresponding to one virtual object (virtual object (310) within the state (305) of FIG. 3e) that is positioned relatively close to the reference point (pv) among the plurality of virtual objects (310, 320). For example, when a plurality of virtual objects (310, 320) are covered by the virtual object (340), the wearable device may display a visual object (312) corresponding to any one of the plurality of virtual objects (310, 320) based on the position of the virtual object (340) within the virtual space (120). Within the state (305) of FIG. 3e, the wearable device can display a visual object (312) based on a position of a virtual object (340) within the virtual space (120) that is spaced apart from a reference point (pv) by a minimum distance. For example, the distance between the visual object (312) and the reference point (pv) within the virtual space (120) can correspond to the distance between the virtual object (340) and the reference point (pv).

[0071] Although an exemplary state (305) is illustrated in which a visual object (312) corresponding to a virtual object (310) positioned relatively close to a reference point (pv) among a plurality of virtual objects (310, 320) is displayed, the embodiment is not limited thereto. For example, the wearable device may display visual objects corresponding to each of the virtual objects (310, 320) covered by the virtual object (340) within the screen (330). The visual objects may be displayed based on the position and / or binocular disparity of the virtual object (340) covering the virtual objects (310, 320).

[0072] Referring to state (305) of FIG. 3e, according to one embodiment, the wearable device may display a visual object (312) corresponding to at least one of the virtual objects (310, 320) covered by the virtual object (340) within the screen (330). For example, the wearable device may display the visual object (312) at a location within the virtual space (120) of the virtual object (340) that is closest to the reference point (pv) among the virtual objects (310, 320).

[0073] As described above, according to one embodiment, the wearable device can change the visibility of at least one of the virtual objects (310, 320, 330, 340) within the screen (330) by using the positional relationship of the virtual objects (310, 320, 330, 340) identified within the virtual space (120). The wearable device can support a function of improving the visibility by using visual objects (312, 322) corresponding to at least one of the virtual objects (310, 320, 330, 340).

[0074] Hereinafter, with reference to FIGS. 4A to 4B, the operation of a wearable device that receives inputs related to visual objects (312, 322) of FIGS. 3A to 3E is exemplarily described.

[0075] FIGS. 4A and 4B are drawings for explaining the operation of a wearable device (101) that displays a virtual object corresponding to a visual object based on an input related to the visual object. The wearable device (101) of FIGS. 1 and 2 can perform the operation of the wearable device (101) described with reference to FIGS. 4A and 4B. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device (101) described with reference to FIGS. 4A and 4B.

[0076] Referring to FIGS. 4A and 4B , exemplary states (401, 402) of a wearable device (101) displaying a screen (140) corresponding to a FoV formed within a virtual space (120) are illustrated. In state (401), the wearable device (101) can identify a virtual object (420) positioned behind a virtual object (410) along a direction (dv) from a reference point (pv) corresponding to a view point within the virtual space (120). Based on identifying a virtual object (420) obscured by the virtual object (410) within the screen (140), the wearable device (101) can display a visual object (422) corresponding to the virtual object (420). The wearable device (101) can display a visual object (422) based on the period of time during which the virtual object (420) is covered by the virtual object (410) within the screen (140).

[0077] In an exemplary state (401) of FIG. 4A, the wearable device (101) may display a virtual object (410) provided from a web browser application within the screen (140). For example, the wearable device (101) may display a virtual object (410) that is a two-dimensional plane including at least a portion of a website based on the execution of the web browser application. If the virtual object (420) is provided from a messenger application for sending and / or receiving messages, the wearable device (101) may display a visual object (422) including an icon of the messenger application based on the position of the virtual object (410) that obscures the virtual object (420) within the screen (140) and / or binocular disparity. Based on the visual object (422), the wearable device (101) can provide the user (110) with information about a virtual object (420) that is obscured by the virtual object (410) displayed through the screen (140) (e.g., an application corresponding to the virtual object (420)).

[0078] Within the exemplary state (401) of FIG. 4A, according to one embodiment, the wearable device (101) may identify an input indicating selection of a visual object (422). The input may be related to a motion and / or speech of a user (110) wearing the wearable device (101). For example, the wearable device (101) may identify a natural language-based utterance of the user (110) from an audio signal received through a microphone. For example, the wearable device (101) may identify an input indicating selection of a visual object (422) based on recognizing an utterance such as "I will select the icon displayed over the web browser" (or "Show the panel covered by the web browser").

[0079] For example, the wearable device (101) can identify the direction of the user's (110) gaze using a first camera positioned toward the user's (110) head while the wearable device (101) is worn on the user's (110) head. The wearable device (101) can determine the direction of the gaze based on the position and / or direction of the user's (110) pupil identified from the image acquired from the first camera. In the exemplary state (401) of FIG. 4A, the wearable device (101) can identify an input indicating selection of the visual object (422) based on the direction of the gaze toward the visual object (422). For example, if the user's (110) gaze is maintained in a direction toward the visual object (422) for more than a specified period of time, the wearable device (101) can identify an input indicating selection of the visual object (422). For example, if a blink (or wink) of the eye occurs while identifying the direction of the gaze toward the visual object (422), the wearable device (101) can identify an input indicating selection of the visual object (422).

[0080] For example, the wearable device (101) can identify a body part (e.g., a hand) of the user (110) by using a second camera positioned in a direction different from the direction in which the head is facing while the wearable device (101) is worn on the head of the user (110). The wearable device (101) can identify whether a gesture indicating selection of a visual object (422) has been performed based on an image acquired from the second camera. The gesture indicating selection of a visual object (422) can be identified by a hand including a designated finger (e.g., an index finger) selectively extended among a plurality of fingers. The wearable device (101) can identify the gesture based on the direction of the designated finger toward a portion of external space corresponding to the visual object (422). The embodiment is not limited to the pointing gesture described above, and a gesture indicating selection of a visual object (422) may include a gesture such as a pinch gesture, such as holding the visual object (422) by contact between the fingertips of two fingers. Based on the gesture, the wearable device (101) can identify an input indicating selection of a visual object (422).

[0081] For example, the wearable device (101) may receive a signal from an external electronic device (252), which is a remote controller, indicating selection of a visual object (422). The wearable device (101) may receive the signal from the external electronic device (252) connected (or paired) via a communication circuit (e.g., the communication circuit (250) of FIG. 2). In the exemplary state (401) of FIG. 4A, the user (110) may hold the external electronic device (252), move the external electronic device (252), and / or press one or more buttons included in the external electronic device (252). The signal transmitted from the external electronic device (252) to the wearable device (101) may include information about the motion of the user (110) holding the external electronic device (252) and / or one or more buttons pressed by the user (110). Based on the signal, the wearable device (101) can identify the direction (dc) of the external electronic device (252). Based on identifying the direction (dc) based on the signal, the wearable device (101) can display a visual guide (e.g., a ray) extending along the direction (dc) from the external electronic device (252) visible through the screen (140). When the direction (dc) identified by the signal is toward the visual object (422) and a signal indicating the pressing of a designated button (e.g., a click button) from the external electronic device (252) is received, the wearable device (101) can identify an input indicating the selection of the visual object (422).

[0082] The wearable device (101) in the state (401) of Fig. 4a can switch to the state (402) of Fig. 4b in response to an input indicating selection of a visual object (422). In the state (402) of Fig. 4b, the wearable device (101) can improve and / or enhance the visibility of a virtual object (420) corresponding to the visual object (422) based on the input. For example, the wearable device (101) can place the virtual object (420) that was covered by the virtual object (410) in front of the virtual object (410) with respect to a reference point (pv) corresponding to a view point of the screen (140). According to one embodiment, the wearable device (101) can move the virtual object (420) based on the position of the virtual object (410) that is placed closest to the reference point (pv) within the virtual space (120) based on the input.

[0083] Referring to FIG. 4b, based on a virtual object (420) moved forward of a virtual object (410) with respect to a reference point (pv) within a virtual space (120), the wearable device (101) can display a virtual object (420) as an overlay on the virtual object (410) within a screen (140) corresponding to a view of the virtual space (120) as seen from the reference point (pv). Within a state (402) of FIG. 4b in which a virtual object (420) corresponding to a visual object (422) is displayed based on an input related to the visual object (422) of FIG. 4a, the wearable device (101) can at least temporarily stop displaying the visual object (422) within the screen (140). Based on the virtual object (420) moving forward of the virtual object (410) in a direction opposite to the direction (dv) extended from the reference point (pv), the wearable device (101) can display a virtual object (420) superimposed on the virtual object (410). Based on the input, the wearable device (101) can improve the visibility of the virtual object (420).

[0084] As described above, according to one embodiment, the wearable device (101) can move a virtual object (420) covered by a virtual object (410) within a virtual space (120) to a location within the screen (140) that is not covered by the virtual object (410). The movement of the virtual object (420) can be performed by a user input related to a visual object (422) displayed by the wearable device (101). According to one embodiment, the wearable device (101) can adjust the visibility of the virtual object (420) based on the movement of the virtual object (420) covered by the virtual object (410). When a plurality of virtual objects are not displayed through the screen (140), the wearable device (101) can individually adjust the visibilities of the plurality of virtual objects by using visual objects corresponding to each of the plurality of virtual objects.

[0085] An embodiment has been described in which a virtual object (420) covered by a virtual object (410) moves in front of the virtual object (410) with respect to a reference point (pv) corresponding to the screen (140), but the embodiment is not limited thereto. Hereinafter, operations of the wearable device (101) adjusting the visibility of the virtual object are exemplarily described with reference to FIGS. 5A to 5E.

[0086] FIGS. 5A to 5E are drawings for explaining the operation of a wearable device that adjusts the position of a virtual object corresponding to a visual object in a virtual space based on an input related to the visual object. The wearable device (101) of FIGS. 1 to 2 can perform the operation of the wearable device described with reference to FIGS. 5A to 5E. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device (101) described with reference to FIGS. 5A to 5E.

[0087] Referring to FIGS. 5A to 5E , different states (501, 502, 503, 504, 505) of a virtual space (120) managed by a wearable device according to one embodiment are illustrated. According to one embodiment, the wearable device may place virtual objects (520, 530, 540, 550, 560) provided from processes executed by a processor of the wearable device (e.g., processor 210 of FIG. 2 ) within the virtual space (120). The processes executed by the processor may be units of work of the processor corresponding to applications stored in a memory of the wearable device (e.g., memory 220 of FIG. 2 ). Virtual objects (520, 530, 540, 550, 560) placed within the virtual space (120) may include UIs provided from the processes executed by the wearable device based on multi-tasking.

[0088] Referring to FIG. 5A, exemplary positions of virtual objects (520, 530, 540, 550, 560) placed in a virtual space (120) by a wearable device are illustrated. The positions of the virtual objects (520, 530, 540, 550, 560) in a state (501) may be determined or adjusted by a user input for changing the position of at least one virtual object and / or an application executed by the wearable device. The wearable device may perform rendering on one or more virtual objects included in a FoV formed based on a reference point (pv) in the virtual space (120) corresponding to a viewpoint, thereby obtaining a screen (510) to be displayed through the display. Within the exemplary state (501) of FIG. 5A, the wearable device can perform rendering on virtual objects (520, 530, 540, 550) included in the FoV to obtain a screen (510).

[0089] Referring to FIG. 5A, in a state (501) in which an FoV is formed along a direction (dv) from a reference point (pv), the wearable device can identify the visibilities of virtual objects (520, 530, 540, 550) included in the FoV based on the positional relationship of the virtual objects (520, 530, 540, 550). For example, the wearable device that identifies that the virtual object (530) is positioned behind the virtual object (520) along a direction (dv) extending from the reference point (pv) can identify that the virtual object (530) is covered by the virtual object (520) within the screen (510). For example, a wearable device that identifies that a virtual object (550) is positioned behind a virtual object (540) along a direction (dw) extending from a reference point (pv) can identify that the virtual object (540) is superimposed on the virtual object (550) within the screen (510).

[0090] Referring to FIG. 5A, a wearable device that identifies a virtual object (530) covered by a virtual object (520) within a screen (510) can display a visual object (532) corresponding to the virtual object (530) within the screen (510). A wearable device that identifies a virtual object (540) overlapping a virtual object (550) within the screen (510) can display a visual object (552) corresponding to the virtual object (550) within the screen (510). Using the visual objects (532, 552), the wearable device can support a function of changing the positions of the virtual objects (530, 550) corresponding to each of the visual objects (532, 552). The positions of the visual objects (532, 552) within the screen (510) and / or the virtual space (120) may be similar to the positions at which the visual objects (312) of FIGS. 3A to 3E are displayed. For example, the wearable device may display the visual object (532) based on the distance (or binocular disparity) between the virtual object (520) that obscures the virtual object (530) and the reference point (pv). For example, the wearable device may display the visual object (552) based on the position of the virtual object (540) that is displayed as an overlay on the virtual object (550) within the screen (510).

[0091] Referring to FIG. 5A, according to one embodiment, a wearable device may display a visual object (562) corresponding to a virtual object (560) positioned outside the FoV within the virtual space (120). The location of the visual object (562) on the screen (510) and / or within the virtual space (120) may be similar to the location at which the visual object (322) of FIGS. 3A to 3E is displayed. For example, the wearable device may identify a virtual object (520) that is spaced a minimum distance from a reference point (pv) within the FoV, and display the visual object (562) based on the minimum distance.

[0092] Within the exemplary state (501) of FIG. 5A, the wearable device may identify an input for selecting at least one of the visual objects (532, 552, 562). The input may be identified based on a gaze, a hand gesture, a speech, and / or an external electronic device connected to the wearable device (e.g., the external electronic device (252) of FIG. 2 and / or FIG. 4A) of a user wearing the wearable device, as described above with reference to FIG. 4A. In response to an input indicating a selection of a visual object (532), the wearable device may switch from state (501) to state (502) of FIG. 5B and / or state (503) of FIG. 5C. In response to an input indicating a selection of a visual object (552), the wearable device may switch from state (501) to state (504) of FIG. 5D. In response to an input indicating a selection of a visual object (562), the wearable device may switch from state (501) to state (505) of FIG. 5e. Based on an input indicating a selection of any one of the visual objects (532, 552, 562), the wearable device may stop displaying the visual object corresponding to the input.

[0093] Referring to FIG. 5b, based on an input indicating selection of a visual object (532) of FIG. 5a, the wearable device can move a virtual object (530) corresponding to the visual object (532). For example, the wearable device can move the virtual object (530) so that the virtual object (530) is not covered by other virtual objects (e.g., the virtual object (520)) within the screen (510). The wearable device can move the virtual object (530) positioned behind the virtual object (520) along a direction (dv) extended from a reference point (pv) within the virtual space (120) corresponding to the viewpoint of the screen (510) toward the reference point (pv) along another direction opposite to the direction (dv). In the state (502) of FIG. 5b, the virtual object (530) moved toward the reference point (pv) can be positioned in front of the virtual object (520) with respect to the reference point (pv).

[0094] Referring to FIG. 5b, the virtual object (530) may have a three-dimensional shape (e.g., a pyramid) within the virtual space (120). While moving the virtual object (530) in front of the virtual object (520) with respect to the reference point (pv) within the virtual space (120), the wearable device may move the virtual object (530) based on the size of the virtual object (530) and / or the position of the virtual object (520) within the virtual space (120) in order to prevent overlap between the virtual objects (520, 530) due to the movement of the virtual object (530). In the exemplary state (502) of FIG. 5b, the wearable device moving the virtual object (530) in a direction opposite to the direction (dv) may move the virtual object (530) in front of the virtual object (520) by the length (l) of the virtual object (530) with respect to the direction (dv). Since the virtual object (530) is moved in front of the virtual object (520) by a length (l) along the other direction, the virtual objects (520, 530) can be placed in positions that do not overlap each other within the virtual space (120).

[0095] Referring to FIG. 5B, based on the movement of the virtual object (530), the wearable device can display the virtual object (530) as an overlay on the virtual object (520) within the screen (510). In response to an input for the visual object (532) of FIG. 5A, the wearable device can enhance the visibility of the virtual object (530) corresponding to the visual object (532) based on the movement of the virtual object (530) within the virtual space (120). The embodiment is not limited thereto, and in response to the input, the wearable device can move not only the virtual object (530) but also the virtual object (520) that obscures the virtual object (530) within the screen (510). For example, the wearable device can swap the positions of the virtual objects (520, 530) within the virtual space (120). A wearable device that swaps the positions of virtual objects (520, 530) can adjust the positions by using the size (e.g., length (l)) of one of the virtual objects (520, 530) to prevent overlapping of the virtual objects (520, 530).

[0096] Referring to FIG. 5C, based on an input indicating selection of a visual object (532) of FIG. 5A, the wearable device can move all of the virtual objects (530) corresponding to the visual object (532) and the virtual objects (520) covering the virtual objects (530) within the screen (510). In the exemplary state (503) of FIG. 5C, the wearable device can move the virtual object (530) to a position of the virtual object (520) positioned in front of the virtual object (530) with respect to a reference point (pv) within the virtual space (120). Along with moving the virtual object (530) to the position, the wearable device can move the virtual object (520). For example, the wearable device can move the virtual object (520) based on a layout for aligning positions of one or more virtual objects within the screen (510). Referring to FIG. 5c, the wearable device can move a virtual object (520) to a location set by the layout within the screen (510). The layout may be pre-registered by the user of the wearable device. The layout may be related to information regarding the locations and / or arrangement of one or more virtual objects within the screen (510).

[0097] In one embodiment, a wearable device that moves a virtual object (530) in response to an input related to a visual object (532) of FIG. 5A may move another virtual object (e.g., a virtual object (520)) within the screen (510) and / or the virtual space (120). For example, the wearable device may display the virtual object (520) in a different area within the screen (510) than the area occupied by the virtual object (530). As in the screen (510) of FIG. 5B, the wearable device may move the virtual object (520) to prevent virtual objects (520, 530) from being displayed overlapping each other due to the movement of the virtual object (530).

[0098] Referring to FIG. 5d, based on an input indicating selection of a visual object (552) of FIG. 5a, the wearable device can move a virtual object (550) corresponding to the visual object (552). Within the exemplary state (504) of FIG. 5d, the wearable device can move the virtual object (550) in front of the virtual object (540) in a different direction opposite to the direction (dw) extending from the reference point (pv). As the virtual object (550) is moved in front of the virtual object (540), the wearable device can display the virtual object (550) as an overlay on the virtual object (540) within the screen (510).

[0099] According to one embodiment, the wearable device can move a virtual object (550) covered by a virtual object (540) to a part of the virtual space (120) having a distance less than or equal to the distance between the reference point (pv) and the virtual object (540). For example, the wearable device can move the virtual object (550) to an area (555) within the screen (510). The area (555) within the screen (510) can be formed in the direction (dv) of the central axis of the FoV. The area (555) can be formed in front of the virtual object (520) that is spaced apart from the reference point (pv) by a minimum distance among the virtual objects within the screen (510). Based on displaying the virtual object (550) in the area (555), the wearable device can move the virtual object (550) focused by the user to the center of the screen (510).

[0100] Referring to FIG. 5E, based on an input indicating selection of a visual object (562) of FIG. 5A, the wearable device can move a virtual object (560) corresponding to the visual object (560) into the FoV. In state (505) of FIG. 5E, since the virtual object (560) is moved into the FoV, the wearable device can display the virtual object (560) in the screen (510) corresponding to the FoV. The wearable device can move the virtual object (560) into the FoV based on whether it overlaps with another virtual object (e.g., virtual objects (520, 540)) displayed in the screen (510). For example, the wearable device can move the virtual object (560) toward the direction (dv) of the central axis of the FoV until another virtual object is placed behind or in front of the virtual object (560) with respect to the reference point (pv). Based on the above movement of the virtual object (560), within the screen (510), the distance between the direction (dv) of the central axis of the screen (510) and the virtual object (560) may be reduced.

[0101] As described above, according to one embodiment, the wearable device can enhance the visibility of a virtual object corresponding to a visual object within a virtual space (120) based on an input related to a visual object (e.g., visual objects (532, 552, 562)) corresponding to the virtual object. To enhance the visibility of the virtual object, the wearable device can move the virtual object toward a reference point (pv) corresponding to a view point of a screen (510) within the virtual space (120). Although the operation of the wearable device moving the virtual object toward the reference point (pv) has been described with reference to FIGS. 5A to 5E, the embodiment is not limited thereto.

[0102] With reference to FIGS. 1 to 5e, visual objects including icons representing virtual objects (e.g., visual objects (532, 552, 562)) are illustrated, but the embodiment is not limited thereto. Hereinafter, with reference to FIG. 6, an exemplary operation of a wearable device displaying a visual object corresponding to a virtual object is described.

[0103] FIG. 6 is a diagram for explaining the operation of a wearable device that displays a visual object corresponding to another virtual object that is obscured by a specific virtual object within a display. The wearable device (101) of FIGS. 1 and 2 can perform the operation of the wearable device described with reference to FIG. 6. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device described with reference to FIG. 6.

[0104] Referring to FIG. 6, an exemplary state (601) of a virtual space (120) identified by a wearable device according to one embodiment is illustrated. Within the virtual space (120), virtual objects (620, 630, 640, 650, 660) provided from an application executed by a processor of the wearable device (e.g., processor 210 of FIG. 2) may be placed. Within a state (601) in which the wearable device displays a screen (610) with respect to a reference point (pv) within the virtual space (120), the wearable device may identify a virtual object (630) placed behind a virtual object (620) with respect to the reference point (pv), a virtual object (650) placed behind a virtual object (640), and a virtual object (660) placed outside a FoV corresponding to the screen (610).

[0105] In one embodiment, the wearable device may display visual objects (634, 654, 664) corresponding to each of the virtual objects (630, 650, 660) based on identifying virtual objects (630, 650, 660) that are not visible to a user (e.g., the user (110) of FIG. 1). The wearable device may display a visual object (634) including a two-dimensional image of the virtual object (630) covered by the virtual object (620) within the screen (610), within the screen (610). The two-dimensional image included in the visual object (634) may correspond to a screenshot (or snapshot) of the virtual object (630) as viewed from a reference point (pv). The wearable device may display the visual object (634) based on the location of the virtual object (620) within the screen (610) and / or the virtual space (120). For example, the wearable device may display a visual object (634) based on the distance between the virtual object (620) and the reference point (pv). The wearable device may display the visual object (634) at a location adjacent to a portion of the screen (610) occupied by the virtual object (620). Referring to FIG. 6, the wearable device may display the visual object (634) at another portion moved along the +z direction from the portion of the screen (610) where the virtual object (620) is displayed.

[0106] Referring to FIG. 6, similarly to displaying a visual object (634) corresponding to a virtual object (630) based on a virtual object (620) that covers the virtual object (630), the wearable device may display visual objects (654, ​​664) corresponding to each of the virtual objects (650, 660). For example, the wearable device may display a visual object (654) corresponding to the virtual object (650) together with the virtual object (640) within the screen (610) in a state (601) of displaying a virtual object (640) as an overlay on the virtual object (650). The wearable device may display a visual object (654) connected to the virtual object (640) along the +z direction within the screen (610). For example, the wearable device may display a visual object (664) corresponding to a virtual object (660) positioned outside the FoV within the virtual space (120) on the edge of the screen (610) adjacent to the virtual object (660). In the exemplary state (601) of FIG. 6, the wearable device that has identified a virtual object (660) positioned outside the FoV along the direction of the -x axis may display the visual object (664) on the edge of the screen (610) corresponding to the direction of the -x axis.

[0107] In one embodiment, the wearable device can identify an input for each of the visual objects (634, 654, 664) corresponding to the virtual objects (630, 650, 660). Based on the input, the wearable device can change the position of the virtual object associated with the input within the virtual space (120), similarly to what has been described with reference to FIGS. 4A to 4B and 5A to 5E. For example, in response to an input indicating selection of a visual object (634), the wearable device can move the virtual object (630) corresponding to the visual object (634) toward the reference point (pv). For example, based on an input for a visual object (654), the wearable device can move the virtual object (650) corresponding to the visual object (654) in front of the virtual object (640) within the screen (610) that obscures the virtual object (650). For example, in response to an input related to a visual object (664), the wearable device may move a virtual object (660) corresponding to the visual object (664) into the FoV. In the above examples, the operation of the wearable device moving the virtual objects (630, 650, 660) may be performed similarly to the operation described above with reference to FIGS. 4A to 4B and FIGS. 5A to 5E.

[0108] As described above, according to one embodiment, the wearable device can identify a virtual object (e.g., virtual object 630) that cannot be displayed through the screen (610) due to overlap between virtual objects (e.g., virtual objects (620, 630)). The wearable device can display a visual object (e.g., visual object (634)) for movement of the identified virtual object. In response to an input indicating selection of the visual object, the wearable device can display a virtual object that is not displayed on the screen (610) within the screen (610). To display the virtual object that is not displayed on the screen (610) within the screen (610), the wearable device can move the virtual object within the virtual space (120) toward a reference point (pv) corresponding to a viewpoint of the screen (610). The wearable device can provide the user with an option to enhance the visibility of the virtual object that is not displayed on the screen (610).

[0109] According to one embodiment, a wearable device may display a screen in which one or more virtual objects and the external space are mixed using a virtual space (120) mapped to the external space. The wearable device may display a visual object for adjusting the visibility of the virtual object according to the shape of the external space (e.g., the shape of the external space formed by a wall and / or a room). Hereinafter, with reference to FIGS. 7 and 8A to 8C, an operation of the wearable device for enhancing the visibility of a virtual object obscured by a real object will be described.

[0110] FIG. 7 is an exemplary diagram for explaining the operation of a wearable device (101) for identifying a virtual space mapped to an external space. The wearable device (101) of FIGS. 1 and 2 can perform the operation of the wearable device (101) described with reference to FIG. 7. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device (101) described with reference to FIG. 7.

[0111] Referring to FIG. 7, an exemplary state (701) of a wearable device (101) that has acquired information about an external space is illustrated. According to one embodiment, the wearable device (101) may acquire information about an external space including the wearable device (101) using a camera and / or a sensor (e.g., sensor (240) of FIG. 2). The information about the external space may include at least one of a relative position of one or more external objects included in the external space with respect to the wearable device (101), a type (e.g., class and / or category) of the one or more external objects, or a size of the one or more external objects. The information about the external space may include a position and / or orientation of the wearable device (101) within the external space. According to one embodiment, the wearable device (101) may acquire a virtual space (710) mapped with the external space using the information about the external space based on simultaneous localization and mapping (SLAM).

[0112] Referring to FIG. 7, a virtual space (710) identified by a wearable device (101) included in an exemplary external space and mapped with the external space is exemplarily illustrated. That the virtual space (710) is mapped with the external space may mean that a spatial coordinate system of the virtual space (710) and a spatial coordinate system of the external space are at least partially synchronized. That the virtual space (710) is mapped with the external space may mean that the wearable device (101) has acquired positions and / or coordinates of one or more external objects in the external space based on the spatial coordinate system of the virtual space (710). Within the exemplary state (701) of FIG. 7, the wearable device (101) may acquire coordinates within the virtual space (710) corresponding to a flower vase placed in the external space and / or a wall structure included in the external space. For example, a location (R1) within a virtual space (710) may correspond to an external object (e.g., a flower vase) identified by a wearable device (101). For example, a positional relationship between a location (L1) within a virtual space (710) and a location (R1) may correspond to a positional relationship between a location (L1) of a wearable device (101) and an external object (e.g., a flower vase) within an external space.

[0113] According to one embodiment, the wearable device (101) may display a screen corresponding to an FoV formed based on a position (L1) of the wearable device (101) within a virtual space (710) within a display (e.g., the display (230) of FIG. 2). According to one embodiment, the wearable device (101) may identify a virtual object covered by an external object within the display based on a position of the external object identified based on a virtual space (710) mapped with an external space. In order to enhance the visibility of the virtual object covered by the external object, the wearable device (101) may display a visual object corresponding to the virtual object on the display. In response to an input related to the visual object, the wearable device (101) may move the virtual object within the virtual space (710). Based on the movement of the virtual object, the wearable device (101) can move the virtual object that was covered by the external object to a position where it is not covered by the external object. Hereinafter, with reference to FIGS. 8A to 8C, an exemplary operation of the wearable device (101) performed to improve the visibility of a virtual object covered by an external object within the exemplary state (701) of FIG. 7 is described.

[0114] FIGS. 8A to 8C are exemplary drawings for explaining the operation of a wearable device (101) related to the positional relationship between an external object and a virtual object. The wearable device (101) of FIGS. 1 to 2 can perform the operation of the wearable device (101) described with reference to FIGS. 8A to 8C. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device (101) described with reference to FIGS. 8A to 8C.

[0115] Referring to FIGS. 8A to 8C , different states (801, 802, 803) of a virtual space (710) managed by a wearable device (101) according to one embodiment are illustrated. The wearable device (101) may display a screen (810) that is a mixture of virtual objects and external objects based on the virtual space (710) mapped to the external space, as described above with reference to FIG. 7 . When the wearable device (101) provides a user experience based on VST, the wearable device (101) may display a screen (810) that includes images and / or videos of the external space identified through a camera. In the exemplary state (801) of FIG. 8A , a user (110) wearing the wearable device (101) may identify an external object (840) (e.g., a vase) within the external space through the screen (810). According to one embodiment, the wearable device (101) can identify an external object (840) visible through the display and / or screen (810) based on the image and / or the video. The wearable device (101) identifying the external object (840) may include an operation of identifying a location (R1) corresponding to the external object (840) within the virtual space (710).

[0116] Referring to FIG. 8A, in a state (801) in which a location (L1) of a user (110) in an external space is identified, the wearable device (101) can display a screen (810) corresponding to an FoV formed at a location (L1) in a virtual space (710) corresponding to the location (L1) in the external space. The wearable device (101) that has identified virtual objects (820, 830) having locations (V1, V2) included in the FoV in the virtual space (710) can display the virtual objects (820, 830) based on the locations (V1, V2) in the screen (810). For example, the wearable device (101) can set the binocular parallax of the virtual object (820) displayed on the screen (810) based on the distance between the reference point (L1) corresponding to the viewpoint of the screen (810) and the location (V1) of the virtual object (820) within the virtual space (710). For example, the wearable device (101) can determine the depth perception of the virtual object (830) displayed on the screen (810) based on the distance between the reference point (L1) and the location (V2) of the virtual object (830) within the virtual space (710). In the exemplary state (801) of FIG. 8A, the wearable device (101) can display a screen (810) in which an external object (840) of an external space and virtual objects (820, 830) are mixed to a user (110) wearing the wearable device (101).

[0117] In an exemplary state (801) of FIG. 8A, according to one embodiment, the wearable device (101) can identify a gesture of a user (110) for interacting with at least one of virtual objects (820, 830). The gesture can be identified by an external electronic device (252) having a form that can be held in the hand of the user (110). The gesture can be identified by a body part (e.g., a hand) of the user (110) that is tracked and / or monitored by the wearable device (101). Based on the gesture, the wearable device (101) can execute at least one function supported by a virtual object corresponding to the gesture.

[0118] In one embodiment, the wearable device (101) can determine the visibility of a particular virtual object on the screen (810) using other virtual objects within the virtual space (120) and / or external objects identified by a mapping between the virtual space (120) and the external space. In state (802) of FIG. 8B where the user has moved from a location (L1) within the external space to a location (L2), the wearable device (101) can display one or more virtual objects on the screen (810) based on a location (L2) within the virtual space (710) corresponding to the location (L2) within the external space.

[0119] Referring to FIG. 8B, within a state (802), a location (V2) corresponding to a virtual object (830) of FIG. 8A may be positioned behind a wall (850) of an external space with respect to a reference point (L2) corresponding to a screen (810). A wearable device (101) that has identified a location (V2) within a virtual space (710) positioned behind a wall (850) of an external space may refrain from displaying the virtual object (830) of FIG. 8A within the screen (810) and may display a visual object (832) corresponding to the virtual object (830). The display of the visual object (832) by the wearable device (101) may depend on a period of time during which the virtual object (830) is obscured by an external object (e.g., a wall (850)) visible through the screen (810). The wearable device (101) can display a visual object (832) based on the area and / or size of the virtual object (830) covered by an external object (e.g., a wall (850)) visible through the screen (810). The wearable device (101) can identify whether the virtual object (830) is covered by an external object (e.g., a wall (850)) visible through the display by comparing a portion of the virtual space (710) corresponding to the external object and a location of the virtual object (830) within the virtual space (710) based on the operations described above with reference to FIGS. 3A to 3E.

[0120] According to one embodiment, the wearable device (101) may display a visual object (832) including a two-dimensional image of the virtual object (830) of FIG. 8A, within the screen (810). The two-dimensional image may be an image expressing the appearance of the virtual object (830) based on two-dimensional rendering of the virtual object (830) of FIG. 8A. The wearable device (101) may display the visual object (832) as an overlay on an external object (e.g., a wall (850)) within the screen (810) that covers the virtual object (830) of FIG. 8A corresponding to the visual object (832). For example, the position of the visual object (832) within the virtual space (710) may be related to the position of an external object (e.g., a wall (850)) between the visual object (832) and a reference point (L1) within the virtual space (710). Based on the visual object (832), the wearable device (101) can provide the user (110) with information about a virtual object (e.g., a virtual object (830) of FIG. 8A) that is obscured by an external object (e.g., a wall (850)) visible through the screen (810).

[0121] According to one embodiment, the wearable device (101) may, based on an input indicating selection of a visual object (832), move a virtual object (e.g., virtual object (830) of FIG. 8A) corresponding to the visual object (832) to a location within the virtual space (710) that is not obscured by other virtual objects and / or external objects. In state (802) of FIG. 8B, based on an input for the visual object (832), the wearable device (101) may switch to state (803) of FIG. 8C.

[0122] Referring to FIG. 8C, according to one embodiment, the wearable device (101) may move the virtual object (830) forward of an external object (e.g., a wall (850)) positioned in front of the virtual object (830) corresponding to the visual object (832) based on the input. The wearable device (101) may move the virtual object (830) to a part of the virtual space (710) formed between the location in the virtual space (710) corresponding to the wall (850) and the reference point (L2), based on the input. When the virtual object (830) has a three-dimensional shape, the wearable device (101) may move the virtual object (830) based on the size of the virtual object (830). For example, the wearable device (101) may move the virtual object (830) so that the areas corresponding to the virtual object (830) and other virtual objects and / or external objects within the virtual space (710) do not overlap each other. For example, within the state (803) of FIG. 8c, the location (V2) of the virtual object (830) within the virtual space (710) may be spaced apart from a portion corresponding to the wall (850) within the virtual space (710) by the size of the virtual object (830).

[0123] In one embodiment, based on the movement of the virtual object (830), the wearable device (101) can display the virtual object (830) within the screen (810). Based on the position (V2) of the virtual object (830) moved to the part within the virtual space (710), the wearable device (101) can display the virtual object (830) within the screen (810). In the exemplary state (803) of FIG. 8C, the wearable device (101) can display the virtual object (830) as an overlay on an external object (e.g., a wall (850)) visible through the screen (810). For example, the binocular parallax of the virtual object (830) may be greater than the binocular parallax of the wall (850) visible through the screen (810). Based on the above binocular disparity, a user (110) wearing a wearable device (101) can recognize a virtual object (830) floating closer than a wall (850).

[0124] As described above, according to one embodiment, the wearable device (101) may provide a shortcut (e.g., a visual object (832)) to a virtual object (e.g., a virtual object (830)) obscured by an external object visible through the screen (810). Based on an input for the shortcut, the wearable device (101) may improve the visibility of the virtual object corresponding to the shortcut.

[0125] Below, the operation of the wearable device (101) based on the above-described drawings is described with reference to FIGS. 9 to 11.

[0126] FIG. 9 is a flowchart illustrating operations of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1 and 2 may perform the operations of the wearable device described with reference to FIG. 9. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIG. 9.

[0127] Referring to FIG. 9, in operation (910), a processor of a wearable device according to one embodiment may identify a reference point (e.g., a viewpoint) within a virtual space mapped with an external space based on a location of the wearable device within the external space. The virtual space of operation (910) may include the virtual space (120) of FIGS. 1 , 4A to 4B, 5A to 5E, and 6, and / or the virtual space (710) of FIGS. 7 and 8A to 8C. The reference point of operation (910) may correspond to a viewpoint set within the virtual space for displaying a screen displayed through a display of the wearable device (e.g., the display (230) of FIG. 2). The reference point of operation (910) may be determined as a location within the virtual space corresponding to the location of the wearable device within the external space.

[0128] Referring to FIG. 9, in operation (920), a processor of a wearable device according to one embodiment may identify a first virtual object included in a FoV formed within a virtual space based on a reference point. The processor may determine the visibility of the first virtual object identified by operation (920) based on the operations of FIG. 9 and / or FIG. 10, which will be described later.

[0129] Referring to FIG. 9, in operation (930), the processor of the wearable device according to one embodiment may identify a real object and / or a second virtual object having a position corresponding to a portion of the virtual space between a first virtual object and a reference point. When a plurality of virtual objects are arranged in the virtual space, the processor may identify a second virtual object arranged in front of the first virtual object of operation (920) with respect to the reference point of operation (910) based on the positional relationship of the plurality of virtual objects. Within a virtual space mapped with an external space, the processor may identify a real object within the external space corresponding to a portion of the virtual space between the reference point of operation (910) and the first virtual object of operation (920). If no real object and / or second virtual object that reduces the visibility of the first virtual object is identified between the reference point and the first virtual object (930-No), the processor may perform operation (940). In a state where a real object and / or a second virtual object that reduces the visibility of a first virtual object is identified based on the action (930) (930-example), the wearable device can perform the action (950).

[0130] Referring to FIG. 9, in operation (940), according to one embodiment, the processor of the wearable device may display a screen including a first virtual object based on the rendering of the first virtual object. If the actual object and / or the second virtual object of operation (930) is not identified, the processor may display a screen including the first virtual object.

[0131] Referring to FIG. 9, in operation (950), according to one embodiment, a processor of a wearable device may display a screen including a visual object representing a first virtual object based on a location of a real object and / or a second virtual object. If a real object and / or a second virtual object that obscures the first virtual object is identified, the processor may display a visual object corresponding to the first virtual object instead of displaying the first virtual object. The visual object of operation (950) may include the visual object (150) of FIG. 1, the visual object (322) of FIGS. 3B to 3D, the visual object (312) of FIG. 3E, the visual object (422) of FIG. 4A, the visual objects (532, 552, 562) of FIGS. 5A to 5E, the visual objects (634, 654, 664) of FIG. 6, and / or the visual object (832) of FIG. 8B. Based on the screen of the action (950), the processor can display a visual object to change the visibility of the first virtual object that is obscured by the real object and / or the second virtual object.

[0132] Referring to FIG. 9, within operation (960), a processor of a wearable device according to one embodiment may receive an input indicating a selection of a visual object. The input of operation (960) may be received based on a gesture performed by a user wearing the wearable device, a gaze or speech of the user, and / or an external electronic device connected to the wearable device (e.g., the external electronic device (252) of FIG. 2 ), as described above with reference to FIGS. 4A and 4B . If the input of operation (960) is not received, or prior to receiving the input of operation (960) (960-No), the processor may continue to display the screen of operation (950). In response to receiving the input of operation (960), the processor may perform operation (970).

[0133] Referring to FIG. 9, in operation (970), a processor of a wearable device according to an embodiment may move a first virtual object to a location closer to a reference point than a real object and / or a second virtual object. Operation (970) performed by the processor may include the operations described above with reference to FIGS. 5A to 5E. For example, based on operation (970), the processor may move the first virtual object to a location within the screen and / or virtual space that is not obscured by another virtual object and / or a real object. Based on the movement of the first virtual object in operation (970), the processor may display the first virtual object within the screen. Based on receiving an input in operation (960), the processor may stop displaying the visual object in operation (950) and display the first virtual object moved by operation (970) within the screen.

[0134] FIG. 10 is a flowchart illustrating operations of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1 and 2 may perform the operations of the wearable device described with reference to FIG. 10. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIG. 10. The operations of the wearable device of FIG. 10 may be related to the operations of the wearable device of FIG. 9.

[0135] Referring to FIG. 10, in operation (1010), a processor of a wearable device according to one embodiment may identify a portion of a virtual space to be displayed via a display based on a reference point and a FoV within the virtual space. The processor may perform operation (1010) of FIG. 10 similarly to operation (910) of FIG. 9. The processor may identify a portion of the virtual space of operation (1010) based on a position and / or orientation of the wearable device within an external space identified via a sensor (e.g., sensor (240) of FIG. 2).

[0136] Referring to FIG. 10, in operation (1020), a processor of a wearable device according to an embodiment may identify, among virtual objects included in a portion of an identified virtual space, a second virtual object that is obscured by a first virtual object within a display. The display of operation (1020) may include the display (230) of FIG. 2. For example, the processor may identify, within the virtual space, a second virtual object positioned between a reference point of operation (1010) and the first virtual object.

[0137] Referring to FIG. 10, in operation 1030, according to one embodiment, a processor of a wearable device may display a visual object corresponding to a second virtual object based on an area of ​​a second virtual object covered by a first virtual object and / or a period of time during which the second virtual object is covered by the first virtual object within the display. If a ratio of an area of ​​a portion of the second virtual object covered by the first virtual object and an area of ​​another portion of the second virtual object visible through the screen falls within a specified ratio range for displaying the visual object, the processor may display a visual object corresponding to the second virtual object. If a period of time during which the second virtual object is covered by the first virtual object exceeds a specified period of time, the processor may display a visual object corresponding to the second virtual object.

[0138] In one embodiment, the processor may perform operation (1030) of FIG. 10 similarly to operation (950) of FIG. 9. For example, the processor may display a visual object corresponding to a second virtual object obscured by the first virtual object based on a location of the first virtual object of operation (1030) within the virtual space. The processor may display the visual object of operation (1030) in an area that does not obscure one or more virtual objects within the screen and / or external objects visible through the screen.

[0139] Referring to FIG. 10, in operation (1040), according to an embodiment, a processor of a wearable device may move a second virtual object to a portion of a virtual space between a reference point and a first virtual object based on an input related to a visual object. The processor may perform operation (1040) of FIG. 10 similarly to operations (960, 970) of FIG. 9. For example, the processor that moved the second virtual object based on operation (1040) may display the second virtual object, which was covered by the first virtual object, as an overlay on the first virtual object within the screen, or may move the second virtual object to another area within the screen that is different from the area occupied by the first virtual object within the screen. The embodiment is not limited thereto, and the processor may enhance the visibility of the second virtual object based on the movement of the first virtual object.

[0140] FIG. 11 is a flowchart illustrating operations of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1 and 2 may perform the operations of the wearable device described with reference to FIG. 11. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIG. 11. The operations of the wearable device of FIG. 11 may be related to the operations of the wearable device of FIGS. 9 and 10.

[0141] Referring to FIG. 11, in operation (1110), a processor of a wearable device according to one embodiment may identify a first portion of a virtual space to be displayed via a display within a virtual space. The first portion may be formed around a reference point (e.g., a viewpoint) within the virtual space corresponding to a location within the external space of the wearable device. The first portion of operation (1110) may be referred to as a first FoV.

[0142] Referring to FIG. 11, in operation (1120), a processor of a wearable device according to one embodiment may display a visual object corresponding to a virtual object included in a second portion of a virtual space different from a first portion within a display (e.g., display (230) of FIG. 2). The processor may display a visual object corresponding to a virtual object included in a second portion different from the first portion, while displaying a screen corresponding to the first portion identified by operation (1110) on the display. The visual object displayed by operation (1120) may include the visual object (322) of FIG. 3C and the visual object (562) of FIGS. 5A to 5E.

[0143] Referring to FIG. 11, within operation (1130), a processor of a wearable device according to one embodiment may identify an input indicating selection of a visual object. The processor may perform operation (1130) of FIG. 11 similarly to operation (960) of FIG. 9. For example, the processor may identify the input of operation (1130) using a hand gesture, gaze, speech, and / or an external electronic device connected to the wearable device (e.g., external electronic device (252) of FIG. 2) of a user wearing the wearable device, as described above with reference to FIGS. 4A and 4B. If the processor fails to identify the input of operation (1130) or prior to identifying the input of operation (1130) (1130-No), the processor may continue to display the visual object of operation (1120). In response to identifying the input of action (1130), the processor can perform action (1140).

[0144] Referring to FIG. 11, in operation (1140), a processor of a wearable device according to an embodiment may move a virtual object from a second portion to a first portion. The processor may perform operation (1140) of FIG. 11 similarly to operation (970) of FIG. 9. The processor may move the virtual object into the first portion based on the operations described above with reference to FIGS. 5A to 5E. For example, the processor may move the virtual object toward a reference point (or viewpoint) of the first portion. Referring to FIG. 11, in operation (1150), a processor of a wearable device according to an embodiment may display the virtual object moved into the first portion within a display. The wearable device may display the virtual object moved into the first portion corresponding to the display through the display based on operation (1140).

[0145] As described above, according to one embodiment, a wearable device can adjust the visibility of virtual objects based on the positional relationship of virtual objects within a virtual space. The wearable device can improve or enhance the visibility of the virtual object using a visual object corresponding to the virtual object. The wearable device can compare an area within the virtual space occupied by a virtual object and an area within the virtual space corresponding to an external object within the external space using a virtual space mapped with an external space, thereby identifying whether the virtual object is occluded by the external object visible through the display. The wearable device can move a virtual object occluded by another virtual object within the display and / or an external object visible through the display within the virtual space using a visual object corresponding to the virtual object.

[0146] Hereinafter, an example of a form factor of a wearable device (101) according to one embodiment is described using FIGS. 12a to 12b and / or FIGS. 13a to 13b.

[0147] FIG. 12A illustrates an example of a perspective view of a wearable device according to one embodiment. FIG. 12B illustrates an example of one or more hardware elements disposed within a wearable device (1200) according to one embodiment. The wearable device (1200) of FIGS. 12A and 12B may be an example of the wearable device (101) of FIGS. 1 and 2 . As shown in FIG. 12A , the wearable device (1200) according to one embodiment may include at least one display (1250) and a frame supporting the at least one display (1250).

[0148] According to one embodiment, a wearable device (1200) may be worn on a part of a user's body. The wearable device (1200) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines AR and VR to a user wearing the wearable device (1200). For example, the wearable device (1200) may output a virtual reality image to the user through at least one display (1250) in response to a user's designated gesture acquired through the motion recognition camera (1240-2) of FIG. 12B.

[0149] According to one embodiment, at least one display (1250) within a wearable device (1200) may provide visual information to a user. The at least one display (1250) may include the display (230) of FIG. 2. For example, the at least one display (1250) may include a transparent or translucent lens. The at least one display (1250) may include a first display (1250-1) and / or a second display (1250-2) spaced apart from the first display (1250-1). For example, the first display (1250-1) and the second display (1250-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0150] Referring to FIG. 12B, at least one display (1250) may form a display area on a lens to provide a user wearing the wearable device (1200) with visual information contained in external light passing through the lens, along with other visual information distinct from the visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. The display area formed by at least one display (1250) may be formed on the second surface (1232) among the first surface (1231) and the second surface (1232) of the lens. When the user wears the wearable device (1200), external light may be incident on the first surface (1231) and transmitted through the second surface (1232) to the user. As another example, at least one display (1250) may display a virtual reality image to be combined with a real screen transmitted via external light. The virtual reality image output from the at least one display (1250) may be transmitted to the user's eyes via one or more hardware included in the wearable device (1200) (e.g., optical devices (1282, 1284) and / or at least one waveguide (1233, 1234)).

[0151] According to one embodiment, a wearable device (1200) may include waveguides (1233, 1234) that diffract light transmitted from at least one display (1250) and relayed by optical devices (1282, 1284) and transmit the diffracted light to a user. The waveguides (1233, 1234) may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of the waveguides (1233, 1234). The nanopatterns may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the waveguides (1233, 1234) may be propagated to the other end of the waveguides (1233, 1234) by the nanopatterns. The waveguides (1233, 1234) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, the waveguides (1233, 1234) may be arranged within the wearable device (1200) to guide a screen displayed by at least one display (1250) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the waveguides (1233, 1234).

[0152] According to one embodiment, a wearable device (1200) may analyze an object included in a real image collected through a capturing camera (1240-1), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1250). The virtual object may include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (1200) may analyze the object based on a multi-camera, such as a stereo camera. For the object analysis, the wearable device (1200) may execute time-of-flight (ToF) and / or simultaneous localization and mapping (SLAM) supported by the multi-camera. A user wearing the wearable device (1200) may view an image displayed on at least one display (1250).

[0153] According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (1200) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (1200), the first display (1250-1) and the second display (1250-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (1250). For example, the frame may support the first display (1250-1) and the second display (1250-2) to be positioned corresponding to the user's left and right eyes.

[0154] Referring to FIG. 12A, the frame may include a region (1220) that at least partially contacts a portion of the user's body when the user wears the wearable device (1200). For example, the region (1220) of the frame that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1200) makes contact with. According to one embodiment, the frame may include a nose pad (1210) that contacts a portion of the user's body. When the wearable device (1200) is worn by the user, the nose pad (1210) may contact a portion of the user's nose. The frame may include a first temple (1204) and a second temple (1205) that contact another portion of the user's body that is distinct from the portion of the user's body.

[0155] In one embodiment, the frame may include a first rim (1201) surrounding at least a portion of a first display (1250-1), a second rim (1202) surrounding at least a portion of a second display (1250-2), a bridge (1203) disposed between the first rim (1201) and the second rim (1202), a first pad (1211) disposed along a portion of an edge of the first rim (1201) from one end of the bridge (1203), a second pad (1212) disposed along a portion of an edge of the second rim (1202) from the other end of the bridge (1203), a first temple (1204) extending from the first rim (1201) and secured to a portion of an ear of the wearer, and a second temple (1205) extending from the second rim (1202) and secured to a portion of an ear opposite the ear. The first pad (1211) and the second pad (1212) may be in contact with a portion of the user's nose, and the first temple (1204) and the second temple (1205) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (1204, 1205) may be rotatably connected to the rim through the hinge units (1206, 1207) of FIG. 12B. The first temple (1204) may be rotatably connected to the first rim (1201) through the first hinge unit (1206) disposed between the first rim (1201) and the first temple (1204). The second temple (1205) can be rotatably connected to the second rim (1202) via a second hinge unit (1207) disposed between the second rim (1202) and the second temple (1205). In one embodiment, the wearable device (1200) can identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame.

[0156] According to one embodiment, the wearable device (1200) may include hardwares that perform various functions (e.g., hardwares described above based on the block diagram of FIG. 2). For example, the hardwares may include a battery module (1270), an antenna module (1275), optical devices (1282, 1284), speakers (1292-1, 1292-2), microphones (1294-1, 1294-2, 1294-3), a light-emitting module (not shown), and / or a printed circuit board (1290). The various hardwares may be arranged within a frame.

[0157] According to one embodiment, microphones (1294-1, 1294-2, 1294-3) of a wearable device (1200) may be arranged on at least a portion of a frame to acquire sound signals. A first microphone (1294-1) arranged on a nose pad (1210), a second microphone (1294-2) arranged on a second rim (1202), and a third microphone (1294-3) arranged on a first rim (1201) are illustrated in FIG. 12B , but the number and arrangement of the microphones (1294) are not limited to the embodiment of FIG. 12B . When the number of microphones (1294) included in the wearable device (1200) is two or more, the wearable device (1200) may identify the direction of a sound signal by using a plurality of microphones arranged on different portions of the frame.

[0158] According to one embodiment, the optical devices (1282, 1284) can transmit a virtual object transmitted from at least one display (1250) to the wave guides (1233, 1234). For example, the optical devices (1282, 1284) can be projectors. The optical devices (1282, 1284) can be positioned adjacent to at least one display (1250) or can be included within at least one display (1250) as a part of the at least one display (1250). The first optical device (1282) can correspond to the first display (1250-1), and the second optical device (1284) can correspond to the second display (1250-2). The first optical device (1282) can transmit light output from the first display (1250-1) to the first waveguide (1233), and the second optical device (1284) can transmit light output from the second display (1250-2) to the second waveguide (1234).

[0159] In one embodiment, the camera (1240) may include an eye tracking camera (ET CAM) (1240-1), a motion recognition camera (1240-2), and / or a recording camera (1240-3). The recording camera (1240-3), the eye tracking camera (1240-1), and the motion recognition camera (1240-2) may be positioned at different locations on the frame and may perform different functions. The recording camera (1240-3), the eye tracking camera (1240-1), and the motion recognition camera (1240-2) may be referred to as image sensors. The eye tracking camera (1240-1) may output data representing the gaze of a user wearing the wearable device (1200). For example, the wearable device (1200) can detect the gaze from an image including the user's pupils obtained through the gaze tracking camera (1240-1). An example in which the gaze tracking camera (1240-1) is positioned toward the user's right eye is illustrated in FIG. 12B, but the embodiment is not limited thereto, and the gaze tracking camera (1240-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0160] In one embodiment, the capturing camera (1240-3) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1250). The at least one display (1250) can display a single image in which information about the actual image or background including the image of the specific object acquired using the capturing camera is superimposed on a virtual image provided through optical devices (1282, 1284). In one embodiment, the capturing camera can be placed on a bridge (1203) disposed between the first rim (1201) and the second rim (1202).

[0161] In one embodiment, the gaze tracking camera (1240-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (1200) and thereby aligning the user's gaze with visual information provided to at least one display (1250). For example, when the wearable device (1200) looks straight ahead, the wearable device (1200) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1250). The gaze tracking camera (1240-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1240-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1240-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1240-1) may be positioned within the first rim (1201) and / or the second rim (1202) to face the direction in which the user wearing the wearable device (1200) is positioned.

[0162] In one embodiment, the gesture recognition camera (1240-2) can provide a specific event on a screen provided on at least one display (1250) by recognizing the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the user's body. The gesture recognition camera (1240-2) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1250). The processor can identify the signal corresponding to the gesture and perform a designated function based on the identification. In one embodiment, the gesture recognition camera (1240-2) can be disposed on the first rim (1201) and / or the second rim (1202).

[0163] In one embodiment, the camera (1240) included in the wearable device (1200) is not limited to the gaze tracking camera (1240-1) and the motion recognition camera (1240-2) described above. For example, the wearable device (1200) can identify an external object included in the FoV using the photographing camera (1240-3) positioned toward the user's FoV. The wearable device (1200) can identify an external object based on a sensor for identifying the distance between the wearable device (1200) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1240) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1200) may include a camera (1240) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1200).

[0164] Although not shown, in one embodiment, the wearable device (1200) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (1240). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (1206, 1207).

[0165] According to one embodiment, the battery module (1270) may supply power to the electronic components of the wearable device (1200). In one embodiment, the battery module (1270) may be disposed within the first temple (1204) and / or the second temple (1205). For example, the battery module (1270) may be a plurality of battery modules (1270). The plurality of battery modules (1270) may be disposed within each of the first temple (1204) and the second temple (1205). In one embodiment, the battery module (1270) may be disposed at an end of the first temple (1204) and / or the second temple (1205).

[0166] In one embodiment, the antenna module (1275) can transmit signals or power to, or receive signals or power from, the outside of the wearable device (1200). The antenna module (1275) can be electrically and / or operatively connected to communication circuitry within the wearable device (1200) (e.g., communication circuitry (250) of FIG. 2 ). In one embodiment, the antenna module (1275) can be positioned within the first temple (1204) and / or the second temple (1205). For example, the antenna module (1275) can be positioned proximate one surface of the first temple (1204) and / or the second temple (1205).

[0167] In one embodiment, the speakers (1292-1, 1292-2) may output audio signals to the outside of the wearable device (1200). The audio output module may be referred to as a speaker. In one embodiment, the speakers (1292-1, 1292-2) may be positioned within the first temple (1204) and / or the second temple (1205) so as to be positioned adjacent to the ears of a user wearing the wearable device (1200). For example, the wearable device (1200) may include a second speaker (1292-2) positioned within the first temple (1204) and thus adjacent to the user's left ear, and a first speaker (1292-1) positioned within the second temple (1205) and thus adjacent to the user's right ear.

[0168] In one embodiment, the light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide a user with information regarding a specific state of the wearable device (1200). For example, when the wearable device (1200) requires charging, the wearable device (1200) may repeatedly emit red light at a designated time. In one embodiment, the light-emitting module may be disposed on the first rim (1201) and / or the second rim (1202).

[0169] Referring to FIG. 12B, according to one embodiment, a wearable device (1200) may include a printed circuit board (PCB) (1290). The PCB (1290) may be included in at least one of the first temple (1204) or the second temple (1205). The PCB (1290) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (1200) (e.g., hardwares illustrated by the blocks described above with reference to FIG. 2) may be disposed on the PCB (1290). The wearable device (1200) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0170] According to one embodiment, a wearable device (1200) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1200) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1200). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1200) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (1200) based on the IMU.

[0171] Figures 13a and 13b illustrate an example of an exterior appearance of a wearable device (1300) according to one embodiment. The wearable device (1300) of Figures 13a and 13b may be an example of the wearable device (101) of Figures 1 and 2. An example of an exterior appearance of a first side (1310) of a housing of the wearable device (1300) according to one embodiment is illustrated in Figure 13a, and an example of an exterior appearance of a second side (1320) opposite to the first side (1310) may be illustrated in Figure 13b.

[0172] Referring to FIG. 13A, a first surface (1310) of a wearable device (1300) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1300) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1204) and / or the second temple (1205) of FIGS. 12A and 12B). A first display (1350-1) for outputting an image to a left eye among the user's two eyes, and a second display (1350-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1310). The wearable device (1300) may be formed on the first surface (1310) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (1350-1) and the second display (1350-2).

[0173] According to one embodiment, a wearable device (1300) may include cameras (1340-3, 1340-4) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1350-1) and the second display (1350-2). The cameras (1340-3, 1340-4) may be referred to as ET cameras. According to one embodiment, a wearable device (1300) may include cameras (1340-1, 1340-2) for photographing and / or recognizing a user's face. The cameras (1340-1, 1340-2) may be referred to as FT cameras.

[0174] Referring to FIG. 13B, a camera (e.g., cameras (1340-5, 1340-6, 1340-7, 1340-8, 1340-9, 1340-10)) and / or a sensor (e.g., a depth sensor (1330)) for obtaining information related to the external environment of the wearable device (1300) may be disposed on a second surface (1320) opposite to the first surface (1310) of FIG. 13A. For example, the cameras (1340-5, 1340-6, 1340-7, 1340-8, 1340-9, 1340-10) may be disposed on the second surface (1320) to recognize external objects different from the wearable device (1300). For example, using cameras (1340-9, 1340-10), the wearable device (1300) can acquire images and / or media to be transmitted to each of the user's eyes. The camera (1340-9) can be positioned on the second face (1320) of the wearable device (1300) to acquire a frame to be displayed through a second display (1350-2) corresponding to the right eye among the two eyes. The camera (1340-10) can be positioned on the second face (1320) of the wearable device (1300) to acquire a frame to be displayed through a first display (1350-1) corresponding to the left eye among the two eyes.

[0175] According to one embodiment, the wearable device (1300) may include a depth sensor (1330) disposed on the second surface (1320) to identify a distance between the wearable device (1300) and an external object. Using the depth sensor (1330), the wearable device (1300) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1300).

[0176] Although not shown, a microphone may be placed on the second side (1320) of the wearable device (1300) to acquire sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0177] As described above, according to one embodiment, the wearable device (1300) may have a form factor for being worn on a user's head. The wearable device (1300) may provide a user experience based on augmented reality and / or mixed reality while being worn on the head. Using the first display (1350-1) and the second display (1350-2), the wearable device (1300) may display a UI including an external electronic device (e.g., the external electronic device (252) of FIG. 2). Through the UI, a user wearing the wearable device (1300) may view media content displayed by the wearable device (1300) together with the external electronic device. The wearable device (1300) may execute a function for moving the media content to the external electronic device based on a deformation of the external electronic device.

[0178] In one embodiment, a method for controlling the visibility of a virtual object that is not visible through a screen of a wearable device may be required. As described above, according to one embodiment, a wearable device (e.g., the wearable device (101) of FIGS. 1 to 2) may include a display (e.g., the display (230) of FIG. 2) and a processor (e.g., the processor (210) of FIG. 2). The processor may be configured to display, on the display, a first virtual object included in a field-of-view (FoV) formed based on a reference point within a virtual space (e.g., the virtual space (120) of FIGS. 1, 4A to 4B, 5A to 5E, and 6, and / or the virtual space (710) of FIGS. 7, 8A to 8C). The processor may be configured to display a visual object (e.g., a visual object (150) of FIG. 1 , a visual object (322) of FIGS. 3B to 3D , a visual object (312) of FIG. 3E , a visual object (422) of FIG. 4A , visual objects (532, 552, 562) of FIGS. 5A to 5E , visual objects (634, 654, 664) of FIG. 6 , and / or a visual object (832) of FIG. 8B ) related to the second virtual object, based on a location of the first virtual object within the display, based on an input indicating a selection of the visual object. The processor may be configured to display the second virtual object, which has been moved to a portion of the virtual space formed between the reference point and the first virtual object, based on an input indicating a selection of the visual object.

[0179] For example, the processor may be configured to display the visual object based on identifying the second virtual object included in the area of ​​the display where the first virtual object is displayed.

[0180] For example, the processor may be configured to display the first virtual object in a different area of ​​the display than the area based on displaying the second virtual object based on the input.

[0181] For example, the processor may be configured to display the visual object based on whether the period of time for which the second virtual object is included in the area exceeds a specified period of time.

[0182] For example, the processor may be configured to display the visual object based on a ratio between a first area in which the first virtual object is displayed within the display and a second area in which the second virtual object, which is at least partially obscured by the first virtual object, is displayed within the display.

[0183] For example, the wearable device may further include a sensor for measuring motion of the wearable device (e.g., sensor (240) of FIG. 2). The processor may be configured to change the FoV within the virtual space based on the motion measured by the sensor. The processor may be configured to display another visual object related to the third virtual object on the display based on identifying a third virtual object moved outside the FoV by the change in the FoV.

[0184] For example, the processor may be configured to identify, among a plurality of virtual objects positioned between the reference point and the second virtual object, the first virtual object having a minimum distance from the reference point. The processor may be configured to display the visual object positioned in the virtual space based on the minimum distance from the first virtual object.

[0185] For example, the processor may be configured to display the second virtual object moved along the portion in a direction opposite to the direction, and to move the first virtual object along the direction based on a size of the second virtual object within the virtual space.

[0186] According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of displaying a first virtual object included in a field-of-view (FoV) formed based on a reference point in a virtual space on a display of the wearable device. The method may include an operation of displaying a visual object related to a second virtual object in the virtual space, based on identifying a second virtual object positioned behind the first virtual object in a direction from the reference point toward the first virtual object, based on a position of the first virtual object in the display. The method may include an operation of displaying the second virtual object, which has been moved to a portion of the virtual space between the reference point and the first virtual object, based on an input indicating a selection of the visual object.

[0187] For example, the act of displaying the visual object may include an act of displaying the visual object based on identifying the second virtual object included in the area of ​​the display where the first virtual object is displayed.

[0188] For example, the action of displaying the second virtual object may include an action of displaying the first virtual object in a different area of ​​the display than the area based on displaying the second virtual object based on the input.

[0189] For example, the act of displaying the visual object may include an act of displaying the visual object based on whether the period of time during which the second virtual object is included in the area exceeds a specified period of time.

[0190] For example, the act of displaying the visual object may include an act of displaying the visual object based on a ratio between a first area in which the first virtual object is displayed within the display and a second area in which the second virtual object, which is at least partially obscured by the first virtual object, is displayed within the display.

[0191] For example, the method may include an operation of changing the FoV within the virtual space based on motion of the wearable device measured using a sensor of the wearable device. The method may include an operation of displaying another visual object related to the third virtual object on the display based on identifying a third virtual object that has moved outside the FoV due to the change in the FoV.

[0192] For example, the operation of displaying the visual object may include an operation of identifying, among a plurality of virtual objects positioned between the reference point and the second virtual object, the first virtual object having the minimum distance from the reference point. The method may include an operation of displaying the visual object positioned in the virtual space based on the minimum distance of the first virtual object.

[0193] For example, the action of displaying the second virtual object may include an action of moving the first virtual object along the direction based on the size of the second virtual object moved to the part along another direction opposite to the direction within the virtual space.

[0194] According to one embodiment, a wearable device as described above may include a camera, a display, and a processor. The processor may be configured to identify an external object visible through the display using the camera. The processor may be configured to display a visual object related to the virtual object within the display based on identifying a virtual object located behind a second location within the virtual space corresponding to the external object, in a direction from a first location within the virtual space corresponding to the camera to a second location within the virtual space corresponding to the external object, within a virtual space mapped with the external space. The processor may be configured to display the virtual object moved to a portion of the virtual space defined between the first location and the second location, based on an input indicating selection of the visual object.

[0195] For example, the processor may be configured to display the visual object based on the second location within the virtual space.

[0196] For example, the processor may be configured to display the visual object, which includes an image representing the appearance of the virtual object based on a two-dimensional rendering of the virtual object.

[0197] For example, the processor may be configured to identify whether to display the visual object based on a period of time during which the virtual object is positioned behind the second location along the direction.

[0198] For example, the processor may be configured to identify whether to display the visual object based on a ratio between a first area in which the external object is visible through the display and a second area in which the virtual object, which is at least partially obscured by the first area, is displayed within the display.

[0199] For example, the processor may be configured to move the virtual object from the second location, based on the input, to a third location along a different direction opposite to the direction, based on a size of the virtual object.

[0200] According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of identifying an external object visible through a display of the wearable device using a camera of the wearable device. The method may include an operation of displaying a visual object related to the virtual object within the display based on identifying a virtual object located behind a second location within the virtual space corresponding to the external object in a direction from a first location within the virtual space corresponding to the camera to a second location within the virtual space corresponding to the external object, within a virtual space mapped with the external space. The method may include an operation of displaying the virtual object moved to a portion of the virtual space formed between the first location and the second location, based on an input indicating a selection of the visual object.

[0201] For example, the action of displaying the visual object may include an action of displaying the visual object based on the second location within the virtual space.

[0202] For example, the act of displaying the visual object may include an act of displaying the visual object including an image representing the appearance of the virtual object based on a two-dimensional rendering of the virtual object.

[0203] For example, the identifying action may include an action of identifying whether to display the visual object based on a period of time during which the virtual object is positioned behind the second location along the direction.

[0204] For example, the identifying action may include identifying whether to display the visual object based on a ratio between a first area in which the external object is visible through the display and a second area in which the virtual object, which is at least partially obscured by the first area, is displayed within the display.

[0205] For example, the action of displaying the virtual object may include an action of moving the virtual object from the second location to a third location along a different direction opposite to the direction, based on the size of the virtual object, based on the input.

[0206] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0207] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0208] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0209] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0210] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a wearable device (101), Display (230); A memory (220) comprising one or more storage media storing instructions; and At least one processor (210) including a processing circuit, wherein the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Displaying a first virtual object included in a field-of-view (FoV) formed based on a reference point in a virtual space (120; 710) on the display; Based on identifying a second virtual object in the virtual space, positioned behind the first virtual object along a direction from the reference point to the first virtual object, displaying a visual object (150; 312; 322; 422; 532, 552, 562; 634, 654, 664; 832) related to the second virtual object according to the position of the first virtual object in the display; and Causing to display the second virtual object, moved to a part of the virtual space formed between the reference point and the first virtual object, based on an input indicating selection of the visual object. Wearable devices.

2. In claim 1, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Causing to display the visual object based on identifying the second virtual object included in the area of ​​the display where the first virtual object is displayed. Wearable devices.

3. In claims 1 and 2, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Based on the display of the second virtual object based on the input, causing the first virtual object to be displayed in another area of ​​the display different from the area, Wearable devices.

4. In claims 1 to 3, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Causing the visual object to be displayed based on whether the period of time included in the area of ​​the second virtual object exceeds a specified period of time; Wearable devices.

5. In claims 1 to 4, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Causing the visual object to be displayed based on a ratio between a first area where the first virtual object is displayed within the display and a second area where the second virtual object, which is at least partially obscured by the first virtual object, is displayed within the display. Wearable devices.

6. In claims 1 to 5, Further comprising a sensor (240) for measuring motion of the wearable device; The above instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Based on the motion measured by the sensor, changing the FoV within the virtual space; Based on identifying a third virtual object moved out of the FoV by a change in the FoV, causing the display to display another visual object related to the third virtual object. Wearable devices.

7. In claims 1 to 6, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Among a plurality of virtual objects positioned between the reference point and the second virtual object, identifying the first virtual object having the minimum distance from the reference point; Causing to display the visual object placed in the virtual space based on the minimum distance of the first virtual object; Wearable devices.

8. In claims 1 to 7, the instructions, when individually and / or collectively executed by the at least one processor, cause the wearable device to: Displaying the second virtual object moved to the part along a different direction opposite to the said direction, and causing the first virtual object to move along the direction based on the size of the second virtual object within the virtual space. Wearable devices.

9. In the method of a wearable device, An action of displaying a first virtual object included in a field-of-view (FoV) formed based on a reference point in a virtual space on a display of the wearable device; An operation of displaying a visual object related to the second virtual object based on identifying a second virtual object positioned behind the first virtual object in the virtual space along a direction from the reference point toward the first virtual object, according to the position of the first virtual object within the display; and An operation of displaying a second virtual object, the second virtual object being moved to a portion of the virtual space between the reference point and the first virtual object, based on an input indicating a selection of the visual object. method.

10. In claim 9, the action of displaying the visual object comprises: An operation of displaying the visual object based on identifying the second virtual object included in the area of ​​the display in which the first virtual object is displayed, method.

11. In claims 9 to 10, the operation of displaying the second virtual object comprises: Based on the input, displaying the second virtual object, comprising an action of displaying the first virtual object in another area of ​​the display different from the area. method.

12. In claims 9 to 11, the action of displaying the visual object comprises: Including an action of displaying the visual object based on whether the period of time included in the area of ​​the second virtual object exceeds a specified period of time; method.

13. In claims 9 to 12, the action of displaying the visual object comprises: An operation of displaying the visual object based on a ratio between a first area in which the first virtual object is displayed within the display and a second area in which the second virtual object, which is at least partially obscured by the first virtual object, is displayed within the display. method.

14. In claims 9 to 13, An operation of changing the FoV within the virtual space based on the motion of the wearable device measured using a sensor of the wearable device; and Further comprising an action of displaying another visual object related to the third virtual object on the display based on identifying a third virtual object moved out of the FoV by a change in the FoV. method.

15. In claims 9 to 14, the action of displaying the visual object comprises: An operation of identifying the first virtual object having the minimum distance from the reference point among a plurality of virtual objects positioned between the reference point and the second virtual object; and Including an action of displaying the visual object placed in the virtual space based on the minimum distance of the first virtual object. method.

Citation Information

Patent Citations

  • Apparatus and Method for Providing Augmented Reality Information of invisible Reality Object

    KR1020120015922A

  • Pass-through camera user interface elements for virtual reality

    KR1020180102171A

  • Method and device for a virtual reality interface that provides fusion with real space

    KR102056681B1

  • Head-mounted information processing apparatus and head-mounted display system

    US20220172439A1

  • KR20220165840A