Wearable device for arranging virtual screen and control method therefor

Abstract

This wearable device comprises a communication circuit, a first camera display memory, and at least one processor, wherein the processor: displays, on a display, a first virtual window for augmented reality (AR); detects a user device on the basis of an image acquired from a first camera; in response to the detection of the user device, identifies whether the user device overlaps the first virtual window on the basis of the position of the user device and the position of the first virtual window; if it is identified that the user device overlaps the first virtual window, requests interworking information from the user device through the communication circuit; receives the interworking information from the user device through the communication circuit in response to the request; displays content corresponding to the interworking information in the first virtual window on the basis of the overlap state being a first state; displays a second virtual window in the vicinity of the first virtual window on the basis of the overlap state being a second state; and displays the content corresponding to the interworking information in the second virtual window.

Description

Wearable device for arranging a virtual screen and its control method

[0001] The present disclosure relates to a wearable device and a control method thereof, and more particularly, to a wearable device that arranges a virtual screen and a control method thereof.

[0002] Augmented Reality (AR) is a computer graphics technology that superimposes virtual objects or information onto real-world spaces, making them appear as if they were real objects. Wearable devices can provide users with an AR environment using video see-through (VST) or smart lenses. Wearable devices may include AR glasses and head-mounted devices (HMDs).

[0003] According to one embodiment of the present disclosure, a wearable device may include at least one processor including a communication circuit, a first camera, a display, a memory for storing instructions, and processing circuitry. The instructions, when individually or collectively executed by the at least one processor, cause the wearable device to display a first virtual window on the display in a state in which a user wearing the wearable device (100) can view the real world around the user, detect a user device in the real world based on an image acquired from the first camera, and in response to the detection of the user device, identify whether the user device overlaps the first virtual window in a field of view of the user based on a location of the user device and a location of the first virtual window, and if the user device is identified as being overlapped with the first virtual window, request linkage information to the user device through the communication circuit, and in response to the request, receive the linkage information from the user device through the communication circuit, and display content corresponding to the linkage information on the first virtual window based on the overlap state being a first state, and based on the overlap state being a second state, display content corresponding to the linkage information on the first virtual window. A second virtual window can be displayed in the vicinity, and content corresponding to the linkage information can be displayed in the second virtual window.

[0004] According to one embodiment of the present disclosure, a control method of a wearable device including a first camera, a display, and a communication circuit, the control method comprises: an operation of displaying a first virtual window on the display in a state in which a user wearing the wearable device (100) can view the real world around the user; an operation of detecting a user device in the real world based on an image acquired from the first camera; an operation of identifying whether the user device overlaps the first virtual window in a field of view of the user based on a location of the user device and a location of the first virtual window in response to the detection of the user device; an operation of requesting linkage information to the user device through the communication circuit and receiving the linkage information from the user device through the communication circuit in response to the request; an operation of displaying content corresponding to the linkage information on the first virtual window based on the overlap state being a first state; and an operation of displaying a second virtual window around the first virtual window based on the overlap state being a second state, and receiving information corresponding to the linkage information. It may include an action of displaying corresponding content in the second virtual window.

[0005] FIG. 1 is a drawing illustrating an operation of a wearable device displaying a screen according to an embodiment of the present disclosure.

[0006] FIG. 2 is an exemplary block diagram of a user device according to one embodiment.

[0007] FIG. 3 is a drawing for explaining the configuration of a wearable device according to one embodiment.

[0008] FIG. 4 is an exemplary block diagram of a wearable device according to one embodiment.

[0009] FIG. 5 is a drawing for explaining the operation of a wearable device according to one embodiment.

[0010] FIG. 6A and FIG. 6B are diagrams for explaining an overlap state of a user device and a virtual window according to one embodiment.

[0011] FIG. 7 is a flowchart illustrating an operation for activating a screen expansion mode according to one embodiment.

[0012] FIG. 8 is a diagram for explaining an operation of activating a screen expansion mode according to one embodiment.

[0013] FIG. 9 is a flowchart illustrating a method for expanding a screen based on an overlap state according to one embodiment.

[0014] FIGS. 10A to 10C are drawings for explaining the arrangement of a second virtual window based on the overlap state being the second state according to one embodiment.

[0015] FIG. 11 is a drawing for explaining the arrangement of virtual windows based on the overlap state being the first state according to one embodiment.

[0016] FIG. 12 is a flowchart illustrating a method for displaying content corresponding to linkage information in a virtual window based on linkage information according to one embodiment.

[0017] FIGS. 13 and 14 are diagrams for explaining an operation of displaying content in a virtual window based on an overlap state according to one embodiment.

[0018] FIG. 15 is a drawing for explaining a screen layout method in a multi-screen situation according to one embodiment.

[0019] FIG. 16 is a drawing for explaining an example of adjusting the size of a virtual window according to one embodiment.

[0020] FIG. 17 is a diagram illustrating a method for displaying content running on a user device when there is no virtual screen, according to one embodiment.

[0021] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on their meanings and the overall content of this disclosure.

[0022] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0023] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0024] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0025] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0026] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0027] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0028] In this specification, the term user may refer to a person using a user device or a device using a user device (e.g., an artificial intelligence user device).

[0029] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0030] FIG. 1 is a drawing explaining an operation of a wearable device (100) displaying a screen according to an embodiment of the present disclosure.

[0031] According to an embodiment, the wearable device (100) may include a display that can be worn on the user's head (e.g., a head-mounted display (HMD)).

[0032] In Fig. 1, the wearable device (100) is illustrated in the form of glasses, but of course, it is not limited thereto.

[0033] According to an embodiment, the wearable device (100) can execute functions related to video see-through (VST), see-through, and virtual reality (VR).

[0034] As illustrated in FIG. 1, the wearable device (100) may include a housing that covers the eyes of a user wearing the wearable device (100). According to an embodiment, the wearable device (100) may include a display positioned on a first side of the housing facing the user's eyes, and a camera positioned on a second side opposite the first side.

[0035] According to an embodiment, the camera acquires an image including ambient light of the wearable device (100), and the display displays the image so that the user can recognize the surroundings (or front) of the wearable device (100) even while wearing the wearable device (100).

[0036] According to an embodiment, the wearable device (100) can synthesize a virtual object into an image displayed through a display, thereby allowing the user to recognize real objects and virtual objects located around (or in front of) the wearable device (100).

[0037] According to an embodiment, the wearable device (100) may perform functions related to augmented reality (AR) and mixed reality (MR). For example, the wearable device (100) may include at least one lens positioned adjacent to the user's eyes. The wearable device (100) may combine ambient light passing through the at least one lens with light emitted from the display of the wearable device (100). For example, the wearable device (100) may provide a user with a mixture of a real object recognizable by the ambient light passing through the at least one lens and a virtual object emitted by the display and formed within the at least one lens.

[0038] Referring to FIG. 1, a wearable device (100) according to an embodiment can display a screen with a sense of perspective by utilizing binocular disparity. According to an embodiment, the screen can include an image acquired through a camera and a virtual object corresponding to an application.

[0039] According to an embodiment, the wearable device (100) can execute an application, and the virtual object can include an execution screen of the application.

[0040] For example, the virtual object may include a window (e.g., an activity) and / or a widget (or gadget) provided by a program (e.g., a software application) executed by the wearable device (100). The wearable device (100) may use binocular disparity to display a virtual object floating in the user's field-of-view (FoV).

[0041] According to an embodiment, a wearable device (100) may display one or more visual objects on a screen. A visual object may refer to an object that can be deployed on a screen for transmitting and / or interacting with information, such as a virtual window, text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, a time picker, a progress bar, and / or a table. A visual object may be referred to as a virtual object, a visual element, a user interface (UI), and / or a UI element.

[0042] According to an embodiment, the wearable device (100) can execute an application and control the display to display a virtual object corresponding to the application.

[0043] For example, the wearable device (100) can execute a document editing processing application capable of creating, storing, or editing text documents.

[0044] The wearable device (100) can recognize the user's terminal device. The wearable device's recognition of the user's terminal device can be performed by using images and / or videos acquired from a camera included in the wearable device (100). The wearable device's recognition of the user's terminal device can include an operation of acquiring information indicating the location and / or shape of the user's terminal device from the images and / or videos.

[0045] FIG. 2 is an exemplary block diagram of a user device according to one embodiment.

[0046] The components, their relationships, and their functions depicted in FIG. 2 are exemplary only and do not limit the implementations described or claimed in this document.

[0047] The user device (200) may include components including at least one processor (205) (hereinafter, referred to as processor (205)), at least one memory (215) (hereinafter, referred to as memory (215)), at least one display (220) (hereinafter, referred to as display (220)), and at least one communication circuit (210) (hereinafter, referred to as communication circuit (210)). The above components are merely exemplary. For example, the user device (200) may include other components (e.g., power management integrated circuitry (PMIC), sensors, audio processing circuitry, antenna, rechargeable battery, or input / output interface). For example, some components may be omitted from the user device (200). For example, some components may be integrated into one component.

[0048] The processor (205) may be implemented as one or more integrated circuit (IC) (or circuitry) chips and may perform various data processing operations. The processor (205) may include at least one electrical circuit and may individually or collectively distribute and process instructions (or programs, data, etc.) stored in the memory (215). The processor (205) may include a processor assembly including one or more processing circuits.

[0049] For example, the processor (205) may include a central processing unit (CPU) (not shown), a graphics processing unit (GPU) (not shown), a neural processing unit (NPU) (not shown), an image signal processor (ISP) (not shown), a display controller (not shown), a memory controller (not shown), a storage controller (not shown), a communication processor (CP) (not shown), and / or a sensor interface (not shown). These components of the processor (205) are merely exemplary. For example, the processor (205) may further include other components.

[0050] The processor (205) may cause other components of the user device (200) to perform various operations by executing instructions stored in the memory (215). The CPU (not shown) (or central processing circuit) may be configured to control components of the processor (205) based on the execution of instructions stored in the memory (215) (e.g., volatile memory (not shown) and / or non-volatile memory (not shown)).

[0051] A communication circuit (210) according to one embodiment may include various hardware and / or software configurations for supporting wired or wireless communication with a wearable device (100). The communication circuit (210) may include a communication circuit capable of performing data communication between external user devices. The user device (200) may transmit and receive various data or control commands with the wearable device (100) via the communication circuit (210) via wired / wireless communication. In one embodiment, the communication circuit (210) may support wireless communication. The wireless communication may include, but is not limited to, at least one of Bluetooth, BLE (Bluetooth Low Energy), ZigBee, ANT+, Wi-Fi, Cellular (LTE, 5G, 6G, NB-IoT), NFC (near field communication), RFID (radio frequency identification), UWB (ultrawide band), GNSS (global navigation satellite system) and / or MST (magnetic secure transmission). According to some embodiments, the communication circuit (210) may be implemented in an integrated form with the processor (205).

[0052] The memory (215) may include one or more storage media (or one or more storage devices). For example, the memory (215) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (not shown)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (not shown)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof.

[0053] For example, the memory (215) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (205). For example, the memory (215) may store instructions callable by an application programming interface (API). For example, the memory (215) may store instructions within a library.

[0054] The display (220) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, the display (220) may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (220) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

[0055] FIG. 3 is a drawing for explaining the configuration of a wearable device (100) according to one embodiment.

[0056] FIG. 3 illustrates an example of one or more hardwares arranged within a wearable device (100), according to one embodiment.

[0057] According to one embodiment, a wearable device (100) may include at least one display (350) and a frame supporting at least one display (350).

[0058] According to one embodiment, a wearable device (100) can be worn on a part of a user's body. The wearable device (100) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (100). For example, the wearable device (100) can output a virtual reality image to the user through at least one display (350) in response to a user's designated gesture acquired through the motion recognition camera (340-2) of FIG. 3.

[0059] According to one embodiment, at least one display (350) (e.g., display (120) of FIG. 4) within a wearable device (100) may provide visual information to a user. For example, at least one display (350) may include a transparent or translucent lens. At least one display (350) may include a first display (350-1) and / or a second display (350-2) spaced apart from the first display (350-1). For example, the first display (350-1) and the second display (350-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0060] Referring to FIG. 3, at least one display (350) may form a display area on a lens to provide a user wearing the wearable device (100) with visual information contained in external light passing through the lens, along with other visual information distinct from the visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens.

[0061] A display area formed by at least one display (350) may be formed on the second surface (332) among the first surface (331) and the second surface (332) of the lens. When a user wears the wearable device (100), external light may be incident on the first surface (331) and transmitted to the user by being transmitted through the second surface (332).

[0062] As another example, at least one display (350) may display a virtual reality image to be combined with a real screen transmitted via external light. The virtual reality image output from the at least one display (350) may be transmitted to the user's eyes via one or more hardware (e.g., optical devices (382, 384) and / or at least one waveguide (333, 334)) included in the wearable device (100).

[0063] According to one embodiment, a wearable device (100) may include waveguides (333, 334) that diffract light transmitted from at least one display (350) and relayed by optical devices (382, 384) and transmit the diffracted light to a user. The waveguides (333, 334) may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of the waveguides (333, 334). The nanopatterns may be formed based on a grating structure having a polygonal and / or curved shape.

[0064] Light incident on one end of the waveguides (333, 334) can be propagated to the other end of the waveguides (333, 334) by the nano-pattern. The waveguides (333, 334) can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, the waveguides (333, 334) can be arranged in the wearable device (100) to guide a screen displayed by at least one display (350) to the user's eyes. For example, the screen can be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the waveguides (333, 334).

[0065] According to one embodiment, a wearable device (100) may analyze an object included in a real-world image collected through a shooting camera (340-1), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (350). The virtual object may include at least one of text and an image for various information related to the object included in the real-world image.

[0066] The wearable device (100) can analyze objects based on multi-cameras, such as stereo cameras. For the object analysis, the wearable device (100) can perform time-of-flight (ToF) and / or simultaneous localization and mapping (SLAM) supported by the multi-cameras.

[0067] A user wearing a wearable device (100) can view an image displayed on at least one display (350). According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (100) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (100), the first display (350-1) and the second display (350-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (350). For example, the frame may support the first display (350-1) and the second display (350-2) to be positioned corresponding to the user's left and right eyes.

[0068] According to one embodiment, the wearable device (100) may include hardware that performs various functions. For example, the hardware may include a battery module (370), an antenna module (375), optical devices (382, 384), speakers (392-1, 392-2), microphones (394-1, 394-2, 394-3), a light-emitting module (not shown), and / or a printed circuit board (390). The various hardware may be arranged within the frame.

[0069] According to one embodiment, microphones (394-1, 394-2, 394-3) of the wearable device (100) may be arranged on at least a portion of the frame to acquire sound signals. A first microphone (394-1) arranged on the nose pad (310), a second microphone (394-2) arranged on the second rim (302), and a third microphone (394-3) arranged on the first rim (301) are illustrated in FIG. 3, but the number and arrangement of the microphones (394) are not limited to the embodiment of FIG. 3.

[0070] When the number of microphones (394) included in the wearable device (100) is two or more, the wearable device (100) can identify the direction of a sound signal by using a plurality of microphones placed on different parts of the frame.

[0071] According to one embodiment, the optical devices (382, 384) can transmit a virtual object transmitted from at least one display (350) to the wave guides (333, 334). For example, the optical devices (382, 384) can be projectors. The optical devices (382, 384) can be positioned adjacent to at least one display (350) or can be included within at least one display (350) as a part of the at least one display (350). The first optical device (382) can correspond to the first display (350-1), and the second optical device (384) can correspond to the second display (350-2). The first optical device (382) can transmit light output from the first display (350-1) to the first waveguide (333), and the second optical device (384) can transmit light output from the second display (350-2) to the second waveguide (334).

[0072] In one embodiment, the camera (340) may include an eye tracking camera (ET CAM) (340-1), a motion recognition camera (340-2), and / or a recording camera (340-3). The recording camera (340-3), the eye tracking camera (340-1), and the motion recognition camera (340-2) may be positioned at different locations on the frame and may perform different functions.

[0073] The gaze tracking camera (340-1) can output data indicating the gaze of a user wearing the wearable device (100). For example, the wearable device (100) can detect the gaze from an image including the user's pupils obtained through the gaze tracking camera (340-1). Although an example in which the gaze tracking camera (340-1) is positioned toward the user's right eye is illustrated in FIG. 3, the embodiment is not limited thereto, and the gaze tracking camera (340-1) can be positioned solely toward the user's left eye, or toward both eyes.

[0074] In one embodiment, the capturing camera (340-3) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (350). The at least one display (350) can display a single image in which information about the actual image or background including the image of the specific object acquired using the capturing camera is superimposed with a virtual image provided through optical devices (382, 384). In one embodiment, the capturing camera can be placed on a bridge (303) disposed between the first rim (301) and the second rim (302).

[0075] In one embodiment, the gaze tracking camera (340-1) can track the gaze of a user wearing the wearable device (100) and thereby align the user's gaze with visual information provided to at least one display (350) to implement a more realistic augmented reality. For example, when the user looks straight ahead, the wearable device (100) can naturally display environmental information related to the user's front at the location where the user is located on at least one display (350).

[0076] The gaze tracking camera (340-1) may be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (340-1) may receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light.

[0077] In one embodiment, the gaze tracking camera (340-1) may be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (340-1) may be positioned within the first rim (301) and / or the second rim (302) so as to face the direction in which the user wearing the wearable device (100) is positioned.

[0078] In one embodiment, the gesture recognition camera (340-2) can provide a specific event on a screen provided on at least one display (350) by recognizing the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body. The gesture recognition camera (340-2) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (350). The processor can identify the signal corresponding to the gesture and perform a designated function based on the identification. In one embodiment, the gesture recognition camera (340-2) can be disposed on the first rim (301) and / or the second rim (302).

[0079] In one embodiment, the camera (340) included in the wearable device (100) is not limited to the gaze tracking camera (340-1) and the motion recognition camera (340-2) described above. For example, the wearable device (100) can identify an external object included in the user's FoV using the photographing camera (340-3) positioned toward the user's FoV. The wearable device (100) can identify an external object based on a sensor for identifying the distance between the wearable device (100) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (340) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (100) may include a camera (340) (e.g., a face tracking (FT) camera) positioned toward the face of a user wearing the wearable device (100) to obtain an image including the face of the user. Although not shown, in one embodiment, the wearable device (100) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., the user's eyes, face, and / or an external object within the FoV) to be captured using the camera (340). The light source may include an LED of an infrared wavelength. The light source may be positioned in at least one of the frame and the hinge units (306, 307).

[0080] According to one embodiment, the battery module (370) may supply power to electronic components of the wearable device (100). In one embodiment, the battery module (370) may be disposed within the first temple (304) and / or the second temple (305). For example, the battery module (370) may be a plurality of battery modules (370). The plurality of battery modules (370) may be disposed within each of the first temple (304) and the second temple (305). In one embodiment, the battery module (370) may be disposed at an end of the first temple (304) and / or the second temple (305).

[0081] In one embodiment, the antenna module (375) can transmit signals or power to the outside of the wearable device (100), or receive signals or power from the outside. The antenna module (375) can be electrically and / or operatively connected to communication circuitry within the wearable device (100).

[0082] In one embodiment, the antenna module (375) may be positioned within the first temple (304) and / or the second temple (305). For example, the antenna module (375) may be positioned close to one surface of the first temple (304) and / or the second temple (305). In one embodiment, the speakers (392-1, 392-2) may output audio signals to the outside of the wearable device (100). The audio output module may be referred to as a speaker.

[0083] In one embodiment, the speakers (392-1, 392-2) may be positioned within the first temple (304) and / or the second temple (305) so as to be positioned adjacent to the ears of a user wearing the wearable device (100). For example, the wearable device (100) may include a second speaker (392-2) positioned within the first temple (304) and thus adjacent to the user's left ear, and a first speaker (392-1) positioned within the second temple (305) and thus adjacent to the user's right ear.

[0084] In one embodiment, the light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide information regarding a specific state of the wearable device (100) to the user. For example, when the wearable device (100) requires charging, the wearable device (100) may repeatedly emit red light at a designated time. In one embodiment, the light-emitting module may be disposed on the first rim (301) and / or the second rim (302).

[0085] Referring to FIG. 3, according to one embodiment, a wearable device (100) may include a printed circuit board (PCB) (390). The PCB (390) may be included in at least one of the first temple (304) and the second temple (305). The PCB (390) may include an interposer disposed between at least two sub-PCBs. One or more hardware components included in the wearable device (100) may be disposed on the PCB (390). The wearable device (100) may include a flexible PCB (FPCB) for interconnecting the hardware components.

[0086] According to one embodiment, a wearable device (100) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (100) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (100). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (100) can identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (100) based on the IMU.

[0087] FIG. 4 is an exemplary block diagram of a wearable device according to one embodiment.

[0088] According to one embodiment, a wearable device (100) may include at least one processor (e.g., the processor of FIG. 2), at least one of a communication circuit (110), a memory (115), a display (120), a camera (130), and a sensor (125). The processor, the communication circuit (110), the memory, the display (120), the camera (130), and the sensor (125) may be electrically and / or operably coupled with each other by electronic components such as a communication bus (not shown). Hereinafter, the hardwares being operably coupled may mean that a direct connection or an indirect connection is established between the hardwares, either wired or wireless, so that the second hardware is controlled by the first hardware among the hardwares.

[0089] The type and / or number of hardware included in the wearable device (100) is not limited to that shown in FIG. 4. For example, the wearable device (100) may include only some of the hardware components shown in FIG. 4.

[0090] According to one embodiment, the processor of the wearable device (100) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor may have a single-core processor structure or a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0091] According to one embodiment, the memory of the wearable device (100) may include a hardware component for storing data and / or instructions input and / or output to the processor of the wearable device (100). The memory may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multimedia card (eMMC).

[0092] According to one embodiment, a communication circuit of a wearable device (100) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (100) and an external user device. The communication circuit may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0093] According to one embodiment, the display of the wearable device (100) can output visualized information (e.g., virtual objects according to the execution of an application) to a user (e.g., the user of FIG. 1). For example, the display can be controlled by a processor including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0094] According to one embodiment, a sensor (125) of a wearable device (100) may generate electrical information that may be processed by a processor and / or memory of the wearable device (100) from non-electronic information related to the wearable device (100). The information may be referred to as sensor data. The sensor (125) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (100), and an inertial measurement unit (IMU) for detecting physical motion of the wearable device (100).

[0095] According to one embodiment, the camera (130) of the wearable device (100) may include a gesture camera, an eye tracking camera, a depth camera, and / or an RGB camera. According to one embodiment, the gesture camera may detect a user's movement. For example, at least one gesture camera (311) may be disposed in the wearable device (100) and may detect a user's hand movement within a preset distance. The gesture camera may include a SLAM camera (simultaneous localization and mapping camera) for recognizing information (e.g., location and / or direction) related to the surrounding space of the wearable device (100). The gesture recognition area of ​​the gesture camera may be set based on the capture range of the gesture camera.

[0096] In one embodiment, the gaze tracking camera can track the movements of the user's left and right eyes. In one embodiment, the processor can use the gaze tracking camera to determine the gaze direction of the left eye and the gaze direction of the right eye. For example, the gaze tracking camera (313) may include a first gaze tracking camera for determining the gaze direction of the left eye and a second gaze tracking camera for determining the gaze direction of the right eye.

[0097] According to one embodiment, a distance measuring camera (315) can measure a distance to an object located in front of a wearable device (100). The distance measuring camera can include a time of flight (TOF) camera and / or a depth camera.

[0098] According to one embodiment, the wearable device (100) can measure the distance to an object using a distance measurement camera. For example, the wearable device (100) can recognize one of the objects located in the user's FoV and measure the distance to the object based on a depth camera or a TOF camera. According to one embodiment, the RGB (red green blue) camera can obtain color information of the object and distance information to the object.

[0099] FIG. 5 is a drawing for explaining the operation of a wearable device according to one embodiment.

[0100] In operation 510, the wearable device (100) can display a virtual window on the display (120) in a state where a user wearing the wearable device (100) can view the real world surrounding the user. The real world surrounding the user may refer to a location where the user wearing the HMD is located.

[0101] For example, a virtual window may refer to a visual object in the form of a screen displayed on the display (120) of a wearable device (100). The wearable device (100) may execute an application and display content corresponding to the application through the virtual window. The wearable device (100) may use binocular disparity to display a virtual window floating in the space where the user is located.

[0102] In operation 515, the wearable device (100) can detect a user device (200) in the real world based on an image acquired from a first camera (135). The first camera (135) may be an RGB camera that captures an image of the front of the wearable device (100). The wearable device (100) can detect a user device (200) in the real world located in the user's FoV area using the image captured by the first camera (135). For example, the wearable device (100) can identify the user device (200) from the image using computer vision, sensor data, SLAM, and / or a pre-learned database.

[0103] According to one embodiment, the wearable device (100) can identify whether a virtual window is not displayed in the user's field of view (FoV) based on the 3D coordinates of the virtual window. The wearable device (100) can identify the user device in the image, and if it is determined that the virtual window is not displayed within the user's FoV area, the wearable device (100) can display a UI for recommending content near the user device (200). A method for obtaining the 3D coordinates of the virtual window is described in operation 520. For example, the wearable device (100) can display a UI corresponding to an application executable on the wearable device (100) near the user device (200).

[0104] In operation 520, the wearable device (100) can identify whether the user device (200) overlaps the virtual window in the user's line of sight based on the location of the user device (200) and the location of the virtual window.

[0105] In another embodiment, the wearable device (100) can represent the location of objects, virtual objects, etc. located within a space using a 3D coordinate system. The wearable device (100) can identify the physical structure of objects within the space, including their types and locations, using SLAM. Objects located within the space may include, for example, a sofa, a desk, a wall, a chair, or a floor. The wearable device (100) can determine a reference point (or origin) within the space based on information about the identified physical structure of the space, and obtain a 3D coordinate system corresponding to the space. The wearable device (100) can determine coordinates at which a virtual window is placed within the space, and place the virtual window at the determined coordinates. The wearable device (100) can obtain 3D coordinates corresponding to the location of the user device (200) within the space using a camera (130) or a sensor (125).

[0106] According to one embodiment, the wearable device (100) can identify whether the user device (200) and the virtual window overlap based on coordinates corresponding to the location of the user device (200) and the location of the virtual window. Overlapping the user device (200) and the virtual window may mean that the user device (200) and the virtual window appear to overlap from the user's viewpoint.

[0107] According to one embodiment, the wearable device (100) can project the 3D coordinates of the virtual window and the user device (200) into a 2D coordinate system displayed on the user's screen. For example, the wearable device (100) can set a coordinate axis in which the front-back direction of the wearable device (100) is the Z-axis, the left-right direction is the X-axis, and the up-down direction is the Y-axis. The wearable device (100) can project the 3D coordinates of the user device (200) and the virtual window onto an XY plane. The wearable device (100) can identify a first area corresponding to the virtual window projected onto the XY plane and a second area corresponding to the user device (200) projected onto the XY plane. The wearable device (100) can identify that the virtual window and the user device (200) overlap when the first area and the second area are identified as overlapping in the XY plane.

[0108] In operations 520-Y and 525, if the wearable device (100) identifies that the user device (200) overlaps with the wearable device (100), the wearable device (100) may request linkage information from the user device (200) and receive the linkage information obtained from the user device through the communication circuit (110). The linkage information may include identification information of an application running on the user device (200) and information about content provided through the application.

[0109] According to one embodiment, the wearable device (100) can identify applications related to the application running on the user device (200) based on the linkage information. A detailed description of the related applications is provided in detail in FIG. 12 below.

[0110] Information about content provided through an application can refer to the main information or content format provided to users by a specific application. Information about content provided through an application can include information about the type of content, its content, and its format.

[0111] For example, the type of content could refer to the primary type of information provided by the application. For example, if the user is running a document editing application, the content could be documents; if the user is running a music application, the content could be music; or if the user is running a video streaming application, the content could be videos.

[0112] For example, the content of a content may refer to the information an application provides to a user through the content. For example, the content may include text, images, and / or graphs if the content is a document; video and / or audio if the content is a video; or sound and / or lyrics if the content is a music.

[0113] For example, the format of the content may refer to the type of file that the application uses to execute or present the content to the user. For example, the format of the content may include the format of a document (e.g., docx, pdf, txt) if the content is a document editing application, the format of a music file (e.g., mp3, aac) if the content is music, or the format of a video file (e.g., mp4, mkv, etc.) if the content is video.

[0114] However, information about content provided through the application is not limited to the examples described above, and may include information necessary to display the content in a virtual window, such as metadata and application functions.

[0115] In operation 530, the wearable device (100) can identify an overlap state between the user device (200) and the virtual window. A detailed description of the overlap state between the user device (200) and the virtual window is described with reference to FIGS. 6A and 6B.

[0116] FIG. 6A and FIG. 6B are diagrams for explaining an overlap state of a user device and a virtual window according to one embodiment.

[0117] According to one embodiment, the wearable device (100) can identify an overlap state based on a first area corresponding to a virtual window projected on the XY plane and a second area corresponding to the user device (200).

[0118] FIG. 6A is a drawing illustrating a screen (600) displayed on a display (220) of a wearable device when the overlap state is a first state according to one embodiment.

[0119] Referring to FIG. 6A, the first state (615) may mean a state in which the entire user device (610) overlaps the virtual window (605). According to one embodiment, the wearable device (100) may identify the overlap state as the first state when the entire area of ​​the second area overlaps the first area.

[0120] FIG. 6b is a diagram illustrating a case where the overlap state is a second state according to one embodiment.

[0121] Referring to FIG. 6B, the second state (665) may mean a state in which a portion of the user device (660) overlaps with the virtual window (655). According to one embodiment, the wearable device (100) may identify the overlap state as the second state when a portion of the second region overlaps with the first region.

[0122] In actions 530-Y and 535, the wearable device (100) can display content corresponding to the linking information in the first virtual window based on the overlap state being identified as the first state.

[0123] Based on whether the overlap state is not identified as the first state in operations 530-N and 540, the wearable device (100) can identify whether the overlap state is identified as the second state. Based on whether the overlap state is identified as the second state in operations 540-Y and 545, the wearable device (100) can display a second virtual window around the first virtual window. The specific details of displaying the second virtual window are described in detail in FIGS. 9 and 10 .

[0124] In operation 550, the wearable device (100) can display content corresponding to the linkage information in a second virtual window. The specific details of displaying content corresponding to the linkage information in the virtual window in operations 535 and 550 are described in detail in FIG. 12.

[0125] FIG. 7 and FIG. 8 are drawings for explaining a screen expansion mode according to one embodiment.

[0126] FIG. 7 is a flowchart illustrating an operation for activating a screen expansion mode according to one embodiment.

[0127] The screen expansion mode may refer to a state in which a virtual window is determined to display content corresponding to the linking information based on an overlap state. For example, the screen expansion mode may include two modes. The first mode of the two expansion modes may refer to the wearable device (100) displaying content corresponding to the linking information in the first virtual window based on the overlap state being identified as the first state. The second mode of the two expansion modes may refer to the wearable device (100) displaying content corresponding to the linking information in the second virtual window based on the overlap state being identified as the second state.

[0128] Referring to FIG. 7, in operation 705, the wearable device (100) can identify the user device (200) based on an image acquired through the camera (130). In operation 710, the wearable device (100) can identify whether the user device (200) and the first virtual window overlap.

[0129] In steps 710-Y and 715, the wearable device (100) can identify whether the direction of the user's gaze intersects with the user device (200) or the virtual window. According to one embodiment, the wearable device (100) can obtain a vector corresponding to the direction of the gaze of the user wearing the wearable device (100) in 3D space using the second camera (140). The second camera (140) can be a gaze tracking camera for obtaining information about the direction of the user's gaze. The wearable device (100) can identify whether the vector representing the direction of the user's gaze in 3D space intersects the virtual window and the user device (200) based on 3D coordinates corresponding to the positions of the virtual window and the user device (200).

[0130] In actions 715-Y and 720, the wearable device (100) may display a UI for inputting whether to activate the screen extension mode on the display (120) when the user's gaze direction is identified as intersecting the first virtual window and not intersecting the user device.

[0131] According to one embodiment, the wearable device (100) can identify a virtual window to display content corresponding to the linking information based on whether the screen expansion mode is activated. The wearable device (100) can maintain the screen displayed in the first virtual window based on whether the screen expansion mode is deactivated.

[0132] In actions 715-N, 720-N, and 725, the wearable device (100) may disable the screen expansion mode based on whether the user's gaze direction does not intersect the first virtual window, whether the user device is identified as intersecting, or whether the user has made an input to disable the expansion mode.

[0133] In actions 720-Y and 730, the wearable device (100) can activate the screen expansion mode based on an input corresponding to the user's activation of the screen expansion mode.

[0134] FIG. 8 is a diagram for explaining an operation of activating a screen expansion mode according to one embodiment.

[0135] Referring to FIG. 8, according to one embodiment, the wearable device (100) may display a UI (830) for allowing an input to determine whether to activate the screen expansion mode on the display (120) when the user's gaze direction (805, 815) intersects with the virtual window (820, 825) and does not intersect with the user device.

[0136] According to one embodiment, the wearable device (100) may deactivate the screen expansion mode when the user's gaze direction (810) intersects the virtual window (820, 825) and the user device.

[0137] FIG. 9 is a flowchart illustrating a method for expanding a screen based on an overlap state according to one embodiment.

[0138] Referring to FIG. 9, in operation 905, the wearable device (100) may activate the screen layout mode. In operation 910, the wearable device (100) may identify whether the overlap state is the first state. In operations 910-Y and 915, the wearable device (100) may operate the screen expansion mode as the first mode based on the overlap state being identified as the first state. The wearable device (100) may display content corresponding to the linked data in the first virtual window.

[0139] In operations 910-N and 920, the wearable device (100) can identify whether the overlap state is a second state based on the overlap state being identified as not being a first state. In operations 920-Y and 925, the wearable device (100) can receive information about the arrangement state of the user device (200) based on the overlap state being identified as the second state. The information about the arrangement state of the user device (200) can include information for identifying whether the screen of the user device (200) is in portrait mode or landscape mode.

[0140] According to one embodiment, the wearable device (100) may determine the ratio of the width and height of the second virtual window based on information about the arrangement state of the user device. For example, when the arrangement state of the user device (200) is landscape mode, the wearable device (100) may identify that the width and height of the second virtual window is higher than the height ratio. For example, when the arrangement state of the user device (200) is landscape mode, the wearable device (100) may determine the width and height ratio based on an application running on the user device (200), the width and height ratio of the display of the user device, such as 16:9, 18:9, or 7:3.

[0141] For example, if the arrangement state of the user device (200) is portrait mode, the wearable device (100) can identify that the vertical ratio of the second virtual window is higher than the horizontal ratio. For example, if the arrangement state of the user device (200) is portrait mode, the wearable device (100) can determine the horizontal and vertical ratio, such as 9:16 or 9:18, based on the application running on the user device (200), the horizontal and vertical ratio of the display of the user device, etc.

[0142] In operation 930, the wearable device (100) can identify an area where a second virtual window is to be displayed based on an edge overlapped with the user device (200) among a plurality of edges of the first virtual window. According to one embodiment, the wearable device (100) can identify an edge overlapped with the user device (200) among a plurality of edges of the first virtual window. If there are multiple edges overlapped with the user device (200) among a plurality of edges of the first virtual window, the wearable device (100) can identify a reference edge. For example, the wearable device (100) can identify the reference edge by receiving an input from the user to select a reference edge, or a longest edge among the multiple edges overlapped with the user device (200). The wearable device (100) can identify an area where a second virtual window is to be displayed near the reference edge. The wearable device (100) can display a second virtual window in the identified area. In operation 930, the wearable device (100) can display content corresponding to the linked data in the second virtual window.

[0143] FIGS. 10A to 10C are drawings for explaining the arrangement of a second virtual window based on the overlap state being the second state according to one embodiment.

[0144] Referring to FIG. 10A, the wearable device (100) can identify an edge (1000, 1005) where the user device (200) overlaps among multiple edges of the virtual window. If there are multiple edges (1000, 1005) where the user device (200) overlaps, the wearable device (100) can identify the longer edge (1005) as a reference edge. Based on the user device (200) being in portrait mode, the wearable device (100) can identify an area (1015) where a second virtual object is to be displayed near the reference edge. The wearable device (100) can additionally place a second virtual object in an area (1020) where the second virtual object is to be displayed.

[0145] Referring to FIG. 10b, the wearable device (100) can identify an edge (1005) that overlaps the user device (200) among multiple edges of the virtual window. Based on the user device (200) being in landscape mode, the wearable device (100) can identify an area (1020) near the reference edge where a second virtual object is to be displayed. The wearable device (100) can additionally place a second virtual object in the area (1020) where the second virtual object is to be displayed.

[0146] Referring to FIG. 10c, the wearable device (100) can identify an edge (1025) where the user device (200) overlaps among multiple edges of the virtual window. Based on the user device (200) being in landscape mode, the wearable device (100) can identify an area (1030) near the reference edge where a second virtual object is to be displayed. The wearable device (100) can additionally place a second virtual object in the area (1020) where the second virtual object is to be displayed.

[0147] FIG. 11 is a drawing for explaining the arrangement of virtual windows based on the overlap state being the first state according to one embodiment.

[0148] Referring to FIG. 11, the wearable device (100) can identify that the entirety (1110) of the user device (200) overlaps within the virtual window (1105). The wearable device (100) can display content corresponding to the linkage information on the first virtual object.

[0149] FIG. 12 is a flowchart illustrating a method for displaying content corresponding to linkage information in a virtual window based on linkage information according to one embodiment.

[0150] Referring to FIG. 12, in operation 1205, the wearable device (100) may receive linkage information from the user device (200) if the user device (200) is identified as overlapping with the virtual window. In operation 1210, the wearable device (100) may identify an application that is related to the wearable device (100) based on the linkage information.

[0151] A relevant application may mean an application that can execute content running through the application on the user device (200).

[0152] According to one embodiment, the wearable device (100) can identify a related application based on application identification information included in the linking information. For example, the wearable device (100) can identify an application of the same type as an application running on the wearable device (100) identified based on the application identification information as a related application. The same type of application may refer to an application that performs the same function. For example, if a messenger application is running on the user device (200), the related application may be the messenger application. For example, if a web browser is running on the user device (200), the related application may be the web browser. For example, if a streaming application is running on the user device (200), the related application may be the streaming application. For example, if the user device (200) is displaying content such as an image or video, the related application may be an application capable of displaying the image or video.

[0153] According to one embodiment, the wearable device (100) can identify a relevant application based on the format of the content included in the linking information. For example, if the format of the content included in the linking information is a format supported by an application included in the wearable device (100), the wearable device (100) can identify the application as an application related to the linking information. For example, if a document editing application is running on the user device (200), the relevant application may be the document editing application. For example, if content such as an image or video is running on the user device (200), the relevant application may include an application capable of editing the image or video.

[0154] According to one embodiment, the wearable device (100) may store mapping information for a related application corresponding to an application running on the user device (200) in the memory (115). For example, when the user device (200) is displaying an image through an image display application, the wearable device (100) may identify a related application (e.g., an image display application, an image editing application, a search application, etc.) based on the mapping information.

[0155] In operations 1215-Y and 1220, the wearable device (100) may identify whether the application running in the first virtual window and the linkage information are related based on the fact that the overlap state is identified as the first state. In one embodiment, the wearable device (100) may identify the application running in the first virtual window as having a linkage information if the application running in the first virtual window corresponds to the application having a linkage information identified based on the linkage information in operation 1210. For example, if the application having a linkage identified in operation 1210 is an image editing application or an image display application, and the application running in the first virtual window is an image editing application, the wearable device (100) may identify the application running in the first virtual window as having a linkage information related to the application.

[0156] If the application running in the first virtual window is identified as not being related to the linkage information in actions 1220-N and 1225, the wearable device (100) can maintain the screen displayed in the first virtual window.

[0157] In one embodiment, the wearable device (100) may, when it is determined that the application running in the first virtual window and the linkage information are not related in operation 1225, terminate the application running in the first virtual window and run an application corresponding to the linkage information in the first virtual window.

[0158] In operations 1220-Y and 1230, the wearable device (100) can display content corresponding to the linkage information through the application running in the first virtual window based on the identification that the application running in the first virtual window and the linkage information are related. The wearable device (100) can obtain content corresponding to the linkage information based on information about content provided through the application included in the linkage information. The wearable device (100) can display content of the linkage information using the application based on the fact that the format of the content included in the linkage information is a format supported by the application running in the first virtual window.

[0159] In operations 1215-N and 1235, the wearable device (100) can identify whether the overlap state is a second state based on the identification that the overlap state is not a first state. In operations 1235-Y and 1240, the wearable device (100) can execute an application associated with the second virtual object based on the linkage information based on the identification that the overlap state is a second state.

[0160] In operation 1245, the wearable device (100) can display content corresponding to the linkage information through an application running in a second virtual window.

[0161] FIGS. 13 and 14 are drawings for explaining an operation of displaying content in a virtual window based on an overlap state according to one embodiment.

[0162] FIG. 13 is a diagram for explaining an operation of displaying content corresponding to linkage information when a part of a user device overlaps a virtual window according to one embodiment.

[0163] Referring to FIG. 13, the wearable device (100) can display a second virtual window (1315) on the display (120) based on the identification that a part of the user device overlaps the first virtual window (1310).

[0164] According to one embodiment, the wearable device (100) may execute a messenger in a second virtual window (1315) based on the identification that the application running on the user device (200) is a messenger (1305). According to one embodiment, the wearable device (100) may execute a messenger and display a chat room activated in the messenger of the user device (200) in a second virtual window (1315) based on metadata included in the linking information.

[0165] Referring to FIG. 13, the wearable device (100) can display content corresponding to linking information in the first virtual window (1410) based on the identification that a part of the user device overlaps the first virtual window (1410).

[0166] According to one embodiment, the wearable device (100) may execute a messenger and display a chat room activated in the messenger of the user device (200) in a second virtual window (1315) based on metadata included in the linking information. The metadata may include information for identifying the chat room activated in the messenger of the user device (200).

[0167] FIG. 14 is a diagram for explaining an operation of displaying content corresponding to linkage information when the entire user device overlaps a virtual window according to one embodiment.

[0168] According to one embodiment, a wearable device (100) can identify whether the linkage information is related based on the identification that the application running on the user device (200) is an image display application (1405) and the application running on the first virtual window is an image editing application. The wearable device (100) can display an image using the image editing application.

[0169] FIG. 15 is a drawing for explaining a screen layout method in a multi-screen situation according to one embodiment.

[0170] Referring to FIG. 15, a multi-screen situation (1505) may refer to a situation in which a user device (200) divides multiple screens and displays them on a display (220). According to one embodiment, a wearable device (100) may generate a second virtual window based on the identification of an overlap state between the user device (200) and a first virtual window as a second state. The wearable device (100) may display multiple screens on a second virtual window (1510).

[0171] According to one embodiment, the wearable device (100) may generate a plurality of virtual windows based on the identification of the overlap state as the second state and the identification of the multi-screen being displayed on the user device (200). The number of the plurality of virtual windows may be equal to or smaller than the number of screens constituting the multi-screen displayed on the user device (200). For example, the wearable device (100) may display a plurality of virtual windows (1715) corresponding to the screens constituting the multi-screen displayed on the user device (200). For example, when a user input for selecting some of the plurality of screens displayed on the user device (200) is received, the wearable device (100) may generate a virtual window (1720) corresponding to some of the plurality of screens displayed on the user device (200). The wearable device (100) may display the screens corresponding to the multi-screen in a parallel form (1515) in the generated plurality of virtual windows.

[0172] According to one embodiment, the wearable device (100) may place a virtual window near an edge where the user device (200) overlaps among a plurality of edges of the first virtual window. For example, the wearable device (100) may place all of the plurality of virtual windows near an edge where the user device (200) overlaps among a plurality of edges of the first virtual window (1515). For example, the wearable device (100) may place some of the plurality of virtual windows near an edge where the user device (200) overlaps among a plurality of edges of the first virtual window, and place the remaining virtual windows, excluding the placed virtual windows, near an edge located opposite to an edge where the user device (200) overlaps among a plurality of edges of the first virtual window, centered around the first virtual window (1525).

[0173] FIG. 16 is a drawing for explaining an example of adjusting the size of a virtual window according to one embodiment.

[0174] According to one embodiment, the wearable device (100) can obtain the distance (z) between the wearable device (100) and the user device using a distance measuring camera. The wearable device (100) can determine the size of the virtual window (1610) based on the distance (z) between the user device (200) and the wearable device (100). For example, when the distance (z) between the user device (200) and the wearable device (100) increases by 2 cm, the wearable device (100) can display a virtual window (1605) whose size is reduced by 20%.

[0175] According to one embodiment, the wearable device (100) can adjust the size of a virtual window based on sensor information received from the user device (200). According to one embodiment, the wearable device (100) can receive sensor information from the user device (200) that senses a motion of the user swiping two points on the screen. Based on the sensor information that senses the motion of the user swiping two points on the screen, the wearable device (100) can identify the distance between the user's touch points. The wearable device (100) can determine the size of the virtual window screen in proportion to the distance between the user's touch points.

[0176] According to one embodiment, the wearable device (100) may receive sensor information from the user device (200) that senses a user's swiping motion on the screen. The wearable device (100) may identify the direction in which the user swiped the screen of the user device (200) based on the sensor information. The wearable device (100) may adjust the size of the virtual window (1610) based on the direction in which the user swiped the screen.

[0177] FIG. 17 is a diagram illustrating a method for displaying content running on a user device when there is no virtual screen, according to one embodiment.

[0178] Referring to FIG. 17, the wearable device (100) can identify a plane or display for identifying a virtual area (1705) within the user's FoV area using SLAM. The plane for identifying the virtual area may include, for example, a whiteboard, a wall, or the display of a powered-off display device. The display may include, for example, a display that constitutes a TV, a laptop, or a monitor.

[0179] According to one embodiment, the wearable device (100) can identify whether the user's gaze direction obtained using the gaze tracking camera intersects a plane. If the wearable device (100) identifies the user's gaze direction intersecting the plane, the wearable device (100) can identify an area corresponding to the plane as a virtual area (1705) for displaying content. The wearable device (100) can display content in the virtual area (1705) based on the identification that the virtual window and the user device (200) overlap.

[0180] According to one embodiment, the wearable device (100) can identify whether the direction of the user's gaze intersects the display of the display device. If the wearable device (100) identifies that the direction of the user's gaze intersects the display, the wearable device (100) can identify an area corresponding to the display as a virtual area (1705). The wearable device (100) can identify whether the virtual area and the user device (200) overlap. If the wearable device (100) identifies that the user device (200) overlaps the virtual area, the wearable device (100) can display content corresponding to content running on the user device (200) in the virtual area (1705).

[0181] According to one embodiment, the wearable device (100) may display content in a virtual area (1705) corresponding to the display device when the overlap state between the display and the user device (200) is identified as a first state. According to one embodiment, the wearable device (100) may create a second virtual window near the virtual area (1705) when the overlap state between the display and the user device (200) is identified as a second state. The wearable device (100) may display content in the second virtual window.

[0182] As described above, a wearable device according to one embodiment of the present disclosure may include at least one processor including a communication circuit, a first camera, a display, a memory for storing instructions, and processing circuitry.

[0183] The instructions, when individually or collectively executed by the at least one processor, cause the wearable device to display a first virtual window on the display in a state in which a user wearing the wearable device (100) can view the real world around the user, detect a user device in the real world based on an image acquired from the first camera, and in response to the detection of the user device, identify whether the user device overlaps the first virtual window in a field of view of the user based on a location of the user device and a location of the first virtual window, and if the user device is identified as being overlapped with the first virtual window, request linkage information to the user device through the communication circuit, and in response to the request, receive the linkage information from the user device through the communication circuit, and display content corresponding to the linkage information on the first virtual window based on the overlap state being a first state, and based on the overlap state being a second state, display content corresponding to the linkage information on the first virtual window. A second virtual window can be displayed in the vicinity, and content corresponding to the linkage information can be displayed in the second virtual window.

[0184] The first state may be a state in which the entire user device overlaps the virtual window, and the second state may be a state in which a part of the user device overlaps within the virtual window.

[0185] The above linkage information may include either identification information of an application running on the user device or information about content provided through the application.

[0186] The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an area around the first virtual window in which the second virtual window is to be displayed based on an edge of the first virtual window that overlaps the user device among a plurality of edges of the first virtual window, and to display the second virtual window in the identified area, based on the overlap state being the second state.

[0187] The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive information about a placement state of the user device from the user device through the communication circuit based on the overlap state being the second state, determine a ratio of horizontal and vertical lengths of the second virtual window based on the information about the placement state, and display the second virtual window based on the determined ratio.

[0188] The wearable device further includes a second camera, and the instructions, when individually or collectively executed by the at least one processor, can cause the wearable device to identify a gaze direction of a user wearing the wearable device using the second camera based on whether the user device overlaps the first virtual window, identify an overlap state of the first virtual window and the user device based on whether the gaze direction intersects the first virtual window and does not intersect the user device, and maintain a screen displayed in the first virtual window based on whether the gaze direction does not intersect the first virtual window or intersects the user device.

[0189] The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify whether the linking information is related to an application displayed in the first virtual window based on the overlap state being the first state, to display content corresponding to the linking information in the first virtual window based on the linking information being identified as being related to the application, and to maintain the screen displayed in the first virtual window based on the linking information being identified as not being related to the application.

[0190] The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to execute an application running on the user device using identification information of the application based on the overlap state being a second state, and to display content provided by the application running on the user device in the second virtual window based on information about content provided through the application.

[0191] According to one embodiment of the present disclosure, a control method of a wearable device including a first camera, a display, and a communication circuit, the control method comprises: an operation of displaying a first virtual window on the display in a state in which a user wearing the wearable device (100) can view the real world around the user; an operation of detecting a user device in the real world based on an image acquired from the first camera; an operation of identifying whether the user device overlaps the first virtual window in a field of view of the user based on a location of the user device and a location of the first virtual window in response to the detection of the user device; an operation of requesting linkage information to the user device through the communication circuit and receiving the linkage information from the user device through the communication circuit in response to the request; an operation of displaying content corresponding to the linkage information on the first virtual window based on the overlap state being a first state; and an operation of displaying a second virtual window around the first virtual window based on the overlap state being a second state, and receiving information corresponding to the linkage information. It may include an action of displaying corresponding content in the second virtual window.

[0192] The first state may be a state in which the entire user device overlaps the virtual window, and the second state may be a state in which a part of the user device overlaps within the virtual window.

[0193] The above linkage information may include either identification information of an application running on the user device or information about content provided through the application.

[0194] The operation of displaying the content in the second virtual window may include an operation of identifying an area in which the second virtual window is to be displayed around the first virtual window based on an edge overlapped by the user device among a plurality of edges of the first virtual window, based on the overlap state being the second state, and an operation of displaying the second virtual window in the identified area.

[0195] The operation of displaying the content in the second virtual window may include an operation of receiving information about the arrangement state of the user device from the user device through the communication circuit based on the overlap state being the second state, an operation of determining a ratio of the horizontal and vertical lengths of the second virtual window based on the information about the arrangement state, and an operation of displaying the second virtual window based on the determined ratio.

[0196] The wearable device may further include a second camera, and the control method may further include an operation of identifying a gaze direction of a user wearing the wearable device using the second camera based on whether the user device overlaps the first virtual window, an operation of identifying an overlap state between the first virtual window and the user device based on whether the gaze direction intersects the first virtual window and does not intersect the user device, and an operation of maintaining a screen displayed on the first virtual window based on whether the gaze direction does not intersect the first virtual window or intersects the user device.

[0197] The control method may further include an operation of identifying whether the linking information is related to an application displayed in the first virtual window based on the overlap state being the first state, an operation of displaying content corresponding to the linking information in the first virtual window based on the linking information being identified as being related to the application, and an operation of maintaining the screen displayed in the first virtual window based on the linking information being identified as not being related to the application.

[0198] The operation of displaying the content in the second virtual window may include an operation of executing an application running on the user device using identification information of the application based on the overlap state being the second state, and displaying content provided by the application running on the user device in the second virtual window based on information about the content provided through the application.

[0199] Although various embodiments have been described above, each embodiment is not necessarily implemented individually, and may be implemented together in a single product by being combined in whole or in part with at least one other embodiment.

[0200] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0201] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0202] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0203] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0204] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In a wearable device (100), Communication circuitry (110); First camera (135); display (120); Memory (115) for storing instructions; and At least one processor (105) comprising processing circuitry; The above instructions, when individually or collectively executed by the at least one processor (105), cause the wearable device (100) to: A first virtual window is displayed on the display (120) in a state where a user wearing the wearable device (100) can see the real world around the user, Detecting the real-world user device based on the image acquired from the first camera (135), In response to the user device being detected, identifying whether the user device overlaps the first virtual window in a field of view of the user based on the location of the user device and the location of the first virtual window; When the user device is identified as overlapping the first virtual window, linking information is requested from the user device through the communication circuit (110), and in response to the request, the linking information is received from the user device through the communication circuit (110). Based on the above overlap state being the first state, displaying content corresponding to the linking information in the first virtual window, A wearable device (100) that displays a second virtual window around the first virtual window based on the above overlap state being a second state, and displays content corresponding to the linking information in the second virtual window.

2. In paragraph 1, The above first state is a state in which the entire user device overlaps the virtual window, The second state is a wearable device (100) in which a part of the user device overlaps within the virtual window.

3. In either paragraph 1 or paragraph 2, A wearable device (100), wherein the above linkage information includes either identification information of an application running on the user device or information about content provided through the application.

4. In any one of paragraphs 1 to 3, The above instructions, when individually or collectively executed by the at least one processor (105), cause the wearable device (100) to: Based on the above overlapping state being the second state, the user device identifies an area around the first virtual window where the second virtual window is to be displayed based on an overlapping edge among a plurality of edges of the first virtual window, A wearable device (100) that displays the second virtual window in the identified area.

5. In any one of paragraphs 1 to 4, The above instructions, when individually or collectively executed by the at least one processor (105), cause the wearable device (100) to: Based on the above overlap state being the second state, information on the arrangement status of the user device is received from the user device through the communication circuit (110), Determine the ratio of the horizontal and vertical lengths of the second virtual window based on information about the above arrangement status, A wearable device (100) that displays the second virtual window based on the determined ratio.

6. In any one of paragraphs 1 to 5, Further including a second camera (140); The above instructions, when individually or collectively executed by the at least one processor (105), cause the wearable device (100) to: Based on the user device overlapping the first virtual window, the gaze direction of the user wearing the wearable device (100) is identified using the second camera (140), Identifying an overlap state between the first virtual window and the user device based on the identification that the gaze direction intersects the first virtual window and does not intersect the user device, A wearable device (100) that maintains the screen displayed in the first virtual window based on whether the gaze direction does not intersect the first virtual window or whether the user device is identified as intersecting.

7. In any one of paragraphs 1 to 6, The above instructions, when individually or collectively executed by the at least one processor (105), cause the wearable device (100) to: Based on the above overlap state being the first state, identifying whether the linkage information is related to the application displayed in the first virtual window, Based on the identification that the above linkage information is related to the application, displaying content corresponding to the above linkage information in the first virtual window, A wearable device (100) that maintains the screen displayed in the first virtual window based on the identification that the above linkage information is not related to the application.

8. In any one of paragraphs 1 to 7, The above instructions, when individually or collectively executed by the at least one processor (105), cause the wearable device (100) to: A wearable device (100) that executes an application running on the user device using the identification information of the application based on the overlap state being a second state, and displays content provided by the application running on the user device in the second virtual window based on information about content provided through the application.

9. In a control method of a wearable device (100) including a first camera (135), a display (120), and a communication circuit (110), An action of displaying a first virtual window on the display (120) in a state where a user wearing the wearable device (100) can see the real world surrounding the user; An operation of detecting a user device in the real world based on an image obtained from the first camera (135); In response to the user device being detected, an action of identifying whether the user device overlaps the first virtual window in a field of view of the user based on a location of the user device and a location of the first virtual window; When the user device is identified as overlapping the first virtual window, an operation of requesting linkage information from the user device through the communication circuit (110) and receiving the linkage information from the user device through the communication circuit (110) in response to the request; An operation of displaying content corresponding to the linking information in the first virtual window based on the overlap state being the first state; and A control method comprising: an operation of displaying a second virtual window around the first virtual window based on the overlap state being a second state, and displaying content corresponding to the linking information in the second virtual window; 10. In paragraph 9, The above first state is a state in which the entire user device overlaps the virtual window, A control method in which the second state is a state in which a part of the user device overlaps within the virtual window.

11. In either of paragraphs 9 or 10, A control method, wherein the above linkage information includes either identification information of an application running on the user device or information about content provided through the application.

12. In either of paragraphs 9 or 11, The action of displaying the above content in the second virtual window is: An operation of identifying an area in which the second virtual window is to be displayed around the first virtual window based on an edge overlapped by the user device among a plurality of edges of the first virtual window, based on the overlap state being the second state; and A control method comprising: an operation for displaying the second virtual window in the identified area; 13. In either of paragraphs 9 or 12, The action of displaying the above content in the second virtual window is: An operation of receiving information about the arrangement state of the user device from the user device through the communication circuit (110) based on the overlap state being the second state; An operation of determining a ratio of the horizontal and vertical lengths of the second virtual window based on information about the above arrangement state; and A control method comprising: an operation of displaying the second virtual window based on the determined ratio; 14. In either of paragraph 9 or paragraph 13, The above wearable device (100) further includes a second camera (140); The above control method is, An operation of identifying a gaze direction of a user wearing the wearable device (100) using the second camera (140) based on the user device being overlapped with the first virtual window; An operation of identifying an overlap state between the first virtual window and the user device based on the identification that the gaze direction intersects the first virtual window and does not intersect the user device; and A control method further comprising: an operation of maintaining a screen displayed on the first virtual window based on whether the gaze direction does not intersect the first virtual window or whether the user device is identified as intersecting the first virtual window; 15. In either of paragraph 9 or paragraph 14, An operation of identifying whether the linkage information is related to an application displayed in the first virtual window based on the overlap state being the first state; An operation of displaying content corresponding to the linkage information in the first virtual window based on the identification that the linkage information is related to the application; and A control method further comprising: an operation of maintaining a screen displayed in the first virtual window based on the identification that the linkage information is not related to the application;

Images