Wearable device for communicating with external electronic device, and method therefor

Abstract

A processor of a wearable device, according to one embodiment, can display a first visual object corresponding to an external object on a display. The processor can: identify, on the basis of the direction of a gaze toward the first visual object, indicated by sensor data of a sensor, a motion of the external object by using a camera; establish a communication link with at least one external electronic device capable of tracking the external object identified on the basis of the motion; and acquire information about the external object from the at least one external electronic device through the communication link on the basis of identifying that the display of the first visual object is stopped after the communication link is established, and display, on the display on the basis of the acquired information, a second visual object related to the external object.

Description

Wearable device for communicating with external electronic devices and method thereof

[0001] The present disclosure relates to a wearable device and method for communicating with an external electronic device.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] According to an embodiment, a wearable device may include at least one processor including one or more sensors, communication circuitry, a camera, a display, a memory storing instructions including one or more storage media, and processing circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to control the camera to acquire an image to be displayed on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a first visual object corresponding to an external object within the acquired image. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display the image on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a gaze of a user wearing the wearable device using the one or more sensors. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a motion of the external object using the camera based on a direction of the gaze toward the first visual object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to establish a communication link using the communication circuitry with at least one external electronic device capable of tracking the external object identified based on the motion.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain information about the external object from the at least one external electronic device through the communication link based on identifying that the display of the first visual object has been discontinued after establishing the communication link, and to display a second visual object related to the external object on the display based on the obtained information.

[0004] In one embodiment, a method of a wearable device may include an operation of controlling a camera of the wearable device to obtain an image to be displayed on a display of the wearable device. The method may include an operation of identifying a first visual object corresponding to an external object within the obtained image. The method may include an operation of displaying the image on the display of the wearable device. The method may include an operation of identifying a gaze of a user wearing the wearable device using one or more sensors of the wearable device. The method may include an operation of identifying a motion of the external object using the camera based on a direction of the gaze toward the first visual object. The method may include an operation of establishing a communication link with at least one external electronic device capable of tracking the external object identified based on the motion using a communication circuit of the wearable device. The method may include an operation of obtaining information about the external object from the at least one external electronic device through the communication link based on determining that the display of the first visual object has been discontinued after establishing the communication link, and displaying a second visual object related to the external object on the display based on the obtained information.

[0005] According to one embodiment, a wearable device may include one or more sensors, a communication circuit, a camera, a display, and at least one processor. The at least one processor may be configured to control the camera to obtain an image to be displayed on the display. The at least one processor may be configured to display the image including a visual object corresponding to an external object on the display. The at least one processor may be configured to identify a direction of a gaze of a user wearing the wearable device using sensor data of the one or more sensors. The at least one processor may be configured to identify a motion of the external object moving along a direction from a first location using the camera based on the direction of the gaze toward the visual object. The at least one processor may be configured to establish a communication link with a first external electronic device adjacent to the first location using the communication circuit based on the motion. The at least one processor may be configured to change the communication link to share information about the external object with the first external electronic device and a second external electronic device adjacent to the second location, based on identifying that the external object has moved from the first location to the second location along the direction after establishing the communication link.

[0006] In one embodiment, a method of a wearable device may include an operation of controlling a camera of the wearable device to obtain an image to be displayed on the display. The method may include an operation of displaying the image including a visual object corresponding to an external object on the display of the wearable device. The method may include an operation of identifying a direction of a gaze of a user wearing the wearable device using sensor data of one or more sensors of the wearable device. The method may include an operation of identifying a motion of the external object moving along a direction from a first location using the camera based on the direction of the gaze toward the visual object. The method may include an operation of establishing a communication link with a first external electronic device adjacent to the first location using a communication circuit of the wearable device based on the motion. The method may include, after establishing the communication link, an operation of changing the communication link to share information about the external object with the first external electronic device and a second external electronic device adjacent to the second location, based on identifying that the external object has moved from the first location to the second location along the direction.

[0007] FIG. 1 is a drawing for explaining an example of a screen displayed through a display of a wearable device according to one embodiment.

[0008] FIG. 2 illustrates an example block diagram of a wearable device according to one embodiment.

[0009] FIG. 3 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0010] FIG. 4 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0011] FIG. 5 illustrates an exemplary operation of a wearable device for identifying a user's motion relative to an external object.

[0012] FIG. 6 illustrates an exemplary operation of a wearable device to form a cluster including one or more external electronic devices.

[0013] FIG. 7 illustrates an exemplary operation of a wearable device to form a cluster including one or more external electronic devices.

[0014] FIG. 8 illustrates an example of a screen displayed by a wearable device communicating with an external electronic device.

[0015] FIG. 9 illustrates an example of a screen displayed by a wearable device communicating with an external electronic device.

[0016] FIG. 10 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0017] FIG. 11 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0018] FIG. 12 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0019] FIG. 13 illustrates an example of a flowchart of a wearable device according to one embodiment.

[0020] FIG. 14 illustrates an example of a signal flow diagram between a wearable device and an external electronic device according to one embodiment.

[0021] FIG. 15A illustrates an example of a perspective view of a wearable device according to one embodiment.

[0022] FIG. 15b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0023] FIGS. 16A and 16B illustrate an example of an appearance of a wearable device according to one embodiment.

[0024] Figure 17 is an example diagram of a network environment related to a metaverse service.

[0025] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0026] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0027] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0028] FIG. 1 is a diagram illustrating an example of a screen (130) displayed through a display of a wearable device (101) according to one embodiment. The wearable device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). The wearable device (101) may be referred to as a head-mounted device (HMD), a headgear electronic device, a glasses-type electronic device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of one or more hardware components included in the wearable device (101) is exemplarily described with reference to FIG. 2. An example of the structure of a wearable device (101) that can be worn on the head of a user (110) is described with reference to FIGS. 15A, 15B, 16A, and / or 16B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may be combined with an accessory to be attached to the head of a user to form an HMD.

[0029] In one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). The wearable device (101), in this state, may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or active area) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.

[0030] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0031] Referring to FIG. 1, a plurality of wearable devices (e.g., a first wearable device (101-1) and / or a second wearable device (101-2)) worn by different users (e.g., a first user (110-1) and / or a second user (110-2)) are illustrated. As shown in FIG. 1, the first wearable device (101-1) is positioned at a point (p1), and the second wearable device (101-2) is positioned at a point (p2) spaced apart from the point (p1). The first wearable device (101-1) may display an image and / or video of the front direction of the first user (110-1) wearing the first wearable device (101-1) on a screen (130). In the exemplary situation of FIG. 1 where an external object (120) (e.g., a ball) is positioned at a point (o1) in front of a first user (110-1), the first wearable device (101-1) may display a screen (130) including a visual object (132) corresponding to the external object (120). In one embodiment where the first wearable device (101-1) provides a screen (130) based on a VST, the visual object (132) may correspond to at least a portion of an image and / or video acquired by the first wearable device (101-1).

[0032] In one embodiment, the wearable device (101) may execute a function for tracking an external object (120) corresponding to a visual object (132) displayed through a screen (130). For example, the first wearable device (101-1) may track the external object (120) based on the function, and obtain information about the type (e.g., class and / or category) and / or motion of the external object (120). The first wearable device (101-1) may display at least a portion of the obtained information within the screen (130). Referring to FIG. 1, while displaying a visual object (132) at a point (s1) within a screen (130) corresponding to a point (o1) of an external object (120), the first wearable device (101-1) may display a visual object (134) including at least a portion of the information at a point within the screen (130) linked to the visual object (132). A visual object (134) including a numeric value (e.g., “120 km / h”) representing the speed of the external object (120) is illustrated as an example, but the embodiment is not limited thereto.

[0033] In the exemplary case of FIG. 1, the first wearable device (101-1) may communicate with the second wearable device (101-2) to execute a function for tracking an external object (120). For example, the first wearable device (101-1) may identify one or more external electronic devices (e.g., the second wearable device (101-2)) for sharing information about the external object (120) based on the motion of the external object (120) identified by the first wearable device (101-1) (e.g., motion identified by at least one of a moving direction, a speed, and / or a velocity). The first wearable device (101-1) may identify at least one external electronic device capable of tracking the external object (120) among the external electronic devices adjacent to the first wearable device (101-1). Referring to FIG. 1, the first wearable device (101-1) that has identified the movement direction (d) of the external object (120) can identify that the distance between the second wearable device (101-2) and the external object (120) decreases as the external object (120) moves along the movement direction (d). The first wearable device (101-1) can establish a communication link (140) for sharing information about the external object (120) with the second wearable device (101-2) whose distance from the external object (120) decreases. The first wearable device (101-1) can establish a communication link (140) with the second wearable device (101-2) based on identifying the motion of an external object (120) and / or identifying the focus of the first user (110-1) on the external object (120).

[0034] Referring to FIG. 1, an external object (120) moving along a movement direction (d) may move from a point (o1) to a point (o2). At a timing when the external object (120) is located at the point (o2), the first wearable device (101-1) may display a visual object (132) corresponding to the external object (120) at a point (s2) adjacent to the periphery of the screen (130). The periphery of the screen (130) may correspond to a boundary of a field-of-view (FoV) of a first user (110-1) wearing the first wearable device (101-1). After the timing, as the external object (120) moves along the movement direction (d), the external object (120) may move out of the FoV. After the above timing, the first wearable device (101-1) may stop displaying the visual object (132) corresponding to the external object (120) on the screen (130).

[0035] In one embodiment, the first wearable device (101-1), which identifies that the display of a visual object (132) corresponding to an external object (120) within the screen (130) has been interrupted, may obtain information about the external object (120) using a communication link established with the second wearable device (101-2). An exemplary case in which the display of the visual object (132) is interrupted based on the external object (120) moving out of the FoV of the first user (110-1) wearing the first wearable device (101-1) is described, but the embodiment is not limited thereto. For example, if the external object (120) is occluded by another external object, the visual object (132) corresponding to the external object (120) may not be displayed within the screen (130). In the above example, the first wearable device (101-1), which has identified that the display of the visual object (132) has been interrupted, can obtain information about the external object (120) from an external electronic device (e.g., the second wearable device (101-2)) connected via the communication link.

[0036] Referring to FIG. 1, an external object (120) moving along a direction (d) to a point (o3) may be included in the FoV of a second user (101-2) wearing a second wearable device (101-2). The second wearable device (101-2) may transmit an image and / or a video related to the external object (120) at the point (o3) to the first wearable device (101-1) via a communication link. The first wearable device (101-1) that receives the image and / or the video may display a visual object (150) based on the image and / or the video on a screen (130). Within the visual object (150), the first wearable device (101-1) may display an image and / or a video (e.g., an image (152)) obtained from the second wearable device (101-2). The first wearable device (101-1) may display a visual object (150) including text (e.g., “From User B”) indicating the second wearable device (101-2) that provided the image and / or video.

[0037] As described above, after an external object (120) focused on by a first user (110-1) leaves the FoV of the first user (110-1), the first wearable device (101-1) may display a visual object (150) related to the external object (120) within the screen (130) based on information received from an external electronic device, such as a second wearable device (101-2). In order to continuously track the external object (120), the first wearable device (101-1) may establish a communication link based on the motion of the external object (120). When an external object (120) is out of the FoV of the first user (110-1) and / or is not visible through the screen (130) displayed by the first wearable device (101-1), the first wearable device (101-1) can obtain information about the external object (120) through an external electronic device (e.g., the second wearable device (101-2)) connected through the communication link. Based on the information, the first wearable device (101-1) can continuously provide information about the external object (120) to the first user (110-1) wearing the first wearable device (101-1) even after the external object (120) is out of the FoV of the first user (110-1). Using the above communication link, the first wearable device (101-1) can provide information about an external object (120) tracked by the first wearable device (101-1) to an external electronic device, such as the second wearable device (101-2).

[0038] Below, with reference to FIG. 2, an exemplary hardware configuration of a wearable device (101) for exchanging information about an external object (120) is described.

[0039] FIG. 2 illustrates an example block diagram of a wearable device (101) according to one embodiment. The wearable device (101) described with reference to FIG. 1 may include at least one of the hardware components of the wearable device (101) distinguished by different blocks in FIG. 2.

[0040] Referring to FIG. 2, a wearable device (101) according to one embodiment may include at least one of a processor (210), a memory (215), a display (220), a camera (225), a sensor (230), and a communication circuit (235). The processor (210), the memory (215), the display (220), the camera (225), the sensor (230), and / or the communication circuit (235) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (202). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 2. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 2.

[0041] According to one embodiment, the processor (210) of the wearable device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (210) may have a multi-core processor structure, such as a dual core, a quad core, a hexa core, and / or an octa core.

[0042] The memory (215) of the wearable device (101) according to one embodiment may include a hardware component for storing data and / or instructions input to and / or output from the processor (210). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC).

[0043] In one embodiment, the display (220) of the wearable device (101) can output visualized information to a user of the wearable device (101) (e.g., the first user (110-1) and / or the second user (110-2) of FIG. 1). For example, the display (220) can be configured to visualize information provided from a processor (210) including a circuit such as a graphic processing unit (GPU). The display (220) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (220) may include a projector (or projection assembly) for projecting light onto the lens.

[0044] In one embodiment, the camera (225) of the wearable device (101) may include optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The camera (225) may be referred to as an image sensor and may be included in the sensor (230) of FIG. 2. The plurality of optical sensors included in the camera (225) may be arranged in the form of a two-dimensional array. The camera (225) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (225) may mean one (a) two-dimensional frame data acquired from the camera (225). For example, video data captured using a camera (225) may mean a sequence of two-dimensional frame data obtained from the camera (225) according to a frame rate. The camera (225) may be positioned toward the direction in which the camera (225) receives light and may further include a flash light for outputting light toward the direction.

[0045] According to one embodiment, the wearable device (101) may include a plurality of cameras, for example, cameras (225), which are arranged facing different directions. Referring to FIG. 2, the camera (225) included in the wearable device (101) may include a gaze tracking camera (225-1) and / or an outward camera (225-2). The gaze tracking camera (225-1) may be arranged toward at least one of the two eyes of a user wearing the wearable device (101). The processor (210) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking camera (225-1). The gaze tracking camera (225-1) may include an infrared (IR) sensor. The gaze tracking camera (225-1) may be referred to as an eye sensor, a gaze tracker, and / or an eye tracker.

[0046] Referring to FIG. 2, the external camera (225-2) may be positioned to face the front of a user wearing the wearable device (101) (e.g., in a direction in which both eyes may face). Using images and / or videos acquired from the external camera (225-2), the processor (210) may identify an external object (e.g., the external object (120) of FIG. 1). The embodiment is not limited thereto, and the processor (210) may identify a position, shape, and / or gesture (e.g., a hand gesture) of a hand based on images and / or videos acquired from the external camera (225-2).

[0047] According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information from non-electronic information related to the wearable device (101), which may be processed and / or stored by a processor (210) and / or a memory (215) of the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), an inertial measurement unit (IMU) (e.g., an acceleration sensor, a geomagnetic sensor, a gravity sensor, or any combination thereof) for detecting a physical motion of the wearable device (101).

[0048] In one embodiment, the communication circuit (235) of the wearable device (101) may include circuitry for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device. The communication circuit (235) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (235) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), 6G and / or above-6G.

[0049] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (210) may perform at least one of the operations of FIGS. 3 and 10 to 14 when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, and / or application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (215), and that the one or more applications are stored in a format executable by the processor (210) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, an application may include a program and / or a library related to a service provided to a user.

[0050] Referring to FIG. 2, programs installed in the wearable device (101) may be included in any one of different layers, including the application layer (240), the framework layer (250), and / or the hardware abstraction layer (HAL) (280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (220), the camera (225), and / or the sensor (230)) of the wearable device (101) may be included in the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer in terms of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 2 may not mean that the address space of the memory (215) is divided by the layers, as they are logically separated.

[0051] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye-gaze tracker (274), and / or a face tracker (275)) may be included. The programs included in the framework layer (250) may provide an executable API (application programming interface) based on other programs.

[0052] For example, a program designed to target users of a wearable device (101) may be included within the application layer (240). As an example of programs included in the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (240) may call an application programming interface (API) to cause execution of functions supported by programs classified into the framework layer (250).

[0053] For example, the wearable device (101) may display one or more visual objects on the display (220) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (241). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within the virtual space based on the execution of the XR system UI (241).

[0054] Referring to FIG. 2, a lightweight renderer (243) and / or an XR plug-in (244) are illustrated to be included within the XR system UI (241), but are not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute a lightweight renderer (243) and / or an XR plug-in (244) within the framework layer (250).

[0055] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plugin (244) can be referred to as an open XR native client from the perspective of defining (or setting up) the entire rendering pipeline.

[0056] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (220) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with those of the XR plug-in (244) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0057] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting the virtual space service. The wearable device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (230) based on the execution of the virtual space manager (251), and may display at least a portion of the virtual space on the display (220). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0058] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module. The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the wearable device (101) may perform rendering for a virtual space service to the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0059] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the wearable device (101) may display a screen representing a virtual space on the display (220), while superimposing another screen representing an actual space obtained through the camera (225) on at least a portion of the screen.

[0060] For example, the virtual space manager (251) may include an input manager (254). The wearable device (101) may identify data (e.g., sensor data) obtained by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The wearable device (101) may use the obtained data to identify user input related to the wearable device (101). The user input may be related to a motion (e.g., hand gesture), gaze, and / or speech of the user identified by the sensor (230).

[0061] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0062] According to one embodiment, the recognition service layer (270) may include one or more programs for processing data acquired from a sensor (230) (or a camera (225)). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye tracker (274), and / or a face tracker (273). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIG. 2.

[0063] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (230) based on the execution of the position tracker (271). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using the camera (225) and the sensor (230) (e.g., IMU) based on the execution of the position tracker (271). The position tracker (271) may be referred to as a head tracking (HeT) module.

[0064] For example, the wearable device (101) may be used to construct a three-dimensional virtual space around the wearable device (101) (or a user of the wearable device (101)) based on the execution of the space recognizer (272). The wearable device (101) may reconstruct the three-dimensional surroundings of the wearable device (101) using data acquired using the camera (225) based on the execution of the space recognizer (272). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the three-dimensionally reconstructed surroundings of the wearable device (101) based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module.

[0065] For example, the wearable device (101) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (230) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera (225) based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0066] For example, the wearable device (101) may identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (274). As an example, the wearable device (101) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (274). As an example, the wearable device (101) may identify eye movements of the user based on data acquired using a camera (e.g., an gaze tracking camera (225-1)) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0067] For example, the recognition service layer (270) of the wearable device (101) may further include a face tracker (275) for tracking the user's face. For example, the wearable device (103) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (275). The wearable device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (275). As an example, the wearable device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using the camera (225) based on the execution of the face tracker (275).

[0068] According to one embodiment, the processor (210) of the wearable device (101) may communicate with a plurality of external electronic devices including the wearable device (101) based on the execution of the external object tracker (290) to track an external object. Based on the execution of the external object manager (291), the wearable device (101) may identify whether an object of interest of the user wearing the wearable device (101) exists within the FoV of the user. If the object of interest exists within the FoV, the processor (210) may obtain information indicating the motion of the object of interest, such as the direction of movement, speed, and / or distance of the object of interest, using the camera (225). Based on the information, the processor (210) may determine a score and / or priority associated with the object of interest. If the object of interest does not exist within the FoV, the processor (210) may determine a score and / or priority associated with the object of interest based on the relationship between the FoV and the object of interest and / or a situation recognized by the processor (210).

[0069] According to one embodiment, the processor (210) of the wearable device (101) may establish a communication link with one or more external electronic devices based on the execution of the external electronic device connector (292). Through the communication link, the processor (210) may receive information about an external object from the external electronic device. Through the communication link, the processor (210) may receive a signal from the external electronic device for requesting information about the external object. Based on the signal, the processor (210) may transmit information about the external object requested by the signal to the external electronic device.

[0070] According to one embodiment, the processor (210) of the wearable device (101) may establish a communication link with one or more external electronic devices based on a score and / or priority determined by the external object manager (291) based on the execution of the cluster manager (293). For example, the processor (210) may preferentially establish a communication link with an external electronic device capable of tracking an external object having a relatively high priority. The processor (210) may manage a list of one or more external electronic devices connected to the wearable device (101) based on the execution of the cluster manager (293). The list may be divided into a first list for external electronic devices with which only a communication link is established and which do not exchange information about the external object, and a second list for external electronic devices that share information about the external object through the communication link. Hereinafter, a cluster may refer to a group of a plurality of electronic devices included in the list, including the wearable device (101).

[0071] According to one embodiment, the processor (210) of the wearable device (101) may share information about an external object and / or information about an external object related to an emergency situation based on the execution of the cluster communicator (294). The sharing of the information may be performed between electronic devices included in a cluster including the wearable device (101). Based on the information shared within the cluster, the processor (210) may change a priority and / or a score determined by the external object manager (291) and corresponding to the external object.

[0072] According to one embodiment, the processor (210) of the wearable device (101) may obtain information obtained from the sensor (230) and / or the camera (225) based on the execution of the external object information manager (295). Based on the information, the processor (210) may identify the type (e.g., class and / or category) of the external object. Based on the information, the processor (210) may identify the location of the external object (e.g., the relative location of the external object with respect to the wearable device (101). The information about the external object obtained based on the execution of the external object information manager (295) may include the type and / or the location. The processor (210) may transmit the information obtained based on the execution of the external object information manager (295) to one or more external electronic devices included in a cluster including the wearable device (101) via the communication circuit (235).

[0073] As described above, according to one embodiment, the wearable device (101) may include hardware and / or software for tracking an external object. Hereinafter, with reference to FIGS. 3 and / or 4 , exemplary operations of the wearable device (101) clustered with one or more external electronic devices for tracking an external object are described.

[0074] FIG. 3 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operations of the wearable device described with reference to FIG. 3. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operations of FIG. 3.

[0075] Referring to FIG. 3, in operation (310), a processor of a wearable device according to one embodiment may identify a motion of a user associated with an external object. The processor, which displays a visual object corresponding to the external object on a display (e.g., display (220) of FIG. 2) using a camera (e.g., external camera (225-2) of FIG. 2), may identify a direction of gaze toward the visual object, as indicated by data from a sensor (e.g., sensor (230) and / or gaze tracking camera (225-1) of FIG. 2). Based on the direction of gaze, the processor may identify a motion of operation (310). For example, the motion of operation (310) may include a motion of the user to focus on the external object, such as a gaze of the user looking at the external object and / or a hand gesture of the user pointing at the external object. An exemplary operation of the processor identifying the motion of operation (310) is described with reference to FIG. 5.

[0076] The processor that identifies the motion of the action (310) can use a camera to identify the motion of an external object corresponding to the motion. For example, the processor can obtain information indicating the motion of the external object. The information can include the speed and / or distance of the external object (e.g., the distance between the wearable device and the external object). The information can include a score and / or priority for the external object. The processor can determine the priority corresponding to the external object based on the movement direction, speed, and / or distance of the external object.

[0077] Referring to FIG. 3 , in operation (320), a processor of a wearable device according to one embodiment may form a cluster with one or more external electronic devices and associated external objects in order to track an external object. For example, the processor may identify at least one external electronic device to be connected to the wearable device based on the motion and / or velocity of the external object. The processor may establish a communication link with the identified at least one external electronic device, thereby forming a cluster of operation (320). For example, a cluster may refer to a group of multiple electronic devices that are connected to each other to share information about a specific external object. Connecting electronic devices to each other may refer to establishing a communication link (or communication channel, and / or session) for sharing the information.

[0078] In one embodiment, the processor may form a cluster of operations (320) based on the motion of an external object. The processor may establish a communication link to form the cluster by communicating with at least one external electronic device included in an external space having a size related to the speed of the external object. For example, the processor may form the cluster of operations (320) by communicating with at least one external electronic device included in the external space having a shape based on the speed and / or direction of movement of the external object. The processor may form the cluster of operations (320) based on the location of the external electronic device and / or the FoV of a camera included in the external electronic device. Based on the operation (320), the processor may establish a communication link with at least one external electronic device capable of tracking the external object using a communication circuit.

[0079] To form a cluster of operations (320), the processor of the wearable device may obtain capability information about an external electronic device adjacent to the wearable device. The capability information may include a field of view (e.g., direction, azimuth, and / or angle) of a camera included in the external electronic device and / or the type of the external electronic device. Based on the capability information, the processor may classify the external electronic device into one of the specified types.

[0080] The above-mentioned specified type may include a first type for classifying an AR / VR device, such as the wearable device (101) of FIG. 1. Through the first type of external electronic device, the processor may obtain information about the direction of the external electronic device and / or about at least one external object being tracked by the external electronic device.

[0081] The above-mentioned specified type may include a second type for classifying an electronic device capable of collecting information from other electronic devices, such as a server. Through the external electronic device of the second type, the processor may obtain information about an external object collected by another external electronic device connected to the external electronic device. Through the external electronic device of the second type, the processor may obtain information about an external object collected by the external electronic device based on a specific purpose. For example, a server classified as the second type may include a server for collecting traffic information of a specific location and / or road. The embodiment is not limited thereto, and the server may include an electronic device for collecting and / or processing local information, such as mobile edge computing (MEC).

[0082] The above-mentioned designated type may include a third type for classifying electronic devices that are not classified into the first type and the second type. For example, the third type may include closed-circuit television (CCTV) and / or traffic lights. In order to obtain information about external objects included in the designated category (e.g., a category for classifying external objects related to a dangerous situation), the processor may form a cluster including external electronic devices of the third type.

[0083] A processor forming a cluster of operations (320) may classify external electronic devices included in the cluster into different groups based on whether they share information about external objects. For example, the processor may only establish a communication link with external electronic devices classified as a first group and not share information about the external objects. For example, the processor may classify external electronic devices that share information about external objects into a second group, different from the first group.

[0084] Within the state of forming a cluster of operations (320), the processor of the wearable device may establish a communication link with the external electronic device based on whether the external electronic device can track the external object and / or share information about the external object. For example, if the external electronic device allows access to a camera, microphone, and / or sensor to track the external object, the wearable device may establish a communication link with the external electronic device. In one embodiment, the processor may display, via a display, information related to the cluster formed based on operation (320) (e.g., a list of one or more external electronic devices included in the cluster).

[0085] Referring to FIG. 3 , in operation (330), a processor of a wearable device according to one embodiment may identify whether an external object has moved out of the user's FoV. For example, the processor may identify whether an external object focused on by the user has moved out of the FoV. When the wearable device is worn by the user and the wearable device includes a display covering both eyes of the user (e.g., display (220) of FIG. 2 ), the processor may determine that the external object has moved out of the FoV based on identifying that a visual object corresponding to the external object (e.g., visual object (132) of FIG. 1 ) has moved beyond the periphery of the display due to the movement of the external object relative to the wearable device. Before the external object has moved out of the FoV, or if the external object is located within the FoV (330 - No), the processor may maintain the communication link established based on the cluster of operation (320). If an external object moves outside the FoV (330-yes), the processor may perform action (340).

[0086] Referring to FIG. 3, in operation (340), according to one embodiment, a processor of a wearable device may obtain information about an external object from one or more external electronic devices based on a cluster. For example, if an external object moves outside the user's Field of View (FoV), the processor may initiate obtaining information about the external object from the external electronic device. For example, the processor may request transmission of information about the external object to one or more external electronic devices included in the cluster. Based on the request, the processor may receive a signal including the information from the one or more external electronic devices.

[0087] Referring to FIG. 3, in operation (350), according to one embodiment, the processor of the wearable device may provide at least a portion of the acquired information to the user. For example, after identifying an external object that has moved out of the FoV based on operation (330), the processor may display a visual object related to the external object based on operation (350). The visual object may include text, images, and / or icons representing the external electronic device that provided the information. The visual object may include images and / or videos included in the information. The visual object (150) of FIG. 1 may be displayed by the processor that performed operation (350) of FIG. 3.

[0088] As described above, according to one embodiment, a wearable device can obtain information about an external object located outside the Field of View (FoV) of a user wearing the wearable device from an external electronic device. The wearable device can determine whether to provide the information to the user based on the accuracy and / or risk level of the external object identified by the information. The wearable device can share information about the external object based on a peer-to-peer (P2P) basis using the cluster of operation (320). For example, the wearable device can obtain information about the external object from the external electronic device independently of a centralized server. In the above example, the computational load of the centralized server can be reduced.

[0089] FIG. 4 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operations of the wearable device described with reference to FIG. 4. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operations of FIG. 4.

[0090] Referring to FIG. 4, in operation (410), a processor of a wearable device according to an embodiment may receive first information about an external object from an external electronic device. The processor may receive the first information of operation (410) based on a cluster including the external electronic device and the wearable device. The first information may be broadcast and / or transmitted by the external electronic device that has identified a specific external object classified by risk level (e.g., an external object with a relatively high probability of causing an accident, such as a vehicle).

[0091] Referring to FIG. 4, in operation (415), according to one embodiment, the processor of the wearable device may identify whether the external object of operation (410) is an external object recognized by the wearable device. The processor may identify whether the external object corresponding to the first information is an external object recognized by the wearable device based on an ID (identifier) ​​assigned to the external object included in the first information and / or feature information of the external object. If the external object of operation (410) matches the external object recognized by the wearable device (415 - Yes), the processor may perform operation (420). If the external object of operation (410) is not an external object recognized by the wearable device (415 - No), the processor may perform operation (435).

[0092] Referring to FIG. 4, in operation (420), according to one embodiment, the processor of the wearable device may identify whether second information acquired based on recognition of an external object matches first information. If the external object corresponding to the first information of operation (410) matches the external object recognized by the processor, the second information corresponding to the external object may have already been stored in the memory of the wearable device (e.g., memory (215) of FIG. 2). The processor may compare the second information stored in the memory with the first information, and compare the characteristics of the external object included in the second information with the characteristics of the external object included in the first information. If the second information and the first information match (420—Yes), the processor may perform operation (440). If the second information and the first information do not match (420—No), the processor may perform operation (425).

[0093] Referring to FIG. 4, in operation (425), according to one embodiment, the processor of the wearable device may determine whether the score of the first information is higher than the score of the second information. The score of operation (425) may include a priority assigned to the external object. The score of operation (425) may be related to a risk level for the external object. For example, the score of operation (425) may be related to a probability of an accident occurring due to the external object. The processor may compare the score of the first information and the score of the second information. If the score of the first information is less than or equal to the score of the second information (425 - No), the processor may perform operation (430). If the score of the first information exceeds the score of the second information (425 - Yes), the processor may perform operation (450).

[0094] Referring to FIG. 4, in operation (430), according to one embodiment, a processor of a wearable device may request re-verification of an external object from an external electronic device. The processor may notify the external electronic device of operation (410) that the score of the first information is lower than the score of the second information. If the score relates to a risk level for the external object, the processor may transmit the score of the second information identified by the processor to the external electronic device, thereby requesting the external electronic device to recalculate the risk level for the external object.

[0095] Referring to FIG. 4, in operation (435), according to one embodiment, the processor of the wearable device may identify an external object corresponding to the first information based on object recognition. If it is determined that first information regarding an external object not recognized by the processor has been received based on operation (415), the processor may perform operation (435). The processor may perform a scan of an external space including the wearable device to identify an external object corresponding to the first information of operation (410). The processor, having identified an external object corresponding to the first information, may register the external object in a memory of the wearable device. For example, the processor may store information corresponding to the external object (e.g., at least partially corresponding to the first information) in the memory.

[0096] Referring to FIG. 4, in operation (440), the processor of the wearable device according to one embodiment may identify whether the score of the external object included in the first information is higher than the score of the external object included in the second information. Based on operation (420), the processor that has identified the first information and the second information as matching each other may perform operation (440). If the score included in the first information exceeds the score included in the second information (440 - Yes), the processor may perform operation (445). If the score included in the first information is less than or equal to the score included in the second information (440 - No), the processor may perform operation (450).

[0097] Referring to FIG. 4, in operation (445), according to one embodiment, the processor of the wearable device may increase a score included in the second information based on a score included in the first information. If the score is related to a risk level for an external object and the score included in the first information exceeds the score included in the second information, the processor may increase the score for the external object registered in the wearable device based on the score included in the first information, thereby increasing the risk level for the external object. Based on the increased risk level, the processor may notify and / or warn a user wearing the wearable device of the risk level for the external object.

[0098] Referring to FIG. 4, in operation (450), according to one embodiment, the processor of the wearable device may decrease a score included in the second information. If the score is related to a risk level for an external object, a score included in the first information being lower than a score included in the second information may mean that the risk level of the external object evaluated by the external electronic device is lower than the risk level of the external object evaluated by the processor. Based on the score of the first information, the processor may decrease the score for the external object registered in the wearable device, thereby lowering the risk level for the external object.

[0099] As described above, according to one embodiment, the processor of the wearable device can obtain first information about an external object from an external electronic device using a cluster. Based on the first information, the processor can update second information registered with the wearable device and corresponding to the external object. Based on the update, all electronic devices included in the cluster can obtain synchronized information about a specific external object.

[0100] FIG. 5 illustrates an exemplary operation of a wearable device (101) for identifying a motion of a user (110) in relation to an external object (120). The wearable device (101) of FIG. 1 and / or FIG. 2 can perform the operation described with reference to FIG. 5.

[0101] Referring to FIG. 5, an exemplary state (501) of a wearable device (101) worn by a user (110) is illustrated. Within the state (501), the wearable device (101) may display a screen (130) including an image and / or video of the front view. Within the exemplary state (501) in which an external object (120) is positioned in front of the wearable device (101), a visual object (132) corresponding to the external object (120) may be displayed within the screen (130). A point (s1) of the visual object (132) visible through the screen (130) may correspond to a point (o1) of the external object (120).

[0102] According to one embodiment, the wearable device (101) can detect and / or measure the motion of the user (110) to identify an external object (120) focused on by the user (110). When a plurality of external objects adjacent to the wearable device (101) are identified, the wearable device (101) can identify one external object focused on by the user (110) among the plurality of external objects based on the motion. The motion of the user (110) can be identified using a camera (e.g., the camera (225) and / or the gaze tracking camera (225-1) of FIG. 2), a sensor (e.g., the sensor (230) of FIG. 2), and / or a microphone included in the wearable device (101).

[0103] Referring to FIG. 5, the wearable device (101) can identify a motion related to an external object (120) using a gesture performed by a hand (510). The gesture performed by the hand (510) can be identified based on an image and / or video of a camera included in the wearable device (101) (e.g., an external camera (225-2) of FIG. 2). The embodiment is not limited thereto, and when a user (110) holds an external electronic device such as a remote controller, the wearable device (101) can identify the gesture performed by the hand (510) based on a signal received from the remote controller. Referring to FIG. 5, a wearable device (101) that identifies a pointing gesture of a hand (510) pointing in a direction (h) can identify a motion of a user (110) related to an external object (120) based on whether the external object (120) is positioned in the direction (h). The wearable device (101) that acquires an image and / or video related to the hand (510) of the user (110) can display a screen (130) including a visual object (512) corresponding to the hand (510) based on the VST. Within the screen (130), the wearable device (101) can display a virtual object representing the hand (510).

[0104] Referring to FIG. 5, the wearable device (101) can identify the gaze of the user (110) using images (520) related to the eyes of the user (110). Identifying the gaze of the user (110) may include an operation of calculating the direction of the gaze. Referring to FIG. 5, the wearable device (101) can identify the directions in which the two eyes of the user (110) are directed using an image (521) corresponding to the left eye of the user (110) and / or an image (522) corresponding to the right eye of the user (110). Based on at least one of the directions, the wearable device (101) can identify the gaze of the user (110) toward an external object (120). For example, based on whether an external object (120) is positioned in the direction of the user's (110) gaze, the wearable device (101) may determine that the external object (120) is focused on by the user (110).

[0105] Although an embodiment of identifying an external object (120) focused on by a user (110) based on the direction of the user's (110) hand (510) and gaze has been described, the embodiment is not limited thereto. For example, the wearable device (101) may identify an external object (120) focused on by the user (110) based on the direction of the user's (110) head (e.g., the front of the head where the face is located). For example, the wearable device (101) may identify an external object (120) focused on by the user (110) based on the user's (110) speech. For example, when a user (110) utters a natural language sentence including the name of a specific external object, such as “a basketball flying in the air,” the wearable device (101) can identify the user’s (110) focus on the external object (120) by using the name included in the natural language sentence based on speech-to-text (STT).

[0106] In one embodiment, the wearable device (101) may obtain information about an object of interest from the user (110). For example, the wearable device (101) may obtain information about an object of interest based on receiving an input for setting a priority of the object of interest from the user (110). For example, the wearable device (101) may receive an input for setting a priority of the object of interest through a visual object (530) displayed within the screen (130). Within the visual object (530), the wearable device (101) may display a list of objects of interest that may be adjusted by the user (110). Within the visual object (530), texts representing the objects of interest may be arranged according to the priority adjusted by the user (110). Referring to FIG. 5, within the visual object (530), in a state where the ball is set to have a higher priority than the player, the wearable device (101) can determine an external object (120) classified as a ball among a plurality of external objects as the external object focused on by the user (110).

[0107] As described above, the wearable device (101) that has identified an external object (120) focused on by the user (110) can form a cluster for transmitting and / or receiving information about the external object (120). The cluster can be formed based on establishing a communication link with one or more external electronic devices adjacent to the wearable device (101). The wearable device (101) can request and / or transmit information about the external object (120) through the communication link.

[0108] While forming a cluster with one or more external electronic devices, the information that the wearable device (101) exchanges through the cluster is not limited to an external object (120) focused on by the user (110). For example, the wearable device (101) may provide information about a specific external object that may cause an accident and / or a dangerous situation among a plurality of external objects identified by the wearable device (101), to the one or more external electronic devices. For example, the wearable device (101) that identifies an external object (e.g., a vehicle) moving toward the specific external electronic device (or a user wearing the specific external electronic device) from a blind spot of the specific external electronic device (e.g., a blind spot occurring at an intersection) may provide information about the external object to the specific external electronic device. Based on the provision of the information, the wearable device (101) may warn the specific external electronic device of the approach of the external object. For example, the wearable device (101) can transmit traffic conditions (e.g., traffic congestion) identified by the wearable device (101) to one or more external electronic devices through the cluster.

[0109] Hereinafter, with reference to FIG. 6, an exemplary operation of a wearable device (101) forming a cluster to share information about an external object (120) is described.

[0110] FIG. 6 illustrates an exemplary operation of a wearable device (101) for forming a cluster including one or more external electronic devices. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operation of the wearable device (101) described with reference to FIG. 6. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operation of the wearable device (101) described with reference to FIG. 6.

[0111] Referring to FIG. 6, an exemplary distribution of external electronic devices adjacent to a wearable device (101) is illustrated. A square with letters and / or numbers may correspond to one external electronic device. Hereinafter, the external electronic devices of FIG. 6 may be referenced based on the letters and / or numbers written in the squares. For example, an external electronic device "a" may correspond to an external electronic device depicted as a square with the letter "a" written in it.

[0112] Referring to FIG. 6, while displaying a visual object corresponding to an external object (120), the wearable device (101) can identify the focus of the user (110) on the external object (120) based on the direction of the gaze toward the visual object. The wearable device (101) that has identified the focus can identify the motion of the external object (120) using a camera. In the exemplary state of FIG. 6, the wearable device (101) can identify the motion of the external object (120) moving in one direction from a point (p1). Based on the motion, the wearable device (101) can establish a communication link with external electronic devices adjacent to the point (p1). For example, the wearable device (101) may form a cluster including external electronic devices (e.g., 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, e, h external electronic devices) contained within the external space (610).

[0113] In one embodiment, the wearable device (101) may form a cluster in response to at least one of different conditions. For example, if a specified period of time has passed since the wearable device (101) was worn by the user (110), the wearable device (101) may form a cluster. If the wearable device (101) identifies an external object that moves relatively quickly (e.g., has a speed greater than the speed of the user (110)) among the external objects recognized by the wearable device (101) before identifying the focus of the user (110), the wearable device (101) may form a cluster based on the external object. The wearable device (101) may form a cluster based on a beacon indicating a specific location adjacent to the wearable device (101). The wearable device (101) may form a cluster based on a request from a software application executed by the wearable device (101). Meanwhile, the wearable device (101) may not form a cluster based on certain conditions. For example, if an input to stop forming a cluster is identified from the user (110), and / or if the external space containing the wearable device (101) is a closed space, the wearable device (101) may not form a cluster.

[0114] The size and / or shape of the external space (610) corresponding to the cluster may be related to the motion of the external object (120) identified by the wearable device (101). For example, the external space (610) may have an elliptical shape based on the speed of the external object (120) moving in one direction. The direction of the major axis of the ellipse may correspond to the direction of movement of the external object (120). The length of the major axis of the ellipse may be proportional to the speed of the external object (120). For example, as the speed of the external object (120) increases, the wearable device (101) may increase the size of the external space, which serves as a reference for forming a cluster, such as an external space (630) larger than the external space (610). For example, as the speed of the external object (120) decreases, the wearable device (101) may form a cluster based on an external space having a smaller size than the external space (610).

[0115] After forming a cluster, the wearable device (101) may exclude specific external electronic devices from the cluster and / or add specific external electronic devices to the cluster based on the size and / or shape of the external space (610) that is adjusted based on the motion of the external object (120). For example, if the external object (120) is moved along a direction, the wearable device (101) may change the cluster based on the new location of the moved external object (120). For example, the wearable device (101) may establish a communication link with another external electronic device that is adjacent to the new location of the external object (120) and is different from the external electronic device that was connected based on the cluster, and add the other external electronic device to the cluster.

[0116] Referring to FIG. 6, in an exemplary state connected to external electronic devices included in an external space (610), a wearable device (101) can share information about an external object (120) with external electronic devices adjacent to an external object (120) among the external electronic devices (e.g., external electronic devices 1, 2, 3, and 4 within the external space (620). The wearable device (101) can display the shared information on a display.

[0117] Within the exemplary state of FIG. 6, according to one embodiment, the wearable device (101) may select and / or identify an external electronic device with which a communication link is to be established based on the positional relationship between the FoVs of cameras included in the external electronic devices and the external object (120). For example, within a state of forming a cluster with external electronic devices included in the external space (630), the wearable device (101) may identify external electronic devices f, k, j that are positioned adjacent to each other in the external space (640). The wearable device (101) that has identified external electronic devices f, k, j that are positioned at similar points and / or have FoVs that overlap each other may selectively establish a communication link with any one of the external electronic devices f, k, j.

[0118] In one embodiment, the wearable device (101) may select and / or identify external electronic devices with which a communication link is to be established based on the maximum number of external electronic devices that can be connected simultaneously. The wearable device (101) may identify external objects being tracked by the external electronic devices based on capability information of each of the external electronic devices within the external space (610) corresponding to the cluster. The wearable device (101) may select the external electronic device with which a communication link is to be established based on whether the external object being tracked by each of the external electronic devices matches an external object (120) focused on by the user (110) of the wearable device (101). Based on the capability information, the wearable device (101) may establish a communication link with an external electronic device that is highly likely to track the external object (120). Among the external electronic devices within the external space (610) corresponding to the cluster, the wearable device (101) may not establish a communication link with at least one external electronic device. The at least one external electronic device with which a communication link with the wearable device (101) will not be established may be determined by the maximum value and / or the capability information.

[0119] As described above, according to one embodiment, the wearable device (101) can form a P2P-based cluster independently of a centralized device such as a server. Using the cluster, the wearable device (101) can form a local information sharing network for tracking an external object (120) associated with a user (110). Information about the external object (120) shared within the cluster can be provided to users of electronic devices included in the cluster. For example, users (e.g., spectators and / or referees) who are provided with information about an external object (120) corresponding to a ball can use the information to perform an action related to the external object (120). The wearable device (101) can assist the action by providing information about the external object (120) to an external electronic device using the cluster.

[0120] FIG. 7 illustrates an exemplary operation of a wearable device (101) for forming a cluster including one or more external electronic devices. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operation of the wearable device (101) described with reference to FIG. 7. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform the operation of the wearable device (101) described with reference to FIG. 7.

[0121] Referring to FIG. 7, an exemplary distribution of external electronic devices adjacent to a wearable device (101) is illustrated. When the wearable device (101) identifies a focus of a user (110) on an external object (120), the wearable device (101) can establish a communication link with at least one external electronic device included in an external space (710) formed based on the motion of the external object (120), thereby forming a cluster.

[0122] Referring to the exemplary case of FIG. 7, a plurality of electronic devices including a wearable device (101) may be connected to at least one of servers (721, 722, 723, 724). For example, the wearable device (101), three external electronic devices, and a fourth external electronic device may be connected to a first server (721). The first server (721) may obtain information about one or more external objects tracked by each device from each of the wearable device (101), the third external electronic device, and the fourth external electronic device. For example, the third external electronic device and the eighth external electronic device may be connected to a second server (722). For example, the seventh external electronic device and the eighth external electronic device may be connected to a third server (723). For example, the fifth external electronic device and the sixth external electronic device may be connected to a fourth server (724). Each of the first server (721) to the fourth server (724) can manage information about all external objects tracked by connected electronic devices.

[0123] Within the exemplary state of FIG. 7, servers (721, 722, 723, and 724) may be configured to communicate with one or more electronic devices located at a specific location (e.g., a sports facility such as a baseball stadium) to provide services related to the specific location. For example, the first server (721) may be configured to provide services related to sports activities involving an external object (120). Information about the external object (120) acquired by the wearable device (101) may be provided to the first server (721) and utilized for video review (video assistant referee, VAR).

[0124] Within the exemplary state of FIG. 7, in order to form a cluster with external electronic devices included in the external space (710), the wearable device (101) may communicate with a server connected to at least one of the external electronic devices. For example, instead of establishing a communication link with the 7 external electronic devices and / or the e external electronic device, the wearable device (101) may communicate with a third server (723) connected to the 7 external electronic devices and the e external electronic device. Since the third server (723) has information about external objects tracked by the 7 external electronic devices and the e external electronic device, the wearable device (101) may communicate with the third server (723) to obtain all information about external objects tracked by the 7 external electronic devices and the e external electronic device. Similarly, the wearable device (101) may communicate with a fourth server (724) instead of the 5 external electronic devices and the 6 external electronic devices. Similarly, the wearable device (101) can form a cluster by communicating with a second server (722) instead of three external electronic devices and eight external electronic devices.

[0125] Within the exemplary state of FIG. 7, information about an external object (120) can be shared by servers (721, 722, 723, 724) connected to a cluster formed based on an external space (710) and / or external electronic devices included in the cluster. For example, the wearable device (101) can provide information about the external object (120) to a third server (723), which can then (indirectly) transmit the information to the 7 external electronic devices and / or e external electronic devices connected to the third server (723). Similarly, the wearable device (101) can (indirectly) obtain information about an external object tracked by the 5 external electronic devices through the 4th server (724).

[0126] After forming a cluster based on the operations described above with reference to FIGS. 6 and / or 7, the wearable device (101) may disconnect and / or unlink communication links established with external electronic devices included in the cluster based on movement of the external object (120). For example, if the distance between any one external electronic device included in the cluster and the external object (120) increases, and / or if any one external electronic device included in the cluster changes to a state where it cannot track the external object (120), the wearable device (101) may exclude the external electronic device from the cluster. Excluding the external electronic device from the cluster may include an operation in which the communication link between the external electronic device and the wearable device (101) is released, and / or the external electronic device is disconnected or unlinked from the communication link established for the cluster.

[0127] In one embodiment, the wearable device (101) may assign a priority and / or a score to each of the external electronic devices included in the cluster, the priority and / or score indicating whether each of the external electronic devices can track the external object (120). Based on the score assigned to each of the external electronic devices, the wearable device (101) may determine whether to exclude each of the external electronic devices from the cluster. The priority may be determined based on a probability that a user of the external electronic device will focus on the external object (120). The priority may be determined based on a similarity between a preference of a user of the external electronic device and a preference of a user (110) of the wearable device (101). The priority may be related to whether the external electronic device can track the external object (120) and / or can provide more accurate information about the external object (120). For example, if an external electronic device is moved at a relatively fast speed, the wearable device (101) may determine a priority for the external electronic device (e.g., an external electronic device included in a vehicle and / or airplane) that can be moved at a relatively fast speed as a higher priority than the priority for other external electronic devices.

[0128] Hereinafter, with reference to FIG. 8 and / or FIG. 9, a UI displayed by electronic devices included in a cluster (e.g., a wearable device (101)) is exemplarily described.

[0129] FIG. 8 illustrates an example of a screen (130) displayed by a wearable device (101) that communicates with an external electronic device. The wearable device (101) of FIG. 1 and / or FIG. 2 can perform the operation of the wearable device (101) described with reference to FIG. 8. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device (101) described with reference to FIG. 8.

[0130] Referring to FIG. 8, a first wearable device (101-1) and a second wearable device (101-2) included in an exemplary location such as an aquarium are illustrated. The first wearable device (101-1) positioned at a point (p1) can display a screen (130) based on VST. Through the screen (130), a first user (110-1) wearing the first wearable device (101-1) can view a video of captured fish (810, 812, 814, 816) in real time. The first wearable device (101-1) can display visual objects (820, 822, 824, 826) corresponding to each of the fish (810, 812, 814, 816) within the screen (130).

[0131] In the exemplary case of FIG. 8, the first wearable device (101-1) can identify the gaze of the first user (110-1) wearing the first wearable device (101-1). For example, the first wearable device (101-1) can identify the focus of the first user (110-1) on a fish (810) corresponding to the visual object (820) based on identifying the gaze directed toward the visual object (820) within the screen (130). The first wearable device (101-1) that has identified the focus can form a cluster for tracking the fish (810). To form the cluster, the wearable device (101) can establish a communication link (805) with the second wearable device (101-2) based on the motion of the fish (810). For example, a wearable device (101) can establish a communication link (805) with a second wearable device (101-2) located at a point (p2) adjacent to a fish (810) located at a point (o1).

[0132] Within the state where the communication link (805) is established, the first wearable device (101-1) may determine whether to acquire information about the fish (810) through the communication link (805) based on the position and / or size of the visual object (820) within the screen (130) corresponding to the fish (810) focused on by the first user (110-1). For example, if the visual object (820) moves out of the boundary of the screen (130) based on the movement of the fish (810), the wearable device (101) may request the second wearable device (101-2) to transmit information about the fish (810) through the communication link (805). In the exemplary screen (130) of FIG. 8, if a visual object (820) corresponding to a fish (810) is displayed in a size smaller than a specified size, the wearable device (101) may request the second wearable device (101-2) to transmit information about the fish (810) through the communication link (805).

[0133] Referring to FIG. 8, in a state where a signal requesting transmission of information about a fish (810) is received from the first wearable device (101-1), the second wearable device (101-2) can transmit information related to the fish (810) to the first wearable device (101-1). For example, the second wearable device (101-2) can transmit an image and / or video related to the fish (810) to the first wearable device (101-1). The video can include a video being acquired in real time by the second wearable device (101-2). In response to acquiring information related to the fish (810) from the second wearable device (101-2), the first wearable device (101-1) can display a visual object (830) based on the information on the screen (130).

[0134] Referring to FIG. 8, within a visual object (830) based on information acquired from a second wearable device (101-2), a first wearable device (101-1) may display text (e.g., “Taken by User B”) indicating a second wearable device (101-2) that provided the information. The embodiment is not limited thereto, and the first wearable device (101-1) may display information about a fish (810) indicated by the information (e.g., the name and / or type of the fish) within the visual object (830). The first wearable device (101-1) may display a video (832) identified by the second wearable device (101-2) within the visual object (830). In response to a user input related to the above video (832), the first wearable device (101-1) may display the video provided from the second wearable device (101-2) on at least a portion of the screen (130).

[0135] Within the exemplary case of FIG. 8, without tags and / or external cameras for the fishes (810, 812, 814, 816), the first wearable device (101-1) can acquire images and / or videos (832) that cannot be acquired by the camera of the first wearable device (101-1) from the second wearable device (101-2) adjacent to the fish (810). While exemplary operations of the first wearable device (101-1) for forming a cluster based on an external object (e.g., a fish (810)) focused on by a first user (110-1) wearing the first wearable device (101-1) and acquiring information from the cluster based on the movement of the external object have been described, the embodiment is not limited thereto. Below, with reference to FIG. 9, the operation of a wearable device (101) that receives information from another external electronic device through a cluster is described.

[0136] FIG. 9 illustrates an example of a screen (130) displayed by a wearable device (101) that communicates with an external electronic device. The wearable device (101) of FIG. 1 and / or FIG. 2 can perform the operation of the wearable device (101) described with reference to FIG. 9. For example, the wearable device (101) and / or the processor (210) of FIG. 2 can perform the operation of the wearable device (101) described with reference to FIG. 9.

[0137] Referring to FIG. 9, a first wearable device (101-1) and a second wearable device (101-2) positioned at a location such as an intersection and / or a road are illustrated. The second wearable device (101-2) can identify an external object (910), such as a vehicle, using a camera (e.g., the camera (225) and / or the external camera (225-2) of FIG. 2). In one embodiment, the second wearable device (101-2) can determine whether to share information related to the external object (910) with another external electronic device (e.g., the first wearable device (101-1)) based on a priority of the external object (910) included in a result of recognizing the external object (910).

[0138] The priority of the external object (910), determined by the second wearable device (101-2), may be related to one or more criteria. For example, the second wearable device (101-2) may determine the priority of the external object (910) based on the time and / or frequency with which the gaze of the second user (110-2) wearing the second wearable device (101-2) is directed toward the external object (910). As the time and / or frequency increase, the priority corresponding to the external object (910) may increase. For example, the second wearable device (101-2) may determine the priority of the external object (910) based on the category of the external object (910). If an external object (910) is classified into a designated category (e.g., a vehicle) for classifying a risk object, the priority corresponding to the external object (910) may be increased.

[0139] For example, the second wearable device (101-2) may determine the priority of the external object (910) based on the speed and / or distance of the external object (910). As the speed of the external object (910) increases and / or the distance between the external object (910) and the second wearable device (101-2) decreases, the priority corresponding to the external object (910) may increase. For example, the second wearable device (101-2) may determine the priority of the external object (910) based on the moving direction of the external object (910). If the moving direction of the external object (910) is opposite to the moving direction of the second user (110-2) wearing the second wearable device (101-2), the priority corresponding to the external object (910) may increase.

[0140] For example, the second wearable device (101-2) may determine the priority of the external object (910) based on the temperature of the external object (910). If the temperature of the external object (910) falls within a temperature range indicating an abnormal condition (e.g., a temperature range for identifying a fire), the priority corresponding to the external object (910) may be increased. For example, the second wearable device (101-2) may determine the priority of the external object (910) based on a sound generated from the external object (910). If a loud sound exceeding a threshold value is generated from the external object (910), the priority corresponding to the external object (910) may be increased.

[0141] Although an operation is described in which a priority for an external object (910) is determined by a specific wearable device (in the exemplary case of FIG. 9, the second wearable device (101-2)), the embodiment is not limited thereto. For example, based on the number of electronic devices that have identified a focus on the external object (910), the priority corresponding to the external object (910) may be proportionally increased. The number of electronic devices that have identified a focus on the external object (910) may be identified by a server connected to the electronic devices (e.g., the first server (721) to the fourth server (724) of FIG. 7) and / or electronic devices connected via a communication link (920).

[0142] Based on the priority (or score) of the external object (910) determined based on the above-described criteria, the second wearable device (101-2) can determine whether to share information corresponding to the external object (910) via the communication link (920). The second wearable device (101-2), which has identified the external object (910) as a vehicle heading in the direction (d1), can share information about the external object (910) with the first wearable device (101-1) located in the direction (d1).

[0143] In one embodiment, the first wearable device (101-1) may request the cluster formed via the communication link (920) to transmit information about an external object included in the blind spot of the first wearable device (101-1). The second wearable device (101-2), which has identified the request, may transmit information about the external object (910) to the first wearable device (101-1) based on identifying the external object (910) moving toward the first wearable device (101-1) along the direction (d1). The above request may be performed by a first user (110-1) wearing a first wearable device (101-1), and / or may be performed in response to a motion of the first user (110-1) identified by the first wearable device (101-1) (e.g., a motion of the first user (110-1) repeatedly looking at a blind spot).

[0144] Referring to FIG. 9, in a state where information about an external object (910) is shared by a second wearable device (101-2), the first wearable device (101-1) may display a screen (130) including a visual object (e.g., at least one of the visual objects (930, 940)) related to the external object (910) based on the information shared via the communication link (920). For example, the first wearable device (101-1) may display a video (932) captured by the second wearable device (101-2) as the visual object (930). Together with the video (932), the first wearable device (101-1) may display text (e.g., “Vehicle approaching!”) to warn of the approach of the external object (910). Along with the video (932), the first wearable device (101-1) may display text indicating the external electronic device that provided the video (932), e.g., “Taken by User B.”

[0145] The embodiment is not limited thereto, and the first wearable device (101-1) may use a visual object (940) including a map to visualize the location of an external object (910) identified by the second wearable device (101-2). For example, within the visual object (940), the first wearable device (101-1) may display an indicator (944) for indicating the location of an external object (910) shared by the second wearable device (101-2) together with an indicator (942) for indicating the location of the first wearable device (101-1). Referring to FIG. 9, the indicator (944) may include a character (e.g., an exclamation mark) for indicating a dangerous object. Referring to FIG. 9, based on information provided from the second wearable device (101-2) that identifies an external object (910) moving along the direction (d1), the first wearable device (101-1) can display an arrow corresponding to the direction (d1) and extending from the indicator (944).

[0146] Although exemplary operations of the first wearable device (101-1) for displaying information about an external object (910) based on at least one of the visual objects (930, 940) have been described, the embodiments are not limited thereto. For example, the first wearable device (101-1) may notify the approach of the external object (910) based on a vibration notification. The vibration notification may be output based on the control of a haptic actuator (e.g., a vibration motor) included in the first wearable device (101-1). For example, the first wearable device (101-1) may notify the approach of the external object (910) to the first user (110-1) by outputting an audio signal including an utterance (e.g., an utterance such as "a vehicle is approaching"). To output the above audio signal, the first wearable device (101-1) can execute a software application for text-to-speech (TTS). The first wearable device (101-1) can use a visual object (e.g., an image and / or icon in the form of an arrow) displayed on the screen (130) to indicate the direction in which an external object (910) is approaching the first wearable device (101-1). Based on the direction, the first wearable device (101-1) can guide the first user (110-1) wearing the first wearable device (101-1) to look in the direction toward the external object (910).

[0147] FIG. 10 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operations of the wearable device described with reference to FIG. 10. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform at least one of the operations of FIG. 10. The operations of the wearable device described with reference to FIG. 10 may be related to operation (320) of FIG. 3 for forming a cluster.

[0148] Referring to FIG. 10 , in operation (1010), according to one embodiment, a processor of a wearable device may scan external electronic devices and / or servers associated with locations adjacent to the wearable device. For example, the processor may identify external electronic devices and / or servers connected to the external electronic devices located at locations adjacent to the wearable device. The scan of operation (1010) may be performed prior to forming a cluster based on the wearable device and / or to modify a formed cluster.

[0149] Referring to FIG. 10 , in operation 1020, according to one embodiment, a processor of a wearable device may connect to an external electronic device and / or server based on a location associated with the repeatedly scanned external electronic device and / or server. Repeatedly scanning a specific external electronic device and / or server may mean that the specific external electronic device and / or server is positioned adjacent to the wearable device. Based on operation 1020, the wearable device may establish a communication link for a cluster with the external electronic device and / or server.

[0150] Referring to FIG. 10, in operation (1030), according to one embodiment, the processor of the wearable device may change an external electronic device and / or server connected to the wearable device based on a priority (or score) corresponding to the external object. The wearable device may change the priority (or score) of the external object based on the motion of the external object. For example, if the wearable device identifies a user's focus on another external object different from the external object, the wearable device may decrease the priority of the external object. Based on the decreased priority, the wearable device may disconnect the connected external electronic device and / or server based on operation (1020) or re-perform the scan of operation (1010).

[0151] FIG. 11 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operations of the wearable device described with reference to FIG. 10. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform at least one of the operations of FIG. 10.

[0152] Referring to FIG. 11, in operation (1110), according to one embodiment, a processor of a wearable device may receive a first signal for requesting information about an external object from an external electronic device. The first signal may be received via a communication link for a cluster formed with the external electronic device.

[0153] Referring to FIG. 11, in operation (1120), a processor of a wearable device according to an embodiment may identify an external object corresponding to a first signal. Based on receiving the first signal of operation (1110), the processor may perform operation (1120). The processor may search for an external electronic device specified by the first signal using a camera (e.g., camera (225) and / or external camera (225-2) of FIG. 2). The processor may identify the external object of operation (1120) within a list of external objects tracked using a camera and / or a sensor of the wearable device including the processor.

[0154] Referring to FIG. 11, in operation (1130), a processor of a wearable device according to an embodiment may transmit a second signal including information about an external object to an external electronic device. If the processor fails to identify the external object in operation (1120), the processor may not transmit a response to the first signal. If the processor identifies the external object, the processor may perform operation (1130). In operation (1130), the processor may transmit a second signal of operation (1130) to the external electronic device that transmitted the first signal of operation (1110) as a response to the first signal via a communication link for the cluster.

[0155] FIG. 12 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operations of the wearable device described with reference to FIG. 10. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform at least one of the operations of FIG. 10.

[0156] Referring to FIG. 12, in operation (1210), according to one embodiment, a processor of a wearable device may identify an external object. The processor may use a camera (e.g., camera (225) and / or external camera (225-2) of FIG. 2) to identify an external object adjacent to the wearable device. Having identified the external object, the processor may perform operation (1220).

[0157] Referring to FIG. 12, in operation (1220), according to one embodiment, a processor of a wearable device may acquire a category corresponding to an external object from among designated categories. The processor may perform object recognition on the external object and classify the external object into one of the designated categories. The designated categories may be distinguished based on whether information about the external object is to be shared and / or whether a cluster related to the external object is to be formed. The designated categories may be distinguished based on the type of the external object.

[0158] Referring to FIG. 12 , in operation (1230), a processor of a wearable device according to an embodiment may transmit information about an external object to one or more external electronic devices based on whether the external object corresponds to a designated category for classifying a dangerous object. The designated category may be configured to classify a designated type of external object, such as a vehicle. For example, a processor that has identified an external object included in a designated category, such as an external object (910) of FIG. 9 , may transmit information about the external object to the external electronic device based on operation (1230). The one or more external electronic devices of operation (1230) may be connected to the wearable device to form a cluster. The processor may notify the one or more external electronic devices that an external object included in the designated category of operation (1230) has been identified.

[0159] FIG. 13 illustrates an example flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 may perform the operations of the wearable device described with reference to FIG. 10. For example, the wearable device (101) and / or the processor (210) of FIG. 2 may perform at least one of the operations of FIG. 10.

[0160] Referring to FIG. 13, in operation (1310), according to one embodiment, a processor of a wearable device may connect to a server adjacent to the wearable device. The server of operation (1310) may be provided to collect information about external objects from one or more electronic devices, such as the first server (721) to the fourth server (724) of FIG. 7. Based on operation (1310), the processor may establish a communication link with a server that includes a point of the wearable device or corresponds to an external space adjacent to a point of the wearable device. Based on operation (1310), the processor may form a cluster including the server of operation (1310).

[0161] Referring to FIG. 13, in operation (1320), according to one embodiment, the processor of a wearable device may obtain information about one or more external objects registered with a server. The server may store information about the external objects received from external electronic devices connected to the server. The processor may obtain the information stored in the server based on operation (1320).

[0162] Referring to FIG. 13, in operation (1330), the processor of the wearable device according to one embodiment may at least partially update information acquired using the camera. For example, the processor may compare information acquired using the camera with information acquired based on operation (1320) and update the information acquired based on operation (1320).

[0163] Referring to FIG. 13, in operation (1340), according to one embodiment, the processor of the wearable device may transmit updated information to the server based on operation (1330). Based on operation (1340), information stored in the server may be updated by the processor of the wearable device. When the information stored in the server is updated, the updated information may be shared with one or more external electronic devices connected to the server.

[0164] Referring to FIG. 13, in operation (1350), according to one embodiment, the processor of the wearable device may change the server connected to the wearable device by comparing the location of the wearable device with the location associated with the server. After connecting to the server based on operation (1310), the processor may re-establish the communication link between the wearable device and the server based on movement of the wearable device including the processor and / or a change in the location associated with the server. For example, if the wearable device is moved beyond a specified distance or moved out of an external space associated with the server, the processor may disconnect the communication link between the wearable device and the server based on operation (1310) and identify a new server.

[0165] FIG. 14 illustrates an example of a signal flow diagram between a wearable device (101) and an external electronic device (1405), according to one embodiment. The operation of the wearable device (101) described with reference to FIG. 14 may be performed by the wearable device (101) and / or the processor (210) of FIG. 2. The external electronic device (1405) of FIG. 14 may include another wearable device and / or a server connected to the wearable device (101).

[0166] Referring to FIG. 14, in operation (1410), according to one embodiment, a processor of a wearable device (101) may identify an external object. The processor may control a camera of the wearable device to acquire an image. The image may be displayed on a display of the wearable device to provide a user experience based on VST. The processor may perform operation (1410) based on at least one of the operations of FIG. 12. Operation (1410) may be performed based on the execution of the external object manager (291) of FIG. 2.

[0167] Referring to FIG. 14, in operation (1420), according to one embodiment, a processor of a wearable device (101) may form a cluster with one or more external electronic devices (e.g., external electronic device (1405) of FIG. 14). The operation of the processor forming the cluster may be related to at least one of the operations of FIG. 3. The cluster of operation (1420) may be related to a position and / or motion of an external object identified by a direction of a gaze of a user wearing the wearable device (101) within the image of operation (1410). The processor may identify the direction of the gaze using a camera of the wearable device (e.g., gaze tracking camera (225-1) of FIG. 2).

[0168] Based on the cluster formed based on the operation (1420), the external electronic device (1405) can identify an external object focused on by the user of the wearable device (101). In operation (1425), the processor of the external electronic device (1405) can obtain information about the external object. After forming the cluster, the processor of the external electronic device (1405) can monitor the location and / or motion of the external object. The processor of the external electronic device (1405) can refrain from transmitting the location and / or motion of the external object monitored by the external electronic device (1405) until it receives a signal from the wearable device (101) to request information about the external object.

[0169] Referring to FIG. 14, in operation (1430), according to one embodiment, the processor of the wearable device (101) may request information about an external object from one or more external electronic devices based on the location of the external object. For example, if the external object has moved out of an image acquired from a camera of the wearable device (101), the processor may perform operation (1430). Based on operation (1430), the wearable device (101) may transmit a signal to an external electronic device (1405) included in the cluster of operation (1420) to request information about the external object.

[0170] Referring to FIG. 14, the processor of the external electronic device (1405) that received the signal may, based on operation (1435), transmit information about the external object in response to a request included in the signal. The information may include an image, video, and / or sound of the external object identified by the processor of the external electronic device (1405).

[0171] Referring to FIG. 14, in operation (1440), according to one embodiment, the processor of the wearable device (101) may provide at least a portion of information transmitted from an external electronic device (1405) to a user wearing the wearable device (101). For example, the wearable device (101) may display a visual object (e.g., a visual object (830) of FIG. 8) based on the information on the display. For example, the wearable device (101) may output an audio signal related to the information to the user through a speaker. The audio signal may include a natural language sentence indicating a location and / or direction of an external object identified by the external electronic device (1405). For example, the wearable device (101) may output a vibration notification based on the information to the user by using a haptic actuator (or vibration motor).

[0172] Hereinafter, exemplary appearances of the wearable devices described with reference to FIGS. 1 to 13 are illustrated with reference to FIGS. 15A, 15B, 16A and / or 16B. The wearable device (1500) of FIG. 15A and / or 15B and / or the wearable device (1600) of FIG. 16A and / or 16B may be an example of the wearable device (101) of FIG. 1.

[0173] FIG. 15A illustrates an example of a perspective view of a wearable device, according to one embodiment. According to one embodiment, the wearable device (1500) may have the form of glasses that are wearable on a body part of a user (e.g., the head). The wearable device (1500) may include a head-mounted display (HMD). For example, the housing of the wearable device (1500) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (1500) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0174] Referring to FIG. 15A, according to one embodiment, a wearable device (1500) may include at least one display (1550) and a frame (1500) supporting at least one display (1550).

[0175] According to one embodiment, a wearable device (1500) can be worn on a part of a user's body. The wearable device (1500) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (1500). For example, the wearable device (1500) can display a virtual reality image provided from at least one optical device (1582, 1584) of FIG. 15B on at least one display (1550) in response to a user's designated gesture acquired through the motion recognition cameras (1560-2, 1560-3) of FIG. 15B.

[0176] According to one embodiment, at least one display (1550) may provide visual information to a user. For example, at least one display (1550) may include a transparent or translucent lens. At least one display (1550) may include a first display (1550-1) and / or a second display (1550-2) spaced apart from the first display (1550-1). For example, the first display (1550-1) and the second display (1550-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0177] Referring to FIG. 15B, at least one display (1550) can provide visual information transmitted from external light to a user through a lens included in the at least one display (1550), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (1550) can include a first surface (1531) and a second surface (1532) opposite to the first surface (1531). A display area can be formed on the second surface (1532) of the at least one display (1550). When a user wears the wearable device (1500), external light can be transmitted to the user by being incident on the first surface (1531) and transmitted through the second surface (1532). As another example, at least one display (1550) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1582, 1584) on a real screen transmitted through external light, in a display area formed on the second surface (1532).

[0178] In one embodiment, at least one display (1550) may include at least one waveguide (1533, 1534) that diffracts light emitted from at least one optical device (1582, 1584) and transmits the diffracted light to a user. The at least one waveguide (1533, 1534) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1533, 1534). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1533, 1534) may be propagated to the other end of the at least one waveguide (1533, 1534) by the nano-pattern. At least one waveguide (1533, 1534) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)), or at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1533, 1534) may be arranged within the wearable device (1500) to guide a screen displayed by at least one display (1550) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1533, 1534).

[0179] The wearable device (1500) can analyze an object included in a real image collected through a shooting camera (1560-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1550). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (1500) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (1500) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (1500) can view an image displayed on at least one display (1550).

[0180] According to one embodiment, the frame (1500) may be configured as a physical structure that allows the wearable device (1500) to be worn on the user's body. According to one embodiment, the frame (1500) may be configured so that, when the user wears the wearable device (1500), the first display (1550-1) and the second display (1550-2) can be positioned corresponding to the user's left and right eyes. The frame (1500) may support at least one display (1550). For example, the frame (1500) may support the first display (1550-1) and the second display (1550-2) to be positioned corresponding to the user's left and right eyes.

[0181] Referring to FIG. 15A, the frame (1500) may include a region (1520) that at least partially contacts a portion of the user's body when the user wears the wearable device (1500). For example, the region (1520) of the frame (1500) that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1500) makes contact with. According to one embodiment, the frame (1500) may include a nose pad (1510) that contacts a portion of the user's body. When the wearable device (1500) is worn by the user, the nose pad (1510) may contact a portion of the user's nose. The frame (1500) may include a first temple (1504) and a second temple (1505) that contact another part of the user's body that is distinct from the part of the user's body.

[0182] For example, the frame (1500) may include a first rim (1501) that surrounds at least a portion of the first display (1550-1), a second rim (1502) that surrounds at least a portion of the second display (1550-2), a bridge (1503) that is disposed between the first rim (1501) and the second rim (1502), a first pad (1511) that is disposed along a portion of the edge of the first rim (1501) from one end of the bridge (1503), a second pad (1512) that is disposed along a portion of the edge of the second rim (1502) from the other end of the bridge (1503), a first temple (1504) that extends from the first rim (1501) and is fixed to a portion of the wearer's ear, and a second temple (1505) that extends from the second rim (1502) and is fixed to a portion of the ear opposite the ear. There are. The first pad (1511) and the second pad (1512) can be in contact with a part of the user's nose, and the first temple (1504) and the second temple (1505) can be in contact with a part of the user's face and a part of the user's ear. The temples (1504, 1505) can be rotatably connected to the rim through the hinge units (1506, 1507) of FIG. 15B. The first temple (1504) can be rotatably connected to the first rim (1501) through the first hinge unit (1506) disposed between the first rim (1501) and the first temple (1504). The second temple (1505) may be rotatably connected to the second rim (1502) via a second hinge unit (1507) disposed between the second rim (1502) and the second temple (1505). In one embodiment, the wearable device (1500) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame (1500) to identify an external object (e.g., a user's fingertip) touching the frame (1500) and / or a gesture performed by the external object.

[0183] According to one embodiment, the wearable device (1500) may include hardwares that perform various functions (e.g., hardwares to be described later based on the block diagram of FIG. 2). For example, the hardwares may include a battery module (1570), an antenna module (1575), at least one optical device (1582, 1584), speakers (e.g., speakers 1555-1, 1555-2), a microphone (e.g., microphones 1565-1, 1565-2, 1565-3), a light-emitting module, and / or a printed circuit board (PCB) (1590) (e.g., a printed circuit board). The various hardwares may be arranged within the frame (1500).

[0184] According to one embodiment, microphones (e.g., microphones 1565-1, 1565-2, 1565-3) of the wearable device (1500) may be disposed on at least a portion of the frame (1500) to acquire sound signals. A first microphone (1565-1) disposed on the bridge (1503), a second microphone (1565-2) disposed on the second rim (1502), and a third microphone (1565-3) disposed on the first rim (1501) are illustrated in FIG. 15B , but the number and arrangement of the microphones (1565) are not limited to the embodiment of FIG. 15B . When the number of microphones (1565) included in the wearable device (1500) is two or more, the wearable device (1500) can identify the direction of a sound signal by using a plurality of microphones arranged on different parts of the frame (1500).

[0185] According to one embodiment, at least one optical device (1582, 1584) may project a virtual object onto at least one display (1550) to provide various image information to a user. For example, at least one optical device (1582, 1584) may be a projector. At least one optical device (1582, 1584) may be disposed adjacent to at least one display (1550) or may be included within at least one display (1550) as a part of at least one display (1550). According to one embodiment, the wearable device (1500) may include a first optical device (1582) corresponding to a first display (1550-1) and a second optical device (1584) corresponding to a second display (1550-2). For example, at least one optical device (1582, 1584) may include a first optical device (1582) disposed at an edge of a first display (1550-1) and a second optical device (1584) disposed at an edge of a second display (1550-2). The first optical device (1582) may transmit light to a first waveguide (1533) disposed on the first display (1550-1), and the second optical device (1584) may transmit light to a second waveguide (1534) disposed on the second display (1550-2).

[0186] In one embodiment, the camera (1560) may include a recording camera (1560-4), an eye tracking camera (ET CAM) (1560-1), and / or a motion recognition camera (1560-2, 1560-3). The recording camera (1560-4), the eye tracking camera (1560-1), and the motion recognition cameras (1560-2, 1560-3) may be positioned at different locations on the frame (1500) and may perform different functions. The eye tracking camera (1560-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (1500). For example, the wearable device (1500) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1560-1).

[0187] The wearable device (1500) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1560-1). The wearable device (1500) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (1500) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1560-1). The wearable device (1500) can render an image (or screen) displayed on at least one display (1550) based on the position of the user's eyes.

[0188] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI) of a second region distinct from the first region may be different from each other. The wearable device (1500) may obtain an image having the visual quality of the first region matching the user's gaze and the visual quality of the second region using foveated rendering. For example, if the wearable device (1500) supports an iris recognition function, user authentication may be performed based on iris information obtained using a gaze tracking camera (1560-1). Although an example in which the gaze tracking camera (1560-1) is positioned toward the user's right eye is illustrated in FIG. 15B, the embodiment is not limited thereto, and the gaze tracking camera (1560-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0189] In one embodiment, the capturing camera (1560-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1560-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1560-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1550). The at least one display (1550) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1560-4) and a virtual image provided through at least one optical device (1582, 1584) are superimposed. The wearable device (1500) can compensate for depth information (e.g., the distance between the wearable device (1500) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1560-4). The wearable device (1500) can perform object recognition using an image acquired using the capture camera (1560-4). The wearable device (1500) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1560-4). The wearable device (1500) can perform a pass-through function to display an image acquired through the capture camera (1560-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1550). In one embodiment, the capturing camera (1560-4) may be positioned on a bridge (1503) positioned between the first rim (1501) and the second rim (1502).

[0190] The gaze tracking camera (1560-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (1500) and matching the user's gaze with visual information provided to at least one display (1550). For example, when the wearable device (1500) looks straight ahead, the wearable device (1500) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1550). The gaze tracking camera (1560-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1560-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1560-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1560-1) may be positioned within the first rim (1501) and / or the second rim (1502) to face the direction in which the user wearing the wearable device (1500) is positioned.

[0191] The gesture recognition camera (1560-2, 1560-3) can provide a specific event on a screen provided on at least one display (1550) by recognizing the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the user's body. The gesture recognition camera (1560-2, 1560-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1550). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (1560-2, 1560-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1560-2, 1560-3). In one embodiment, the motion recognition cameras (1560-2, 1560-3) may be positioned on the first rim (1501) and / or the second rim (1502).

[0192] The camera (1560) included in the wearable device (1500) is not limited to the above-described gaze tracking camera (1560-1) and motion recognition cameras (1560-2, 1560-3). For example, the wearable device (1500) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (1500) can identify an external object based on a sensor for identifying the distance between the wearable device (1500) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1560) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1500) may include a camera (1560) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1500).

[0193] Although not shown, in one embodiment, the wearable device (1500) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (1560). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame (1500) and the hinge units (1506, 1507).

[0194] According to one embodiment, the battery module (1570) may supply power to the electronic components of the wearable device (1500). In one embodiment, the battery module (1570) may be disposed within the first temple (1504) and / or the second temple (1505). For example, the battery module (1570) may be a plurality of battery modules (1570). The plurality of battery modules (1570) may be disposed within each of the first temple (1504) and the second temple (1505). In one embodiment, the battery module (1570) may be disposed at an end of the first temple (1504) and / or the second temple (1505).

[0195] The antenna module (1575) can transmit signals or power to the outside of the wearable device (1500), or receive signals or power from the outside. In one embodiment, the antenna module (1575) can be positioned within the first temple (1504) and / or the second temple (1505). For example, the antenna module (1575) can be positioned close to one surface of the first temple (1504) and / or the second temple (1505).

[0196] The speaker (1555) can output an audio signal to the outside of the wearable device (1500). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1555) may be positioned within the first temple (1504) and / or the second temple (1505) so as to be positioned adjacent to the ear of a user wearing the wearable device (1500). For example, the speaker (1555) may include a second speaker (1555-2) positioned within the first temple (1504) and thus adjacent to the user's left ear, and a first speaker (1555-1) positioned within the second temple (1505) and thus adjacent to the user's right ear.

[0197] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (1500) to the user. For example, when the wearable device (1500) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1501) and / or the second rim (1502).

[0198] Referring to FIG. 15B, according to one embodiment, a wearable device (1500) may include a printed circuit board (PCB) (1590). The PCB (1590) may be included in at least one of the first temple (1504) or the second temple (1505). The PCB (1590) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (1500) (e.g., hardwares illustrated by different blocks in FIG. 4A) may be positioned on the PCB (1590). The wearable device (1500) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0199] According to one embodiment, a wearable device (1500) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1500) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1500). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1500) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (1500) based on the IMU.

[0200] Figures 16A and 16B illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (1600) of Figures 16A and 16B may include at least a portion of the hardware of the wearable device (1500) described with reference to Figures 15A and / or 15B. An example of an exterior appearance of a first side (1610) of a housing of the wearable device (1600) according to one embodiment is illustrated in Figure 16A, and an example of an exterior appearance of a second side (1620) opposite to the first side (1610) may be illustrated in Figure 16B.

[0201] Referring to FIG. 16A, a first surface (1610) of a wearable device (1600) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1600) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1504) and / or the second temple (1505) of FIGS. 15A and 15B). A first display (1550-1) for outputting an image to a left eye among the user's two eyes, and a second display (1550-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1610). The wearable device (1600) may be formed on the first surface (1610) and may further include a rubber or silicone packing to prevent interference from light (e.g., ambient light) different from the light emitted from the first display (1550-1) and the second display (1550-2).

[0202] According to one embodiment, a wearable device (1600) may include cameras (1560-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1550-1) and the second display (1550-2). The cameras (1560-1) may be referred to as the gaze tracking camera (1560-1) of FIG. 15B. According to one embodiment, a wearable device (1600) may include cameras (1560-5, 1560-6) for photographing and / or recognizing a face of a user. The cameras (1560-5, 1560-6) may be referred to as FT cameras. The wearable device (1600) can control an avatar representing the user in a virtual space based on the facial motion of the user identified using cameras (1560-5, 1560-6). For example, the wearable device (1600) can change the texture and / or shape of a portion of the avatar (e.g., a portion of the avatar representing a human face) using information obtained by cameras (1560-5, 1560-6) (e.g., FT cameras) and representing the facial expression of the user wearing the wearable device (1600).

[0203] Referring to FIG. 16B, a camera (e.g., cameras (1560-7, 1560-8, 1560-9, 1560-10, 1560-11, 1560-12)) and / or a sensor (e.g., a depth sensor (1630)) for obtaining information related to the external environment of the wearable device (1600) may be disposed on a second surface (1620) opposite to the first surface (1610) of FIG. 16A. For example, the cameras (1560-7, 1560-8, 1560-9, 1560-10) may be disposed on the second surface (1620) to recognize external objects. The cameras (1560-7, 1560-8, 1560-9, 1560-10) of FIG. 16b can correspond to the motion recognition cameras (1560-2, 1560-3) of FIG. 15b.

[0204] For example, using cameras (1560-11, 1560-12), the wearable device (1600) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1560-11) can be placed on the second face (1620) of the wearable device (1600) to obtain an image to be displayed through the second display (1550-2) corresponding to the right eye among the two eyes. The camera (1560-12) can be placed on the second face (1620) of the wearable device (1600) to obtain an image to be displayed through the first display (1550-1) corresponding to the left eye among the two eyes. The cameras (1560-11, 1560-12) can correspond to the shooting camera (1560-4) of FIG. 15B.

[0205] According to one embodiment, the wearable device (1600) may include a depth sensor (1630) disposed on a second face (1620) to identify a distance between the wearable device (1600) and an external object. Using the depth sensor (1630), the wearable device (1600) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1600). Although not shown, a microphone may be disposed on the second face (1620) of the wearable device (1600) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0206] The Metaverse is a portmanteau of the English word "meta," meaning "virtual" or "transcendence," and "universe," and refers to a three-dimensional virtual world where social, economic, and cultural activities, like those in the real world, take place. The Metaverse is a concept that is one step more advanced than virtual reality (VR, a cutting-edge technology that allows people to have realistic experiences in a computer-generated virtual world), and has the characteristic of allowing people to engage in social and cultural activities similar to those in the real world, rather than simply enjoying games or virtual reality using avatars. The Metaverse service can provide media content to enhance immersion in the virtual world based on augmented reality (AR), virtual reality environment (VR), mixed reality (MR), and / or extended reality (XR).

[0207] For example, media content provided by a metaverse service may include social interaction content, including avatar-based games, concerts, parties, and / or conferences. For example, the media content may include advertisements, user-created content, and / or information for economic activities such as the sale and / or shopping of products. Ownership of the user-created content may be proven by a blockchain-based non-fungible token (NFT). The metaverse service may support economic activities based on real-world currency and / or cryptocurrency. The metaverse service may provide virtual content linked to the real world, such as digital twins or life logging.

[0208] Figure 17 is an example diagram of a network environment (1701) that provides metaverse services through a server (1710).

[0209] Referring to FIG. 17, a network environment (1701) may include a server (1710), a user terminal (1720) (e.g., a first terminal (1720-1) and a second terminal (1720-2)), and a network connecting the server (1710) and the user terminal (1720). Within the network environment (1701), the server (1710) may provide a metaverse service to the user terminal (1720). The network may be formed by at least one intermediate node (1730) including an access point (AP) and / or a base station. The user terminal (1720) may connect to the server (1720) through the network and output a user interface (UI) related to the metaverse service to the user of the user terminal (1720). Based on the above UI, the user terminal (1720) can obtain information to be input into the metaverse service from the user, or output information related to the metaverse service (e.g., multimedia content) to the user.

[0210] At this time, the server (1710) provides a virtual space so that the user terminal (1720) can be active in the virtual space. In addition, the user terminal (1720) installs a software agent for accessing the virtual space provided by the server (1710) to express information provided by the server (1710) to the user or transmit information that the user wishes to express in the virtual space to the server. The software agent may be provided directly through the server (1710), downloaded from a public server, or embedded and provided when the terminal is purchased.

[0211] In one embodiment, the metaverse service may be provided to a user terminal (1720) and / or a user using a server (1710). The embodiment is not limited thereto, and the metaverse service may be provided through individual contact between users. For example, within a network environment (1701), the metaverse service may be provided independently from the server (1710) through a direct connection between a first terminal (1720-1) and a second terminal (1720-2). Referring to FIG. 17, within the network environment (1701), the first terminal (1720-1) and the second terminal (1720-2) may be connected to each other through a network formed by at least one intermediate node (1730). In one embodiment where the first terminal (1720-1) and the second terminal (1720-2) are directly connected, either of the user terminals of the first terminal (1720-1) and the second terminal (1720-2) may perform the role of the server (1710). For example, a metaverse environment may be configured solely through device-to-device connections (e.g., peer-to-peer (P2P) connections).

[0212] In one embodiment, the user terminal (1720) (or the user terminal (1720) including the first terminal (1720-1) and the second terminal (1720-2)) may be made in various form factors and is characterized by including an output device for providing images and / or sounds to the user and an input device for inputting information to a metaverse service. Examples of various form factors of the user terminal (1720) include a smartphone (e.g., the second terminal (1720-2)), an AR device (e.g., the first terminal (1720-1)), a VR device, an MR device, a VST (Video See Through) device, an OST (Optical See Through) device, a smart lens, a smart mirror, an input / output capable TV or projector.

[0213] The network (e.g., a network formed by at least one intermediate node (1730)) includes various broadband networks including 3G, 4G, and 5G, as well as short-range networks including Wifi and BT (e.g., a wired network or wireless network directly connecting the first terminal (1720-1) and the second terminal (1720-2)).

[0214] The user terminal (1720) of FIG. 17 may include a wearable device described with reference to FIGS. 1 to 13.

[0215] In one embodiment, a method for tracking an external object using an external electronic device may be required while the external object has moved out of an area recognizable by the wearable device. As described above, according to one embodiment, a wearable device (e.g., the wearable device (101) of FIG. 1) may include one or more sensors (e.g., the sensor (230) of FIG. 2), a communication circuit (e.g., the communication circuit (235) of FIG. 2), a camera (e.g., the camera (225) of FIG. 2), a display (e.g., the display (220) of FIG. 2), and at least one processor (e.g., the processor (210) of FIG. 2). The at least one processor may be configured to control the camera to acquire an image to be displayed on the display. The at least one processor may be configured to identify a first visual object (e.g., visual object (132) of FIG. 1) corresponding to an external object (e.g., external object (120) of FIG. 1) within the acquired image. The at least one processor may be configured to display the image on the display. The at least one processor may be configured to identify a gaze of a user wearing the wearable device using the one or more sensors. The at least one processor may be configured to identify a motion of the external object using the camera based on a direction of the gaze toward the first visual object. The at least one processor may be configured to establish a communication link (e.g., communication link (150) of FIG. 1) using the communication circuitry with at least one external electronic device capable of tracking the external object identified based on the motion.The at least one processor may be configured to, after establishing the communication link, obtain information about the external object from the at least one external electronic device through the communication link based on identifying that the display of the first visual object has been discontinued, and display a second visual object (e.g., the visual object (150) of FIG. 1) related to the external object on the display based on the obtained information.

[0216] For example, the at least one processor may be configured to identify the at least one external electronic device to be connected via the communication link based on a velocity of the external object indicated by the motion.

[0217] For example, the at least one processor may be configured to establish a communication link by communicating with the at least one external electronic device contained in an external space having a size related to the speed.

[0218] For example, the at least one processor may be configured to display the second visual object on the display, the second visual object including text representing the at least one external electronic device that provided the information.

[0219] For example, the at least one processor may be configured to display, on the display, the second visual object comprising a video acquired by the at least one external electronic device.

[0220] For example, the at least one processor may be configured to identify the at least one external electronic device with which the communication link is to be established based on fields-of-views (FoVs) of cameras included in a plurality of external electronic devices adjacent to the wearable device and a positional relationship between the external object and the at least one external electronic device.

[0221] For example, the at least one processor may be configured to obtain information from the plurality of external electronic devices, the information representing the FoVs and at least one external object being tracked by each of the plurality of external electronic devices. The at least one processor may be configured to identify the at least one external electronic device with which the communication link is to be established, based on the information obtained from the plurality of external electronic devices.

[0222] For example, the at least one processor may be configured to initiate obtaining the information from the at least one external electronic device based on identifying that the first visual object has moved beyond the periphery of the display covering both eyes of the user while the wearable device is being worn by the user due to movement of the external object relative to the wearable device.

[0223] According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of controlling a camera of the wearable device to obtain an image to be displayed on a display of the wearable device. The method may include an operation of identifying a first visual object corresponding to an external object within the obtained image. The method may include an operation of displaying the image on the display of the wearable device. The method may include an operation of identifying a gaze of a user wearing the wearable device using one or more sensors of the wearable device. The method may include an operation of identifying a motion of the external object using the camera based on a direction of the gaze toward the first visual object. The method may include an operation of establishing a communication link with at least one external electronic device capable of tracking the external object identified based on the motion using a communication circuit of the wearable device. The method may include an operation of obtaining information about the external object from the at least one external electronic device through the communication link based on determining that the display of the first visual object has been discontinued after establishing the communication link, and displaying a second visual object related to the external object on the display based on the obtained information.

[0224] For example, the establishing action may include identifying the at least one external electronic device to be connected via the communication link based on a velocity of the external object indicated by the motion.

[0225] For example, the establishing operation may include establishing the communication link by communicating with at least one external electronic device contained in an external space having a size related to the speed.

[0226] For example, the act of displaying the second visual object may include displaying the second visual object on the display, the second visual object including text indicating the at least one external electronic device that provided the information.

[0227] For example, the act of displaying the second visual object may include an act of displaying the second visual object, including a video acquired by the at least one external electronic device, on the display.

[0228] For example, the establishing operation may include an operation of identifying at least one external electronic device with which the communication link is to be established based on FoVs of cameras included in a plurality of external electronic devices adjacent to the wearable device and a positional relationship between the external object.

[0229] For example, the act of identifying the at least one external electronic device may include the act of obtaining, from the plurality of external electronic devices, information representing the FoVs and at least one external object being tracked by each of the plurality of external electronic devices. The method may include the act of identifying the at least one external electronic device with which the communication link is to be established, based on the information obtained from the plurality of external electronic devices.

[0230] For example, the act of displaying the second visual object may include an act of initiating acquisition of the information from the at least one external electronic device based on identifying that the first visual object has moved beyond the edge of the display due to movement of the external object relative to the wearable device, within the display covering both eyes of the user, while the wearable device is worn by the user.

[0231] According to one embodiment, a wearable device as described above may include one or more sensors, a communication circuit, a camera, a display, and at least one processor. The at least one processor may be configured to control the camera to obtain an image to be displayed on the display. The at least one processor may be configured to display the image including a visual object corresponding to an external object on the display. The at least one processor may be configured to identify a direction of a gaze of a user wearing the wearable device using sensor data of the one or more sensors. The at least one processor may be configured to identify a motion of the external object moving along a direction from a first location using the camera based on the direction of the gaze toward the visual object. The at least one processor may be configured to establish a communication link with a first external electronic device adjacent to the first location using the communication circuit based on the motion. The at least one processor may be configured to change the communication link to share information about the external object with the first external electronic device and a second external electronic device adjacent to the second location, based on identifying that the external object has moved from the first location to the second location along the direction after establishing the communication link.

[0232] For example, the at least one processor may be configured to request the first external electronic device and the second external electronic device to transmit the information about the external object through the communication link based on identifying that the external object has moved out of the field-of-view (FoV) of the camera while displaying the video acquired from the camera on the display.

[0233] For example, the at least one processor may be configured to, in response to obtaining the information about the external object from the first external electronic device or the second external electronic device via the communication link, display another visual object on the display based on the information.

[0234] For example, the at least one processor may be configured to display the other visual object, the other visual object including text representing the external electronic device corresponding to the acquired information, among the first external electronic device or the second external electronic device.

[0235] According to one embodiment of the present invention, a method of a wearable device as described above may include an operation of controlling a camera of the wearable device to obtain an image to be displayed on the display. The method may include an operation of displaying the image including a visual object corresponding to an external object on the display of the wearable device. The method may include an operation of identifying a direction of a gaze of a user wearing the wearable device using sensor data of one or more sensors of the wearable device. The method may include an operation of identifying a motion of the external object moving along a direction from a first location using the camera based on the direction of the gaze toward the visual object. The method may include an operation of establishing a communication link with a first external electronic device adjacent to the first location using a communication circuit of the wearable device based on the motion. The method may include, after establishing the communication link, an operation of changing the communication link to share information about the external object with the first external electronic device and a second external electronic device adjacent to the second location, based on identifying that the external object has moved from the first location to the second location along the direction.

[0236] For example, the method may include an action of requesting the first external electronic device and the second external electronic device to transmit the information about the external object through the communication link based on identifying that the external object has moved out of the FoV of the camera while displaying the video acquired from the camera on the display.

[0237] For example, the method may include, in response to obtaining the information about the external object from the first external electronic device or the second external electronic device via the communication link, displaying another visual object based on the information on the display.

[0238] For example, the action of displaying the other visual object may include an action of displaying the other visual object including text indicating an external electronic device corresponding to the acquired information among the first external electronic device or the second external electronic device.

[0239] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0240] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0241] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0242] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0243] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In wearable devices, One or more sensors; communication circuit; camera; display; A memory storing instructions, comprising one or more storage media; and At least one processor including a processing circuit, wherein the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Controlling the above camera to obtain an image to be displayed on the display; Identifying a first visual object corresponding to an external object within the acquired image; Displaying the image on the display; Using one or more of the sensors, identifying the gaze of a user wearing the wearable device; Based on the direction of the gaze toward the first visual object, the motion of the external object is identified using the camera; Establishing a communication link with at least one external electronic device capable of tracking the external object identified based on the motion using the communication circuit; and After establishing said communication link, based on identifying that the display of said first visual object is discontinued, obtaining information about said external object from said at least one external electronic device through said communication link, and causing a second visual object related to said external object to be displayed on said display based on said obtained information. Wearable devices.

2. In claim 1, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: causing said at least one external electronic device to be connected via said communication link to be identified based on the velocity of said external object, as indicated by said motion; Wearable devices.

3. In claims 1 and 2, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing to establish a communication link by communicating with at least one external electronic device contained in an external space having a size related to said speed, Wearable devices.

4. In claims 1 to 3, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing the display to display a second visual object, the second visual object including text representing the at least one external electronic device that provided the information; Wearable devices.

5. In claims 1 to 4, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing the display to display the second visual object, which includes a video acquired by the at least one external electronic device; Wearable devices.

6. In claims 1 to 5, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Identifying at least one external electronic device with which the communication link is to be established based on the field-of-views (FoVs) of cameras included in a plurality of external electronic devices adjacent to the wearable device and the positional relationship between the external object, Wearable devices.

7. In claims 1 to 6, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: From said plurality of external electronic devices, information representing at least one external object being tracked by each of said FoVs and said plurality of external electronic devices is obtained; Based on the information obtained from the plurality of external electronic devices, causing at least one external electronic device to be identified with which the communication link is to be established. Wearable devices.

8. In claims 1 to 7, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Initiating acquisition of the information from the at least one external electronic device based on identifying that the first visual object has moved beyond the periphery of the display due to movement of the external object relative to the wearable device, within the display covering both eyes of the user, while the wearable device is worn by the user. Wearable devices.

9. In the method of a wearable device, An action of controlling a camera of the wearable device to obtain an image to be displayed on a display of the wearable device; An operation of identifying a first visual object corresponding to an external object within the acquired image; An action of displaying the image on the display of the wearable device; An action of identifying the gaze of a user wearing the wearable device by using one or more sensors of the wearable device; An action of identifying the motion of the external object using the camera based on the direction of the gaze toward the first visual object; An action of establishing a communication link with at least one external electronic device capable of tracking the external object identified based on the motion, using a communication circuit of the wearable device; After establishing said communication link, based on identifying that the display of said first visual object is discontinued, an operation of acquiring information about said external object from said at least one external electronic device through said communication link, and displaying a second visual object related to said external object on said display based on said acquired information, method.

10. In claim 9, the establishing operation comprises: An operation comprising identifying at least one external electronic device to be connected via the communication link based on a velocity of the external object indicated by the motion. method.

11. In claims 9 to 10, the establishing operation comprises: Comprising an operation of establishing a communication link by communicating with at least one external electronic device contained in an external space having a size related to the speed; method.

12. In claims 9 to 11, the action of displaying the second visual object comprises: Including an action of displaying the second visual object on the display, the second visual object including text representing the at least one external electronic device that provided the information; method.

13. In claims 9 to 12, the action of displaying the second visual object comprises: Comprising an action of displaying, on said display, said second visual object including a video acquired by said at least one external electronic device; method.

14. In claims 9 to 13, the establishing operation comprises: An operation of identifying at least one external electronic device with which the communication link is to be established based on the FoVs of cameras included in a plurality of external electronic devices adjacent to the wearable device and the positional relationship between the external object, method.

15. In claims 9 to 14, the operation of identifying at least one external electronic device comprises: An operation of obtaining information from said plurality of external electronic devices, said FoVs and at least one external object being tracked by each of said plurality of external electronic devices; An operation of identifying at least one external electronic device with which the communication link is to be established, based on the information obtained from the plurality of external electronic devices, method.

Images