Electronic device for selecting and controlling object based on classification and method thereof

Abstract

A processor of a wearable device according to an embodiment may be configured to: in response to an input indicating a selection of a first external electronic device and a second external electronic device of a plurality of external electronic devices shown on a display of the wearable device; a third external electronic device in the category including both the first external electronic device and the second external electronic device is identified. The processor may be configured to interact with at least one fourth external electronic device of the third external electronic devices different from the first external electronic device and the second external electronic device, and display at least one third visual object for guiding selection of the at least one fourth external electronic device.

Description

Technical Field

[0001] The following description relates to an electronic device for selecting and controlling an object based on classification and a method thereof. Background Art

[0002] In order to provide an enhanced user experience, an electronic device is being developed that provides an augmented reality (AR) service that displays information generated by a computer in combination with objects in the real world. The electronic device may be a wearable device that can be worn by a user. For example, the electronic device may be AR glasses. Summary of the invention

[0003] Technical Solution

[0004] According to an embodiment, a wearable device may include a display and a processor. The processor may be configured to receive an input indicating a selection of a first external electronic device and a second external electronic device among a plurality of external electronic devices viewable through the display. The processor may be configured to, in response to the input, identify a third external electronic device in a category including both the first external electronic device and the second external electronic device. The third external electronic device may include the first external electronic device and the second external electronic device. The processor may be configured to display a first visual object and a second visual object as being associated with the first external electronic device and the second external electronic device viewable through the display, the first visual object and the second visual object indicating that the first external electronic device and the second external electronic device are selected, respectively. The processor may be configured to display at least one third visual object as being associated with at least one fourth external electronic device among the third external electronic devices that is different from the first external electronic device and the second external electronic device, the at least one third visual object guiding the selection of the at least one fourth external electronic device.

[0005] A method of a wearable device according to an embodiment may include receiving a first input for selecting one or more first external electronic devices among a plurality of external electronic devices viewable through a display. The method may include identifying one or more first functions applicable to the one or more first external electronic devices in response to the first input. The method may include displaying one or more visual objects for performing the one or more first functions within the display. The method may include identifying at least one second function applicable to at least one second external electronic device in response to a second input indicating a selection of at least one second external electronic device in a state where the one or more visual objects are displayed. The method may include stopping displaying at least one visual object about a function different from the at least one second function in the one or more visual objects. A wearable device according to an embodiment may provide a UI in response to a change in a user's selection of an external electronic device.

[0006] A method of a wearable device according to an embodiment may include receiving an input indicating a selection of a first external electronic device and a second external electronic device among a plurality of external electronic devices viewable through a display of the wearable device. The method may include, in response to the input, identifying a third external electronic device in a category that includes both the first external electronic device and the second external electronic device. The third external electronic device may include the first external electronic device and the second external electronic device. The method may include displaying a first visual object and a second visual object as being associated with the first external electronic device and the second external electronic device viewable through the display, the first visual object and the second visual object indicating that the first external electronic device and the second external electronic device are selected, respectively. The method may include displaying at least one third visual object as being associated with at least one fourth external electronic device among the third external electronic devices that is different from the first external electronic device and the second external electronic device, the at least one third visual object guiding the selection of at least one fourth external electronic device.

[0007] According to an embodiment, a wearable device may include a display and a processor. The processor may be configured to receive a first input indicating a selection of one or more first external electronic devices among a plurality of external electronic devices viewable through the display. The processor may be configured to identify one or more first functions applicable to the one or more first external electronic devices in response to the first input. The processor may be configured to display one or more visual objects for performing the one or more first functions within the display. The processor may be configured to identify at least one second function applicable to at least one second external electronic device in response to a second input indicating a selection of at least one second external electronic device while displaying the one or more visual objects. The processor may be configured to stop displaying at least one visual object of the one or more visual objects regarding a function different from the at least one second function. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1a to Figure 1b An example of a user interface (UI) displayed by a wearable device based on multiple external electronic devices according to an embodiment is shown.

[0009] Figure 2 An example of a block diagram of a wearable device according to an embodiment is shown.

[0010] Figure 3a An example of a perspective view of a wearable device according to an embodiment is shown.

[0011] Figure 3b An example of one or more hardware provided in a wearable device according to an embodiment is shown.

[0012] Figure 4An example of a category of an external electronic device that a wearable device recognizes based on metadata according to an embodiment is shown.

[0013] Figure 5a to Figure 5c An example of an operation in which a wearable device receives an input indicating a connection between an external electronic device and the wearable device according to an embodiment is shown.

[0014] Figure 6a to Figure 6b An example of an operation in which a wearable device according to an embodiment receives an input indicating a selection of at least one of external electronic devices based on a user's motion is shown.

[0015] Figure 7a to Figure 7b An example of an operation in which a wearable device according to an embodiment receives an input indicating a selection of at least one of external electronic devices based on a user's motion and voice is shown.

[0016] Figures 8a to 8c An example of an operation in which a wearable device displays a selected visual object for guiding at least one external electronic device according to an embodiment is shown.

[0017] Figure 9a to Figure 9b An example of a screen displayed by the wearable device according to an embodiment is shown based on a category of an external electronic device selected by a user.

[0018] Figures 10a to 10c An example of a screen displayed by a wearable device based on an external electronic device selected by a user according to an embodiment is shown.

[0019] Fig.11 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown.

[0020] Figure 12a to Figure 12b An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown.

[0021] Fig.13 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown.

[0022] Fig.14 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown.

[0023] Fig.15 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown. DETAILED DESCRIPTION

[0024] Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings.

[0025] The various embodiments of this document and the terms used herein are not intended to limit the technology described in this document to specific embodiments, and should be understood to include various modifications, equivalents or substitutes of the corresponding embodiments. With respect to the description of the accompanying drawings, reference numerals may be used for similar components. Singular expressions may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B or C" or "at least one of A, B and / or C" may include all possible combinations of items listed together. Expressions such as "1st", "2nd", "first" or "second" may modify the corresponding components, regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When a (for example, a first) component is referred to as "connected (functionally or communicatively)" or "accessed" to another (for example, a second) component, the component may be directly connected to another component or may be connected through another component (for example, a third component).

[0026] The term "module" used in this document may include a unit configured with hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, etc. A module may be a component of an overall configuration that performs one or more functions or its smallest unit or part. For example, a module may be configured with an application specific integrated circuit (ASIC).

[0027] Figure 1a to Figure 1b An example of a user interface (UI) displayed by the wearable device 101 based on a plurality of external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 according to an embodiment is shown. Figure 1a and Figure 1b According to an embodiment, the wearable device 101 may include a head-mounted display (HMD) that can be worn on a body part (e.g., head) of a user. For example, the housing of the wearable device 101 may include a flexible material (such as rubber and / or silicone) in a form that contacts a portion of the user's head (e.g., a portion of the face covering both eyes). For example, the housing of the wearable device 101 may include one or more straps that can be wrapped around the user's head and / or one or more temples that can be attached to the ears of the head. Figure 2 An example of hardware included in the wearable device 101 according to the embodiment is described. Figure 3a to Figure 3b An example of a structure of the wearable device 101 according to the embodiment is described.

[0028] The wearable device 101 according to an embodiment may communicate with one or more external electronic devices (e.g., external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171) different from the wearable device 101 through a wired network and / or a wireless network. The wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), Ethernet, or a combination thereof. The wireless network may include a network such as long-term evolution (LTE), 5g new air interface (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low energy consumption (BLE), or a combination thereof. For example, the wearable device 101 may communicate with one or more external electronic devices through a server in the network. The signals and / or packets exchanged between the wearable device 101 and the one or more external electronic devices may be relayed by the server. For example, the wearable device 101 may be directly connected to one or more external electronic devices independently of the server by using Bluetooth, BLE, NFC, and / or WiFi direct (or Wi-Fi point-to-point (P2P)). For example, wearable device 101 may be connected indirectly via one or more routers and / or access points (APs) in the network.

[0029] Figure 1a to Figure 1b The external electronic devices 121, 122, 123, 131, 132, 141, 142, 161 and 171 may be examples of external electronic devices accessible by the wearable device 101. The external electronic devices accessible by the wearable device 101 may include external electronic devices that are physically separated from the wearable device 101, such as an Internet of Things (IoT) device. The wearable device 101 according to an embodiment may identify the external electronic device based on metadata of the external electronic device. Identifying the external electronic device by the wearable device 101 may include an operation of identifying at least one function supportable by the external electronic device based on the metadata. Identifying the external electronic device by the wearable device 101 may include an operation of classifying the external electronic device based on hierarchical categories. Hereinafter, a category may be referred to as a classification and / or a class. Hierarchical categories may be used to identify at least one function that is commonly applicable to multiple external electronic devices. Reference will be made to Figure 4 An operation of the wearable device 101 according to an embodiment to classify at least one external electronic device accessible by the wearable device 101 based on hierarchical categories is described.

[0030] Reference Figure 1a , showing different examples of external electronic devices connected to the wearable device 101. Figure 1a , the external electronic devices accessible by the wearable device 101 may include lighting devices (e.g., light bulbs, chandeliers, and / or lamp stands) that emit light and are controlled by the wearable device 101 (such as the external electronic devices 121, 122, and 123). Figure 1a, the external electronic device accessible by the wearable device 101 may include a surveillance camera (such as the external electronic device 131) that transmits media content including video captured by the camera. Figure 1a , the external electronic device accessible by the wearable device 101 may include a television (such as the external electronic device 132) that includes a display and / or speakers for outputting media content. Figure 1a , the external electronic device accessible by the wearable device 101 may include an air purifier (such as the external electronic device 141). Figure 1a , the external electronic device accessible by the wearable device 101 may include an air conditioner (such as the external electronic device 142). Figure 1a , the external electronic device accessible by the wearable device 101 may include a remote controller (e.g., a game controller and / or a TV remote controller) for controlling other external electronic devices (such as the external electronic device 161). Figure 1a , the external electronic devices accessible by the wearable device 101 may include a table (such as the external electronic device 171). Figure 5a to Figure 5c An example of an operation in which the wearable device 101 is connected to an external electronic device according to an embodiment is described. The external electronic device is not limited to the above example, and may include an IoT device that obtains information indicating an environment based on a sensor.

[0031] The wearable device 101 according to the embodiment may perform functions related to augmented reality (AR). The wearable device 101 may include a display for displaying a screen related to augmented reality. The display may include a transparent material. For example, based on the transparent material included in the display, light emitted toward the surface of the display may at least partially penetrate to another surface opposite to the surface. The wearable device 101 according to the embodiment may display a screen related to an external object for viewing by the user in the display through the display based on augmented reality. The wearable device 101 according to the embodiment may display information related to an external object to the user based on the screen. The wearable device 101 according to the embodiment may display information for controlling an external object (such as external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171) to the user by using the screen, or information electrically received from an external object.

[0032] Reference Figure 1a to Figure 1b, shows an example of a field of view (FoV) 110 for viewing by a user through a display of a wearable device 101. According to an embodiment, the wearable device 101 may identify external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 included in the FoV 110 based on object detection and / or object recognition. In order to perform object detection and / or object recognition, the wearable device 101 may include a camera for capturing images and / or videos related to the FoV 110. The screen displayed within the FoV 110 by the wearable device according to an embodiment using the display may include enhanced or annotated information based on the environment viewed by the user.

[0033] Figure 1b An example of a screen displayed by the wearable device 101 within the FoV 110 by using a display according to an embodiment is shown. The wearable device 101 according to an embodiment may display visual objects 151, 152, and 153 indicating different external electronic devices within the FoV 110 by using a display. The visual objects 151, 152, and 153 may be displayed by the wearable device 101 to support selection and / or control of the external electronic device independently of the change of the FoV 110 due to the movement of the wearable device 101. The visual objects 151, 152, and 153 may be added by a user. Each of the visual objects 151, 152, and 153 may be related to any one of the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and / or another external electronic device not included in the FoV 110.

[0034] In a state of recognizing the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171, the wearable device 101 according to an embodiment may receive an input indicating selection of at least one of the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153. The input may include an input of selecting an external electronic device (such as the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171) that is tangible and viewed through the FoV 110, and / or another input indicating selection of an external electronic device by using a visual object (such as the visual objects 151, 152, and 153) representing the external electronic device. The wearable device 101 according to an embodiment may guide selection of a group of external electronic devices distinguished by category by using homogeneity based on categories of each of the external electronic devices corresponding to the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153 for the user. Figure 6a to Figure 6b and / or Figure 7a to Figure 7b Describe exemplary operations of the wearable device 101 receiving input.

[0035] In an embodiment, in response to an input indicating a selection of at least one of the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153, the wearable device 101 may display one or more visual objects for guiding selection of another external electronic device that is different from the at least one external electronic device selected by the input and is similar to the at least one external electronic device selected by the input. The other external electronic device may be executed based on the wearable device 101 classifying the external electronic devices corresponding to the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153 into hierarchical categories. Reference will be made to Figures 8a to 8c An example of operation of the wearable device 101 displaying one or more visual objects based on hierarchical categories according to an embodiment is described.

[0036] In an embodiment, in response to an input indicating a selection of at least one of the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153, the wearable device 101 may display a screen for controlling at least one external electronic device selected by the input. In a state where multiple external electronic devices are selected by the input, the wearable device 101 may selectively and / or preferentially display functions that are commonly applicable to the multiple external electronic devices selected by the input. Figure 9a to Figure 9b and / or Figures 10a to 10c An example of the operation of the wearable device 101 selectively displaying a function in this state according to the embodiment is described.

[0037] As described above, the wearable device 101 according to the embodiment may receive an input indicating a selection of at least two external electronic devices based on the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153 through a display. For example, the wearable device 101 may receive an input indicating a selection of a first external electronic device and a second external electronic device. In response to the input, the wearable device 101 may identify a third external electronic device in a category including both the first external electronic device and the second external electronic device. The third external electronic device may include the first external electronic device and the second external electronic device. The third external electronic device may be identified from other external electronic devices related to the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and the visual objects 151, 152, and 153. The wearable device 101 may display the first visual object and the second visual object as being associated with the first external electronic device and the second external electronic device viewable through the display, and the first visual object and the second visual object respectively indicate that the first external electronic device and the second external electronic device are selected. The wearable device 101 may display at least one third visual object as being associated with at least one fourth external electronic device among the third external electronic devices that is different from the first external electronic device and the second external electronic device, and the at least one third visual object guides the selection of the at least one fourth external electronic device.

[0038] As described above, the wearable device 101 according to the embodiment may support selection and / or control of one or more external electronic devices based on the external electronic devices 121, 122, 123, 131, 132, 141, 142, 161, and 171 and / or the visual objects 151, 152, and 153 that can be viewed through the FoV 110. In response to an input indicating the selection of multiple external electronic devices, the wearable device 101 may identify functions that are commonly applicable to multiple external electronic devices by using metadata, or may identify information that can be commonly received from multiple external electronic devices. Based on the commonly applicable functions, the wearable device 101 may control all of the multiple external electronic devices. Based on the commonly receivable information, the wearable device 101 may at least partially combine the information received from each of the multiple external electronic devices. The wearable device 101 may display at least partially combined information to the user. For example, the wearable device 101 may improve the user experience associated with the selection and / or control of multiple external electronic devices.

[0039] In the following, reference will be made to Figure 2 One or more hardware included in the wearable device 101 according to an embodiment is described.

[0040] Figure 2An example of a block diagram of a wearable device 101 according to an embodiment is shown. Figure 2 The wearable device 101 may be Figure 1a to Figure 1b An example of a wearable device 101. Figure 2 According to an embodiment, the wearable device 101 may include at least one of a processor 210, a memory 220, a display 230, a camera 240, a communication circuit 250, a sensor 260, or a microphone 270. The processor 210, the memory 220, the display 230, the camera 240, the communication circuit 250, the sensor 260, and the microphone 270 may be electronically and / or operably coupled to each other through electronic components (such as a communication bus). Hardware operably coupled may mean establishing a direct connection or an indirect connection between hardware by wire or wireless, so that specific hardware of the hardware is controlled by another hardware, and / or another hardware is controlled by specific hardware. The type and / or number of hardware included in the wearable device 101 is not limited to Figure 2 .

[0041] The processor 210 of the wearable device 101_2 according to the embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors 210 may be one or more. For example, the processor 210 may have a structure of a multi-core processor such as a dual-core, quad-core, or hexa-core.

[0042] The memory 220 of the wearable device 101 according to an embodiment may include a hardware component for storing data and / or instructions input to or output from the processor 210. The memory 220 may include, for example, a volatile memory such as a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, an optical disk, and an embedded multimedia card (eMMC).

[0043] One or more instructions indicating calculations and / or operations to be performed on data by the processor 220 may be stored in the memory 220. A collection of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a subroutine, and / or an application. For example, when executing a collection of multiple instructions distributed in the form of an operating system, firmware, a driver, and / or an application, the wearable device 101 and / or the processor 210 may perform Figures 11 to 15 At least one of the operations of the wearable device 101. Hereinafter, an application installed in the wearable device 101 may mean that one or more instructions provided in the form of an application are stored in the memory 220 of the wearable device 101, and the one or more applications are stored in a format executable by the processor 210 of the wearable device 101 (for example, a file with an extension preset by the operating system of the wearable device 101).

[0044] The display 230 of the wearable device 101 according to an embodiment may output visual information (e.g., Figure 5a to Figure 5c , Figure 6a to Figure 6b , Figure 7a to Figure 7b , Figures 8a to 8c , Figure 9a to Figure 9b and / or Figures 10a to 10c For example, the display 230 may output visual information to the user by being controlled by a controller such as the processor 210 (e.g., a graphics processing unit (GPU)). The display 230 may include a flat panel display (FPD) and / or an electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LED may include an organic LED (OLED). As described above with reference to Figure 1a to Figure 1b As described above, the penetration of light may occur in at least a portion of the display 230 of the wearable device 101. The wearable device 101 may provide a user with a user experience related to augmented reality by providing a combination of light output through the display 230 and light penetrating the display 230.

[0045] The camera 240 of the wearable device 101 according to an embodiment may include one or more optical sensors (e.g., a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor) that generate electronic signals indicating the color and / or brightness of light. The multiple optical sensors included in the camera 240 may be arranged in the form of a 2-dimensional array. The camera 240 can generate an image corresponding to the light reaching the optical sensor of the two-dimensional array and including a plurality of pixels arranged in two dimensions by obtaining the electronic signals of each of the multiple optical sensors substantially at the same time. For example, the shooting data captured by using the camera 240 may mean one image obtained from the camera 240. For example, the video data captured by using the camera 240 may mean a sequence of multiple images obtained from the camera 240 according to a preset frame rate. The wearable device 101 according to an embodiment may also include a flash arranged in a direction in which the camera 240 receives light, for outputting light in that direction. The number of cameras 240 included in the wearable device 101 according to an embodiment may be one or more.

[0046] In an embodiment, the FoV of the camera 240, as an area formed based on the viewing angle at which the lens of the camera 240 can receive light, may correspond to an area corresponding to an image generated by the camera 240. Hereinafter, a subject may refer to an object included in the FOV of the camera 240 and be distinguished from the wearable device 101. In an embodiment, the FoV of the camera 240 may at least partially match an environment viewable by a user through the display 230, such as Figure 1a to Figure 1b FoV 110.

[0047] Although not shown, the wearable device 101 according to an embodiment may include an output mode for outputting information in a form other than a visual form. For example, the wearable device 101 may include a speaker for outputting an acoustic signal. For example, the wearable device 101 may include a motor for providing tactile feedback based on vibration.

[0048] The communication circuit 250 of the wearable device 101 according to an embodiment may include hardware for supporting the transmission and / or reception of electronic signals between the wearable device 101 and the external electronic device 290. For example, the external electronic device 290 may include hardware for transmitting and / or receiving electronic signals between the wearable device 101 and the external electronic device 290. Figure 1a to Figure 1b The communication circuit 250 may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit 250 may support the transmission and / or reception of electronic signals based on various types of protocols such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new air interface (NR).

[0049] Reference Figure 2 , shows an exemplary case where the wearable device 101 according to the embodiment communicates with the external electronic device 290 through the network 280. For example, the wearable device 101 can identify the external electronic device 290 within the network 280 by using the communication circuit 250. The embodiment is not limited thereto, and the wearable device 101 can be directly connected to the external electronic device 290 independently of the network 280 by using the communication circuit 250.

[0050] Although shown based on different blocks, the embodiments are not limited thereto. Figure 2 Part of the hardware components shown in FIG. 2 (eg, at least one of the processor 210 , the memory 220 , and the communication circuit 250 ) may be included in a single integrated circuit such as a system on a chip (SoC).

[0051] The sensor 260 of the wearable device 101 according to an embodiment may generate electronic information to be processed by the processor 210 and / or the memory 220 from non-electronic information related to the wearable device 101. The electronic information generated by the sensor 260 may be stored in the memory 220, processed by the processor 210, and / or sent to an external electronic device distinguished from the wearable device 101.

[0052] According to an embodiment, the sensor 260 of the wearable device 101 may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting the posture of the wearable device 101 and / or the posture of a body part (e.g., head) of the user wearing the wearable device 101. Each of the gravity sensor and the acceleration sensor may measure gravity acceleration and / or acceleration based on preset three-dimensional axes (e.g., x-axis, y-axis, and z-axis) perpendicular to each other. The gyro sensor may measure the angular velocity of each of the preset three-dimensional axes (e.g., x-axis, y-axis, and z-axis). The gravity acceleration, acceleration, and angular velocity measured in each of the gravity sensor, the acceleration sensor, and the gyro sensor may be output in the form of electronic information that can be processed by the processor 210 and / or stored in the memory 220. The gravity sensor, the acceleration sensor, and the gyro sensor may repeatedly output gravity acceleration, acceleration, and angular velocity based on a preset period. At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU).

[0053] The microphone 270 of the wearable device 101 according to an embodiment may output an electronic signal representing air vibration. For example, the wearable device 101 may obtain an audio signal including the user's voice by using the microphone 270. The user's voice included in the audio signal may be converted into information in a format recognizable by the processor 210 of the wearable device 101 based on a speech recognition model and / or a natural language understanding model. For example, the wearable device 101 may perform one or more of a plurality of functions that may be provided by the wearable device 101 by recognizing the user's voice. Figure 7a to Figure 7b An example of an operation performed by the wearable device 101 according to an embodiment based on a voice received through the microphone 270 is described.

[0054] As described above, the wearable device 101 according to the embodiment may include one or more hardware for providing a user experience based on augmented reality. For example, the wearable device 101 may identify an external electronic device 290 that can be viewed through the display 230 by using the communication circuit 250 and / or the camera 240. The wearable device 101 may display a screen for selecting and / or controlling the identified external electronic device 290 within the display 230. The wearable device 101 may receive an input indicating the selection of the external electronic device 290 based on the sensor 260, the microphone 270, and / or the communication circuit 250. Based on identifying the selection of multiple external electronic devices through the input, the wearable device 101 may identify at least one function that is commonly applicable to multiple external electronic devices by using a hierarchical connection of a category assigned to each of the multiple external electronic devices. The wearable device 101 may improve the user experience associated with the control of multiple external electronic devices by displaying a screen for executing at least one function to the user through the display 230.

[0055] In the following, reference will be made to Figure 3a to Figure 3b An example of a structure of the wearable device 101 according to the embodiment is described.

[0056] Figure 3a An example of a perspective view of a wearable device 101 according to an embodiment is shown. Figure 3b An example of one or more hardware provided in the wearable device 101 according to an embodiment is shown. Figure 3a to Figure 3b The wearable device 101 may be as shown in FIG. Figure 2 An example of a wearable device 101. Figure 3a According to an embodiment, the wearable device 101 may include at least one display 230 and a frame 300 supporting the at least one display 230. The at least one display 230 may be Figure 2 An example of display 230 is shown.

[0057] According to an embodiment, the wearable device 101 may be wearable on a part of the user's body. The wearable device 101 may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) combining augmented reality and virtual reality to the user wearing the wearable device 101. For example, in response to a preset gesture of the user obtained by the motion recognition camera 240-2, the wearable device 101 may display a virtual reality image provided from at least one optical device 382 and 384 on at least one display 230.

[0058] According to an embodiment, at least one display 230 may provide visual information to the user. For example, at least one display 230 may include a transparent or translucent lens. At least one display 230 may include a first display 230-1 and / or a second display 230-2 spaced apart from the first display 230-1. For example, the first display 230-1 and the second display 230-2 may be respectively disposed at positions corresponding to the left eye and the right eye of the user.

[0059] Reference Figure 3b , at least one display 230 may provide the user with visual information transmitted from ambient light through a lens included in the at least one display 230 and other visual information distinguished from the visual information. For example, the at least one display 230 may include a first surface 331 and a second surface 332 opposite to the first surface 331. The display area may be formed on the second surface 332 of the at least one display 230. When the user wears the wearable device 101, the ambient light may be transmitted to the user by being incident on the first surface 331 and penetrating the second surface 332. For another example, the at least one display 230 may display an augmented reality image on the display area formed on the second surface 332, in which the virtual reality image provided by the at least one optical device 382 and 384 is combined with the reality screen transmitted by the ambient light. The at least one display 230 may include at least one waveguide 333 and 334 that transmits the light transmitted from the at least one optical device 382 and 384 to the user by diffraction. The wearable device 101 may analyze the light transmitted by the shooting camera (not shown) (for example, Figure 2 The object included in the real image collected by the camera 240 of the wearable device 101 is combined with a virtual object corresponding to the object that becomes the subject of the augmented reality provided among the analyzed objects, and displayed on at least one display 230. The virtual object may include at least one of text and an image for various information associated with the object included in the real image. The user wearing the wearable device 101 can view the image displayed on the at least one display 230.

[0060] According to an embodiment, the frame 300 may be configured with a physical structure in which the wearable device 101 may be worn on the user's body. According to an embodiment, the frame 300 may be configured so that when the user wears the wearable device 101, the first display 230-1 and the second display 230-2 may be placed to correspond to the user's left eye and right eye. The frame 300 may support at least one display 230. For example, the frame 300 may support the first display 230-1 and the second display 230-2 to be placed at positions corresponding to the user's left eye and right eye.

[0061] Reference Figure 3aAccording to an embodiment, in the case where the user wears the wearable device 101, the frame 300 may include an area 320 that at least partially contacts a part of the user's body. For example, the area 320 of the frame 300 that contacts a part of the user's body may include an area that contacts a part of the user's nose, a part of the user's ear, and a part of the side of the user's face that contacts the wearable device 101. According to an embodiment, the frame 300 may include a nose pad 310 that contacts a part of the user's body. When the wearable device 101 is worn by the user, the nose pad 310 may contact a part of the user's nose. The frame 300 may include a first temple 304 and a second temple 305, and the second temple 305 contacts another part of the user's body that is different from the part of the user's body.

[0062] For example, the frame 300 may include a first frame 301 surrounding at least a portion of the first display 230-1, a second frame 302 surrounding at least a portion of the second display 230-2, a bridge 303 disposed between the first frame 301 and the second frame 302, a first pad 311 disposed along a portion of an edge of the first frame 301 from one end of the bridge 303, and a second pad 312 disposed along a portion of an edge of the second frame 302 from the other end of the bridge 303. The first temple 304 extends from the first frame 301 and is fixed to a portion of the wearer's ear, and the second temple 305 extends from the second frame 302 and is fixed to a portion of the ear opposite to the ear. The first pad 311 and the second pad 312 may contact a portion of the user's nose, and the first temple 304 and the second temple 305 may contact a portion of the user's face and the portion of the user's ear. The temples 304 and 305 may be rotatably connected to the frame through hinge units 306 and 307. The first temple 304 may be rotatably connected to the first frame 301 via a first hinge unit 306 disposed between the first frame 301 and the first temple 304. The second temple 305 may be rotatably connected to the second frame 302 via a second hinge unit 307 disposed between the second frame 302 and the second temple 305.

[0063] According to an embodiment, the wearable device 101 may include hardware for performing various functions (e.g., the above-mentioned Figure 2 For example, the hardware may include a battery module 370, an antenna module 375, at least one optical device 382 and 384, a sound output module (not shown), a light emitting module (not shown), and / or a printed circuit board (PCB) 390. Various hardware may be provided in the frame 300.

[0064] According to an embodiment, at least one optical device 382 and 384 may project a virtual object on at least one display 230 so as to provide various image information to the user. For example, at least one optical device 382 and 384 may be a projector. At least one optical device 382 and 384 may be disposed adjacent to at least one display 230, or may be included in at least one display 230 as a part of at least one display 230. According to an embodiment, the wearable device 101 may include a first optical device 382 corresponding to the first display 230-1 and a second optical device 384 corresponding to the second display 230-2. For example, at least one optical device 382 and 384 may include a first optical device 382 disposed at the periphery of the first display 230-1 and a second optical device 384 disposed at the periphery of the second display 230-2. The first optical device 382 may send light to a first waveguide 333 disposed on the first display 230-1, and the second optical device 384 may send light to a second waveguide 334 disposed on the second display 230-2.

[0065] In various embodiments, camera 240 (e.g., Figure 2 The camera 240 may include a photographing camera, an eye tracking camera 240-1, and / or a motion recognition camera 240-2. The photographing camera, the eye tracking camera 240-1, and the motion recognition camera 240-2 may be disposed at different positions on the frame 300 and may perform different functions.

[0066] The capturing camera may capture a real image or background to be matched with the virtual image in order to implement augmented reality or mixed reality content. The capturing camera may capture a real image or background that exists at a location viewed by the user (e.g., Figure 1a to Figure 1b The at least one display 230 may display an image in which a virtual image provided by at least one optical device 382 and 384 overlaps with information about a real image or background including an image of a specific object obtained by using a shooting camera. In an embodiment, the shooting camera may be disposed on a bridge 303 disposed between the first frame 301 and the second frame 302.

[0067] The eye tracking camera 240-1 can achieve a more realistic augmented reality by tracking the gaze of the user wearing the wearable device 101 and matching the user's gaze with the visual information provided on at least one display 230. For example, when the user looks forward, the wearable device 101 can naturally display environmental information associated with the front of the user at the user's location on the at least one display 230. The eye tracking camera 240-1 can be configured to capture an image of the user's pupil in order to determine the user's gaze. For example, the eye tracking camera 240-1 can receive gaze detection light reflected from the user's pupil, and can track the user's gaze based on the position and movement of the received gaze detection light. In an embodiment, the eye tracking camera 240-1 can be set at a position corresponding to the left and right eyes of the user. For example, the eye tracking camera 240-1 can be set in the first frame 301 and / or the second frame 302 to face the direction of the user's location where the wearable device 101 is worn.

[0068] The motion recognition camera 240-2 can provide a specific event to a screen provided on at least one display 230 by recognizing the movement of the whole or part of the user's body (such as the user's torso, hand, or face). The motion recognition camera 240-2 can obtain a signal corresponding to the motion by recognizing the user's motion (e.g., gesture recognition), and can provide a display corresponding to the signal to at least one display 230. The processor can recognize the signal corresponding to the operation and can execute a preset function based on the recognition. In an embodiment, the motion recognition camera 240-2 can be set on the first frame 301 and / or the second frame 302.

[0069] According to an embodiment, the battery module 370 may supply power to the electronic components of the wearable device 101. In an embodiment, the battery module 370 may be disposed in the first temple 304 and / or the second temple 305. For example, the battery module 370 may be a plurality of battery modules 370. A plurality of battery modules 370 may be disposed on each of the first temple 304 and the second temple 305, respectively. In an embodiment, the battery module 370 may be disposed at the end of the first temple 304 and / or the second temple 305.

[0070] The antenna module 375 may transmit a signal or power to the outside of the wearable device 101, or may receive a signal or power from the outside. The antenna module 375 may be electrically and / or operably connected to Figure 2 In an embodiment, the antenna module 375 may be disposed in the first temple 304 and / or the second temple 305. For example, the antenna module 375 may be disposed close to one surface of the first temple 304 and / or the second temple 305.

[0071] The sound output module (not shown) may output a sound signal to the outside of the wearable device 101. The sound output module may be referred to as a speaker. In an embodiment, the sound output module may be provided in the first temple 304 and / or the second temple 305 so as to be provided adjacent to the ear of the user wearing the wearable device 101. For example, the sound output module may include a first sound output module provided adjacent to the left ear of the user by being provided in the first temple 304, and a second sound output module provided adjacent to the right ear of the user by being provided in the second temple 305.

[0072] The light emitting module (not shown) may include at least one light emitting element. The light emitting module may emit light of a color corresponding to a specific state, or may emit light through an operation corresponding to a specific state, so as to visually provide the user with information about a specific state of the wearable device 101. For example, when the wearable device 101 needs to be charged, it may emit red light at a constant period. In an embodiment, the light emitting module may be provided on the first frame 301 and / or the second frame 302.

[0073] As described above, the wearable device 101 according to an embodiment may recognize the movement of the user's head and / or gaze, or may receive the user's voice. The wearable device 101 may execute a function requested by the user in response to the movement and / or voice. The function executed by the wearable device 101 based on the movement and / or voice may include controlling an external electronic device different from the wearable device 101 (e.g., Figure 2 The function performed by the wearable device 101 based on motion and / or voice may include a function of receiving information from the external electronic device. The wearable device 101 according to an embodiment may identify one or more functions that can be supported by the external electronic device by using metadata indicating the capabilities of the external electronic device.

[0074] In the following, reference will be made to Figure 4 An example of an operation in which the wearable device 101 identifies one or more external electronic devices based on metadata according to an embodiment is described.

[0075] Figure 4 Examples of categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451 of external electronic devices are shown, in which the wearable device according to the embodiment is identified based on metadata. Figure 4 The wearable device may be an example of the wearable device 101 of FIGS. 1 to 3 .

[0076] The wearable device according to the embodiment may communicate with the wearable device through a communication circuit (e.g., Figure 2The metadata about at least one external electronic device may include one or more files indicating data about at least one external electronic device based on a format such as extensible markup language (XML) and / or javascript object notation (JSON). For example, metadata about a specific external electronic device may be stored in a single file with a preset extension.

[0077] In an embodiment, the wearable device can communicate with a network (e.g., Figure 2 The server may communicate with a server in a network 280 of the wearable device to obtain metadata. The server may search for metadata to be sent to the wearable device based on a user account logged in to the wearable device. The searched metadata may be related to, for example, an external electronic device previously registered in the user account. The server may send the searched metadata to the wearable device.

[0078] In an embodiment, based on identifying an external electronic device directly connected to an AP connected to the wearable device, the wearable device may obtain metadata from the identified external electronic device. The wearable device may establish a communication link with the external electronic device based on Bluetooth, WiFi, and / or NFC based on a tag attached to the external electronic device, such as a quick response (QR) code. The wearable device may obtain metadata from the external electronic device by using the communication link.

[0079] According to an embodiment, a wearable device may identify at least one external electronic device based on metadata. For example, the metadata may include data related to at least one external electronic device. For example, the wearable device may identify at least one of the type, vendor, name, address in the network (e.g., IP address and / or MAC address), and location (e.g., GPS coordinates) of the external electronic device corresponding to the metadata by using metadata. For example, the wearable device may identify one or more functions applicable to the external electronic device by using metadata. For example, the wearable device may identify available data from the external electronic device by using metadata. For example, the wearable device may identify at least one category including the external electronic device by using metadata.

[0080] The wearable device according to an embodiment may identify one or more categories including external electronic devices in the hierarchical categories based on metadata. Categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451 may be examples of hierarchical categories. Figure 4 , an example of connecting categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451 based on a tree structure is shown, but the embodiment is not limited thereto.

[0081] Figure 4 Category 410 is the highest category in the hierarchical categories and may indicate a general set of external electronic devices that can be connected to the wearable device. Categories 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451 are subcategories of category 410, which is the highest category, and may indicate a partial set of external electronic devices by being directly or indirectly connected to category 410. The wearable device according to an embodiment may classify the external electronic devices in categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451 based on metadata about the external electronic devices. Categorizing the external electronic device by the wearable device may mean selecting the lowest category including the external electronic device from the categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451. The wearable device may identify one or more categories related to the external electronic device based on one or more higher categories sequentially connected to the selected lowest category. For example, the one or more categories related to the external electronic device may include one or more categories connected between the category 410 as the highest category and the selected lowest category.

[0082] Reference Figure 4 , category 420 connected to the highest category 410 may be a category for classifying external electronic devices for providing functions related to lighting. Categories 421, 422, and 423 may be included in category 420 based on the form and / or function of the external electronic devices classified into category 420. For example, category 421 may be a category for classifying independent lighting devices (such as floor lamps). For example, category 422 may be a category for classifying lighting devices fixed to a ceiling (such as chandeliers). For example, category 423 may be a category for classifying lighting devices (such as light bulbs) that are detachable from a socket. Although only categories 421, 422, and 423 are shown as subcategories of category 420, embodiments are not limited thereto.

[0083] Reference Figure 4, category 430 may be a category for classifying external electronic devices for providing functions related to video and / or images. Category 431 is a subcategory of category 430, and may be a category for classifying external electronic devices for capturing videos and / or images. Category 433 is a subcategory of category 431, and may be a category for classifying surveillance cameras such as security cameras and / or CCTV cameras. Category 432 is a subcategory of category 430, and may be a category for classifying external electronic devices for displaying videos and / or images. Category 434 may be a category for classifying televisions.

[0084] Reference Figure 4 , category 440 may be a category for classifying external electronic devices for providing air-related functions. Category 441 is a subcategory of category 440 and may be a category for classifying air conditioners. Category 442 is a subcategory of category 440 and may be a category for classifying air purifiers.

[0085] Reference Figure 4 , category 450 may be a category for classifying controllers for providing a function of controlling an external electronic device. Category 451 is a subcategory of category 450, and may be a category for classifying controllers such as game pads.

[0086] Hereinafter, the wearable device according to the embodiment will be described based on the above reference Figure 4 The categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450 and 451 are described Figure 1a to Figure 1b An exemplary operation of classifying external electronic devices 121, 122, 123, 131, 132, 141, 142, 161 and 171.

[0087] For example, wearable devices can be used as independent lighting devices Figure 1a to Figure 1b External electronic devices 121 are classified into Figure 4 The wearable device can be used as a light bulb connected to a socket. Figure 1a to Figure 1b The external electronic devices 122 are classified into Figure 4 The wearable device can be used as a ceiling mounted lighting device. Figure 1a to Figure 1b External electronic devices 123 are classified into Figure 4 For example, the wearable device may be associated with Figure 1b The external electronic devices corresponding to the visual objects 151 and 152 are classified into Figure 4 The light bulb is in category 423. The wearable device can be connected to Figure 1b The external electronic device corresponding to the visual object 153 is classified into Figure 4 Independent lighting equipment corresponds to category 421. Figure 1a to Figure 1b The external electronic devices 121, 122 and 123 can be classified into Figure 4 In different lowest categories 421, 422 and 423 within category 420.

[0088] For example, wearable devices can be used as surveillance cameras. Figure 1a to Figure 1b External electronic devices 131 are classified into Figure 4 Wearable devices can be used as a television Figure 1a to Figure 1b The external electronic devices 132 are classified into Figure 4 Although classified into different minimum categories 433 and 434, Figure 1a to Figure 1b The external electronic devices 131 and 132 may be classified into the category 430 in terms of providing functions related to video and / or image.

[0089] For example, a wearable device could be used as an air purifier. Figure 1a to Figure 1b External electronic devices 141 are classified into Figure 4 The wearable device can be used as an air conditioner Figure 1a to Figure 1b External electronic devices 142 are classified into Figure 4 The wearable device can be used as a game controller. Figure 1a to Figure 1b External electronic devices 161 are classified into Figure 4 of category 451.

[0090] According to an embodiment, the wearable device may identify at least one function provided by a specific external electronic device based on metadata and / or hierarchical categories (e.g., categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451). For example, the wearable device may identify a device suitable for use as a light bulb based on category 423. Figure 1a to Figure 1b The first function set of the external electronic device 122. For example, the wearable device can identify the wearable device suitable for use as an independent lighting device based on the category 421. Figure 1a to Figure 1b The wearable device may display a screen for executing a function applicable to a specific external electronic device in response to an input indicating selection of the specific external electronic device.

[0091] In an embodiment, the wearable device may receive an input indicating a selection of multiple external electronic devices. The wearable device may use hierarchical categories to interact with the multiple external electronic devices selected by the input. For example, hierarchical categories (e.g., categories 410, 420, 421, 422, 423, 430, 431, 432, 433, 434, 440, 441, 442, 450, and 451) used by the wearable device to classify external electronic devices may be used to identify the homogeneity of external electronic devices. For example, the wearable device may identify a common function of multiple external electronic devices based on the hierarchical categories in response to an input indicating a selection of multiple external electronic devices. For example, each hierarchical category may indicate a function that is commonly applicable to the external electronic devices included in the category.

[0092] For example, in response to an indication of Figure 1a and Figure 1b Based on the input of the selection of all external electronic devices 121 and 122, the wearable device can be classified based on each of the external electronic devices 121 and 122. Figure 4 The categories 423 and 421 are used to identify the first set of functions and the second set of functions applicable to each of the external electronic devices 121 and 122. The intersection of the first set and the second set may include the functions that are commonly applicable to each of the external electronic devices 121 and 122. Figure 1a to Figure 1b The wearable device according to the embodiment may display a screen for executing at least one function included in the intersection in response to an input.

[0093] An exemplary operation of the wearable device based on the intersection of the multi-function sets applicable to each of the plurality of external electronic devices has been described, but the embodiments are not limited thereto. For example, in response to an input indicating selection of the plurality of external electronic devices, the wearable device may identify the lowest category within the hierarchical category that includes all of the plurality of external electronic devices. Similar to this example, upon receiving an input indicating selection of the plurality of external electronic devices, the wearable device may identify the lowest category within the hierarchical category that includes all of the plurality of external electronic devices. Figure 1a to Figure 1b In the state of the selected input of all the external electronic devices 121 and 122, the wearable device can identify each of the external electronic devices 121 and 122 that are classified Figure 4 The higher category of each of categories 423 and 421. Figure 4 , the category 420 including all the categories 423 and 421 may be the lowest category among the categories 410 and 420 including all the categories 423 and 421. The wearable device may identify at least one function commonly applicable to the external electronic devices 121 and 122 selected by the input based on the category 420.

[0094] According to an embodiment, the wearable device may guide the user's selection according to the homogeneity of the external electronic device based on the hierarchical categories. For example, in response to an input selecting at least two external electronic devices, the wearable device may guide the selection of another external electronic device different from the external electronic device based on a union of categories corresponding to each external electronic device selected by the input or a higher category. Similar to this example, upon receiving an instruction to select at least two external electronic devices, the wearable device may guide the selection of at least one external electronic device different from the external electronic device based on a union of categories corresponding to each external electronic device selected by the input or a higher category. Figure 1a to Figure 1b In the state of the selected input of all the external electronic devices 121 and 122, the wearable device can identify each of the external electronic devices 121 and 122 that are classified Figure 4 Another external electronic device included in a higher category of each of the categories 423 and 421. The other external electronic device may include, for example, Figure 4 External electronic devices included in category 420 of higher categories such as categories 421 and 423 Figure 1a to Figure 1b In an example, the wearable device may guide the external electronic device 123. Figure 1a to Figure 1b Additional options for external electronic devices 123.

[0095] As described above, the wearable device according to the embodiment may obtain hierarchical categories (e.g., categories 410, 420, 421, 422, 423, 430, 431, 433, 434, 441, 442, 450, and 451) for classifying multiple external electronic devices based on metadata about multiple external electronic devices connected to the wearable device. Based on the hierarchical categories, the wearable device may identify at least one function that is commonly applicable to the external electronic devices. For example, in addition to the function of turning on or off the external electronic device, the wearable device may identify another function that is commonly applicable to the external electronic device. In addition to turning on or off the function of multiple external electronic devices, the wearable device may also support users to jointly control multiple external electronic devices by displaying a screen for performing another function.

[0096] In the following, reference will be made to Figure 5a to Figure 5c An operation of receiving an input instructing a wearable device according to an embodiment to add an external electronic device is described.

[0097] Figure 5a to Figure 5c An example of an operation in which a wearable device receives an input indicating a connection between an external electronic device and the wearable device according to an embodiment is shown. Figure 5a to Figure 5c A wearable device can be Figure 1a to Figure 1b The wearable device 101 and / or Figures 2 to 4 For example, the wearable device 101 of FIG. 1 may include Figure 5a to Figure 5c wearable devices. For example, Figure 5a to Figure 5c FoV 110 can be used with Figure 1a to Figure 1b The FoV corresponds to 110.

[0098] The wearable device according to an embodiment can be used to Figure 2 The wearable device may use a camera 240 of the wearable device to identify one or more external electronic devices included in the FoV 110. For example, the wearable device may identify one or more external electronic devices included in the FoV 110 based on object detection and / or object recognition. The wearable device may extract one or more feature points from the image of the FoV 110 obtained by the camera. The wearable device may identify the one or more external electronic devices based on the one or more feature points. The wearable device may identify the one or more external electronic devices by using a neural network trained to identify the external electronic device from an image.

[0099] The wearable device according to an embodiment may receive an input for notifying the location of at least one external electronic device (e.g., external electronic device 512) within FoV 110. For example, the wearable device may obtain the location of the external electronic device from the user independently of object detection and / or object recognition. Figure 5a to Figure 5c An operation of the wearable device according to an embodiment to identify the location of the external electronic device 512 based on a user input is described.

[0100] exist Figure 5a In the state 510 of the wearable device, the wearable device may display a visual object 514 superimposed within the FoV 110 for receiving the location of the external electronic device 512. The wearable device may communicate with the external electronic device 512 via a communication circuit (e.g., Figure 2 The communication circuit 250 of the embodiment of the present invention may identify the external electronic device 512 in the state 510 and / or before the state 510. The visual object 514 may be displayed on the display (e.g., Figure 2 The wearable device may display a semi-transparent visual object 514 based on a preset alpha value. The wearable device may display a preset text (e.g., "occurrence") for guiding the location where the external electronic device 512 can be selected within the visual object 514. The visual object 514 may be referred to as an anchor point for selecting an external electronic device or a location for selecting an external electronic device.

[0101] According to an embodiment, the wearable device may receive input for moving the visual object 514. For example, the wearable device may be based on the input for moving the visual object 514. Figure 6a to Figure 6b For example, the wearable device may receive input based on the user's movement described later. Figure 7a to Figure 7b Describes the user's voice to receive input. Figure 5aIn state 510 of the wearable device, the wearable device may receive input indicating movement of the visual object 514 along the path 516. For example, the user may move the visual object 514 so that the visual object 514 may be superimposed on the external electronic device 512 included in the FoV 110.

[0102] Figure 5b State 520 may be a state after the wearable device receives input indicating movement of visual object 514. Figure 5b In response to the input, the wearable device may display options 522, 524, and 526 for selecting the type of the external electronic device 512. For example, the options 522, 524, and 526 may be displayed by the wearable device to receive from the user a category of the external electronic device 512 superimposed with the visual object 514 within the FoV 110.

[0103] Reference Figure 5b Option 522 may include receiving an indication that the external electronic device 512 is classified as a standalone lighting device (e.g., classified as Figure 4 Option 524 may include an icon for receiving an input indicating that the external electronic device 512 is a lighting device fixed to the ceiling (e.g., classified into Figure 4 Option 526 may include an icon for receiving an input indicating that external electronic device 512 is a light bulb (e.g., categorized as a lighting device in category 422). Figure 4 Icon for input of lighting equipment in category 423).

[0104] In an embodiment, in response to an input indicating a selection of any one of the options 522, 524, and 526, the wearable device may classify the external electronic device 512 into at least one of the hierarchical categories. Figure 5b In state 520 , input is received from the user indicating a selection of option 522 .

[0105] Figure 5c State 530 may be a state in which the wearable device is based on Figure 5a to Figure 5b After the states 510 and 520 of the wearable device 512 are identified, a state of a screen 534 for controlling the external electronic device 512 is displayed. For example, in response to an input indicating selection of the external electronic device 512, the wearable device may display a visual object 532 in response to the input to indicate that the external electronic device 512 is selected. Figure 5c , an example of a visual object 532 based on a rectangular line is shown, but the embodiment is not limited thereto. For example, the visual object 532 may have a shape of a contour of the external electronic device 512. For example, the visual object 532 may have a polygonal shape that at least partially overlaps the external electronic device 512.

[0106] The wearable device according to an embodiment may display a screen 534 on at least a portion of the FoV 110 in response to an input indicating selection of the external electronic device 512. The wearable device may display one or more functions applicable to the external electronic device 512 selected by the input within the screen 534. The one or more functions applicable to the external electronic device 512 may be identified based on metadata about the external electronic device 512 and / or one or more categories including the external electronic device 512.

[0107] Reference Figure 5c , the wearable device may display an icon 534-1 and / or text 534-2 indicating the type of the external electronic device 512 in the screen 534. The wearable device may display a button 534-3 for switching the external electronic device 512 between an active state and an inactive state in the screen 534. The wearable device may display a visual object 534-4 (such as a slider) for adjusting the amount of light of the external electronic device 512 in the screen 534. The wearable device may display a visual object 534-5 (such as a palette) for adjusting the color of the light output from the external electronic device 512 in the screen 534. The wearable device may display a visual object 534-6 (such as a slider) for adjusting the color temperature in the screen 534. The wearable device according to an embodiment may control the external electronic device 512 in response to an input based on the screen 534. For example, the wearable device may send a signal indicating the input to the external electronic device 512 in response to the input.

[0108] Although exemplary operations of the wearable device related to the external electronic device 512 have been described, the embodiments are not limited thereto. For example, based on an input indicating selection of another external electronic device that is viewable through the FoV 110 and is different from the external electronic device 512, the wearable device may display a screen similar to the screen 534 for executing a function applicable to the other external electronic device. For example, in response to an input indicating selection of at least one of the visual objects 151, 152, and 153 superimposed on at least a portion of the FoV 110, the wearable device may display a screen for executing a function applicable to at least one external electronic device corresponding to at least one visual object selected by the input.

[0109] As described above, the wearable device according to the embodiment may display a UI (such as screen 534) for controlling the external electronic device based on an input indicating the selection of the external electronic device. In response to the input based on the UI, the wearable device may send a signal for controlling the external electronic device. When the number of external electronic devices selected by the user increases to two or more, the wearable device may display functions that are commonly applicable to the external electronic devices selected by the user by changing the UI. In addition, the wearable device may guide the user to additionally select the external electronic device based on the homogeneity of the external electronic device selected by the user.

[0110] In the following, reference will be made to Figure 6a to Figure 6b and / or Figure 7a to Figure 7b Describes an exemplary operation of receiving an input instructing a wearable device according to an embodiment to select two or more external electronic devices.

[0111] Figure 6a to Figure 6b An example of an operation in which the wearable device 101 receives an input indicating a selection of at least one of the external electronic devices based on the motion of the user according to an embodiment is shown. Figure 6a to Figure 6b The wearable device 101 may be the wearable device 101 of FIG. 1 and Figures 2 to 4 and / or Figure 5a to Figure 5c For example, the wearable device 101 of FIG. 1 may include Figure 6a to Figure 6b Wearable device 101. Figure 6a The FoV 110 of FIG. 6c may include Figure 1a to Figure 1b FoV110.

[0112] Figure 6a to Figure 6b States 610 and 620 may be states in which the wearable device has entered a mode for receiving input indicating selection of one or more external electronic devices (e.g., a multiple selection mode). For example, the wearable device may enter the mode in response to an input by pressing the wearable device for more than a preset duration (e.g., a long press) or an input by tapping the wearable device a preset number of times (e.g., twice) within a preset duration (e.g., a double click).

[0113] Figure 6a FIG. 6 shows an exemplary state 610 in which the wearable device 101 recognizes the movement of a body part (eg, hand 612) of a user according to an embodiment. The wearable device 101 may recognize the movement of a body part (eg, hand 612) of a user by using a camera (eg, Figure 2 The wearable device 101 may obtain an image including the hand 612 using a camera 240 of the wearable device 101. The wearable device 101 may obtain data indicating the movement of the hand 612 based on changes in the position and / or size of the hand 612 within the obtained image. The data may include the path of the hand 612 and / or the fingers included in the hand 612.

[0114] The wearable device 101 according to an embodiment may receive input indicating selection of one or more external electronic devices based on a path of a body part such as the hand 612. Figure 6a In the state 610 of the wearable device 101, the wearable device 101 may display a visual object 614 indicating a path of the hand 612 within the FoV 110. The visual object 614 may indicate a path based on line tracking while the wearable device tracks the path of the hand 612. The wearable device 101 according to an embodiment may select the external electronic devices 121 and 122 included in the path within the FoV 110 based on recognizing a path having a closed curve shape, such as the visual object 614. For example, the movement of the hand 612 moving along the path having a closed curve shape, such as the visual object 614, may be a gesture for selecting the external electronic devices 121 and 122.

[0115] Figure 6b An exemplary state 620 is shown in which the wearable device 101 receives an input indicating a selection of at least one external electronic device based on an external electronic device (such as a pointing device 622) according to an embodiment. The pointing device 622 may be connected to the wearable device 101 by wire or wirelessly to receive a gesture of a user wearing the wearable device 101. The user may control the wearable device 101 by moving the pointing device 622 or pressing a button of the pointing device 622. For example, the pointing device 622 may be an electronic device for remotely controlling another electronic device (e.g., the wearable device 101) different from the pointing device 622, such as Figure 4 Category 450.

[0116] The wearable device 101 according to an embodiment may receive an input detected by the pointing device 622 and indicating a selection of at least one external electronic device. Figure 6b , the wearable device 101 may display a visual object 624 indicating a path of the pointing device 622. The wearable device 101 may receive an input indicating a selection of the external electronic devices 121, 122, and 131 included in the closed curve within the FoV 110 based on the visual object 624 having the shape of the closed curve.

[0117] The wearable device 101 may not only receive input indicating selection of a plurality of external electronic devices by using the pointing device 622 and / or the hand 612, but may also receive input indicating exclusion of at least one of a plurality of pre-selected external electronic devices. For example, the wearable device 101 may recognize the movement of the pointing device 622 moving along the path of the visual object 626 after receiving a first input indicating selection of the external electronic devices 121, 122, and 131 (such as the visual object 624). The wearable device 101 may receive a second input indicating exclusion of the external electronic device 131 specified by the path of the visual object 626 based on the movement of the pointing device 622 moving along the path of the visual object 626. After sequentially receiving the first input and the second input, the wearable device 101 may determine that the user has selected the external electronic devices 121 and 122.

[0118] As described above, the wearable device 101 according to the embodiment may receive an input indicating a selection of one or more external electronic devices based on a movement of the user's hand 612. The wearable device 101 may identify functions that are commonly applicable to multiple external electronic devices based on recognizing that multiple external electronic devices are selected through the input. The wearable device 101 may identify functions that are commonly applicable to multiple external electronic devices based on hierarchical categories (e.g., Figure 4 The wearable device 101 can support the user to jointly control multiple external electronic devices based on functions by displaying the identified functions in the FoV 110.

[0119] The wearable device 101 receives an input indicating a selection of an external electronic device, which is not limited to Figure 6a to Figure 6b Hereinafter, reference will be made to Figure 7a to Figure 7b An operation of the wearable device 101 receiving an input indicating a selection of an external electronic device based on a motion of the wearable device 101 and / or a user's voice according to an embodiment is described.

[0120] Figure 7a to Figure 7b FIG. 1 shows an example of an operation in which the wearable device 101 receives an input indicating a selection of at least one of the external electronic devices based on the user's motion and voice according to an embodiment. The wearable device 101 of FIG. 1 may include Figure 7a to Figure 7b Wearable device 101. Figure 7a to Figure 7b The wearable device 101 may be Figures 2 to 4 and Figure 5a to Figure 5c Wearable devices and / or Figure 6a to Figure 6b An example of a wearable device 101. Similar to Figure 6a and Figure 6b The states 610 and 620, Figure 7a to Figure 7bStates 710 and 720 may be states in which the wearable device has entered a mode for receiving an input indicating a selection of one or more external electronic devices.

[0121] The wearable device 101 according to an embodiment may include a sensor (e.g., Figure 2 Sensors 260) to track eye gaze and / or head movement of a user wearing the wearable device 101. The wearable device 101 may include a microphone (e.g., Figure 2 The wearable device 101 may extract speech 714 from the audio signal. The wearable device 101 may extract speech 714 from the audio signal based on an algorithm and / or a neural network for speech-to-text (STT).

[0122] According to an embodiment, the wearable device 101 may recognize a preset keyword from the audio signal. For example, the wearable device 101 may recognize a preset keyword in the voice 714. The preset keyword may include a demonstrative pronoun, such as Figure 7a The wearable device 101 may identify the movement of the user wearing the wearable device 101 (e.g., changes in the user's eye gaze and / or movement of the user's head) in response to recognizing a noun (such as a demonstrative pronoun) for selecting one or more external electronic devices from the voice 714.

[0123] Reference Figure 7a In response to recognizing a preset keyword (e.g., keyword 716) from the audio signal, the wearable device may receive input indicating selection of at least one external electronic device based on a time period during which the preset keyword is recognized, a user's eye gaze detected by a sensor, and / or a movement of the user's head. For example, the wearable device 101 may obtain data indicating the user's eye gaze and / or the movement of the wearable device 101 during the time period during which the keyword 716 is received. The wearable device 101 may display a visual object 712 superimposed within the FoV 110 based on the data obtained during the time period. For example, the visual object 712 may indicate a path of the wearable device 101 through the user's eye gaze and / or the user's head movement during the time period.

[0124] The wearable device 101 according to an embodiment may receive an input indicating a selection of one or more external electronic devices based on the visual object 712 indicating a received motion within a time period in which the keyword 716 is received. For example, the wearable device 101 may determine the external electronic device 121 included in the closed curve indicated by the visual object 712 within the FoV 110 and the external electronic device corresponding to the visual object 153 as the external electronic device selected by the user.

[0125] The wearable device 101 according to the embodiment may collectively control an external electronic device selected by the keyword 716 in the voice 714 based on another keyword included in the voice 714 and different from the keyword 716. Figure 7a In response to receiving a voice 714 such as “Turn all of those off”, the wearable device 101 may not only recognize a keyword (e.g., “all of those”) in the voice 714 for selecting an external electronic device, but may also recognize a keyword (e.g., “turn off”) for selecting a function that is commonly applicable to the external electronic devices. For example, in response to receiving the voice 714, the wearable device 101 may collectively turn off the external electronic devices (e.g., the external electronic device 121 and the external electronic device corresponding to the visual object 153) selected by the keyword 716 in the voice 714.

[0126] Figure 7b FIG. 1 shows an example of a wearable device 101 according to an embodiment recognizing a voice (eg, Figure 7a 714) while performing a function based on a keyword included in the speech. The wearable device 101 may obtain text from the audio signal based on the STT. The wearable device 101 may respond to identifying a keyword indicating a specific function (e.g., the keyword described above) in the text. Figure 7a Described in the description of selecting one or more external electronic devices) keywords to perform specific functions.

[0127] Reference Figure 7b , the wearable device 101 may obtain text indicating speech from the audio signal from the time point t1. For example, the wearable device 101 may perform conversion between speech and text (eg, STT) within a time period 721 starting from the time point t1. Figure 7b , a preset keyword notifying the start of a voice command for controlling the wearable device 101 may be identified within a time period 722 between time point t1 and time point t2. For example, the preset keyword notifying the start of the voice command may be "Hi bixby", or may include text input by the user (e.g., "Hi (text)").

[0128] Reference Figure 7b , the wearable device 101 may obtain text indicating a voice command that continues after the preset keyword from time point t2 based on recognizing the preset keyword within the time period 722. For example, the wearable device 101 may obtain a voice command from the audio signal within the time period 723 after the time point t2. The time period 725 after the time point t2 may be a time period in which the wearable device 101 performs the operation indicated by the voice command.

[0129] Reference Figure 7b, the wearable device 101 may identify keywords (such as demonstrative pronouns) indicating the selection of one or more external electronic devices from the text in a time period 724 between the time point t3 and the time point t4. In the time period 724, the wearable device 101 may detect the user's movement, as described above with reference to Figure 7a The wearable device 101 may identify one or more external electronic devices based on the detected motion.

[0130] Reference Figure 7b , within the time period 723 for receiving the voice command, the wearable device 101 may identify the function to be performed by one or more external electronic devices selected in the time period 724 based on the keywords recognized in the remaining time period (e.g., the time period between time point t2 and time point t3, and the time period between time point t4 and time point t5) except for the time period 724 for receiving the demonstrative pronoun. For example, the wearable device 101 may send a signal for deactivation to one or more external electronic devices selected based on the time period 724 in response to recognizing a preset keyword (such as "off") for deactivating the external electronic device in the remaining time period. In response to the signal, the one or more external electronic devices may be deactivated collectively. The signal sent by the wearable device 101 may be executed after time point t5 within the time period 725.

[0131] Although an embodiment for recognizing English speech has been described, the embodiment is not limited thereto. For example, the wearable device 101 may control one or more external electronic devices connected to the wearable device 101 based on Korean speech, such as “ (Shut all of these down!)”.

[0132] As described above, the wearable device 101 according to the embodiment can receive input indicating the selection of one or more external electronic devices and the joint control of the selected one or more external electronic devices by using an audio signal and the user's movement. The wearable device 101 can jointly control one or more external electronic devices based on the input. Although it has been described that the wearable device 101 jointly deactivates the operation of the external electronic device based on the user's voice, the embodiment is not limited to this. In response to recognizing text from the user's voice, the wearable device 101 may send a signal to multiple external electronic devices for executing a function corresponding to the text, wherein the text corresponds to any one of the functions commonly applicable to the multiple external electronic devices selected by the user.

[0133] In the following, reference is made to Figures 8a to 8c , the wearable device 101 according to the embodiment will be described based on the above reference Figure 6a to Figure 6b and / or Figure 7a to Figure 7bThe described external electronic device selected by the user's motion and / or voice guides an operation of selecting another external electronic device similar to the external electronic device selected by the user.

[0134] Figures 8a to 8c FIG. 1 shows an example of an operation in which a wearable device displays a visual object for guiding selection of at least one external electronic device according to an embodiment. The wearable device 101 of FIG. 1 may include Figures 8a to 8c wearable device. Figures 8a to 8c A wearable device can be Figures 2 to 4 and Figure 5a to Figure 5c Wearable devices and / or Figure 6a to Figure 6b and Figure 7a to Figure 7b An example of a wearable device 101.

[0135] Figure 8a The state 810 after the wearable device according to the embodiment receives an input indicating the selection of the external electronic device 121 within the FoV 110 is exemplarily shown. Figure 6a to Figure 6b and / or Figure 7a to Figure 7b The wearable device may receive input by processing the user's motion and / or voice described in the embodiment. In response to the input indicating the selection of the external electronic device 121, the wearable device may display a visual object 815 superimposed on the external electronic device 121 within the FoV 110. An example is shown in which the wearable device displays the visual object 815 having the outline shape of the external electronic device 121 within the FoV 110, but the embodiment is not limited thereto. For example, the visual object 815 may have a shape and / or color for emphasizing the external electronic device 121 within the FoV 110.

[0136] exist Figure 8a In state 810 of displaying visual object 815, the wearable device may receive input indicating selection of another external electronic device that is different from external electronic device 121 corresponding to visual object 815. For example, the wearable device may receive input indicating selection of other external electronic devices other than external electronic device 121 (e.g., external electronic devices corresponding to each of external electronic devices 122, 123, 131, 132, 141, 142, 161, and 171 and visual objects 151, 152, and 153).

[0137] Figure 8b The state 820 may be exemplarily shown in Figure 8aThe wearable device may be in a state after receiving an input indicating selection of visual object 152 in state 810. According to an embodiment, the wearable device may visualize that visual object 152 is selected in response to the input indicating selection of visual object 152. For example, the wearable device may indicate that visual object 152 is selected by displaying visual object 821 superimposed on visual object 152.

[0138] exist Figure 8b In state 820 of the embodiment, the wearable device according to the embodiment may display at least one visual object (e.g., visual objects 822, 823, and 824) that guides selection of another external electronic device similar to the multiple external electronic devices selected by the user (e.g., the external electronic device 121 and the external electronic device corresponding to the visual object 152). Based on recognizing that the user has selected multiple external electronic devices (e.g., the external electronic device 121 and the external electronic device corresponding to the visual object 152), the wearable device may identify the category of each of the multiple external electronic devices. For example, the wearable device may select Figure 4 The wearable device may identify a category (e.g., category 421) of the external electronic device 121 on which the visual object 815 is superimposed in the hierarchical categories of the wearable device. The wearable device may identify a category (e.g., category 423) of the external electronic device on which the visual object 152 is superimposed.

[0139] The wearable device according to an embodiment may identify an external electronic device within a category including all external electronic devices selected by the user (e.g., an external electronic device homogeneous with the external electronic device selected by the user). For example, the external electronic device identified by the wearable device may be the union of each of the categories classified in the external electronic devices selected by the user. For example, the external electronic device identified by the wearable device may be identified based on a higher category that the categories have in common.

[0140] After the user selects the external electronic device 121 and the external electronic device corresponding to the visual object 152 Figure 8b In an embodiment of the present invention, the wearable device may identify and classify Figure 4 For example, the wearable device may identify another external electronic device (e.g., an external electronic device corresponding to the visual object 153 and the external electronic device 122) classified into the same category as the external electronic device 121 selected by the user. For example, the wearable device may identify another external electronic device (e.g., an external electronic device corresponding to the visual object 151 and the external electronic device 122) classified into the same category as the external electronic device corresponding to the visual object 152 selected by the user.

[0141] The wearable device according to an embodiment may display a visual object indicating another external electronic device similar to the external electronic device selected by the user (e.g., an external electronic device homogeneous with the external electronic device selected by the user). The wearable device may display the visual object as associated with the other external electronic device. The visual object may be displayed based on a different shape and / or color from the visual objects 815 and 821 displayed in association with the external electronic device selected by the user to guide the selection of the other external electronic device.

[0142] Reference Figure 8b , the wearable device may display the visual object 825 as being associated with another external electronic device (e.g., an external electronic device corresponding to the visual object 153) that is classified into the same category as the external electronic device 121 selected by the user. The visual object 815 displayed as being associated with the external electronic device 121 may be displayed based on a first type indicating that it is selected by the user. At the same time, the visual object 825 displayed as being associated with the visual object 153 may be displayed based on a second type different from the first type for guiding an additional selection of the user.

[0143] Reference Figure 8b , the wearable device may display visual objects 823 and 822 by overlaying with another electronic device (e.g., the external electronic device 122 and the external electronic device corresponding to the visual object 151), which is classified into the same category as the electronic device selected by the user and corresponding to the visual object 152. The wearable device according to an embodiment may emphasize other electronic devices similar to the external electronic device selected by the user within the FoV 110 by displaying the visual objects 822, 823, and 825 together with the visual objects 815 and 821. The wearable device according to an embodiment may display a visual object 824 for selecting all other external electronic devices. The visual object 824 may include a preset text (e.g., "Select the remaining objects") for guiding the selection of all other external electronic devices.

[0144] Figure 8c The state 830 may exemplarily show that upon receiving the indication Figure 8b In response to the input, the wearable device may replace visual objects 822, 823, and 825 with visual objects 831, 832, and 833 indicating that they are selected by the user. Figure 8b The state 820 is displayed to guide the user's selection. Figure 8c In state 830, the wearable device can determine that all external electronic devices 121 and 122 and external electronic devices corresponding to visual objects 151, 152 and 153 are selected by the user.

[0145] As described above, the wearable device according to the embodiment may display visual objects 815 and 821 of the first type indicating the selection of multiple external electronic devices based on an input indicating the selection of multiple external electronic devices. When displaying the visual objects 815 and 821 of the first type, the wearable device may display a visual object different from the first type (e.g., visual objects 822, 823, 824, and 825) for guiding the selection of another external electronic device homogeneous with the multiple external electronic devices. The wearable device may notify the user of another external electronic device homogeneous with the multiple external electronic devices by using a visual object different from the first type, or may guide the selection of the other external electronic device.

[0146] In the following, reference is made to Figure 9a to Figure 9b and / or Figures 10a to 10c , an example of an operation in which a wearable device displays a screen for executing functions commonly applicable to multiple external electronic devices selected by a user or visualizes data obtained from multiple external electronic devices will be described.

[0147] Figure 9a to Figure 9b FIG. 1 shows an example of a screen displayed by a wearable device based on a category of an external electronic device selected by a user according to an embodiment. The wearable device 101 of FIG. 1 may include Figure 9a to Figure 9b wearable device. Figure 9a to Figure 9b A wearable device can be Figures 2 to 4 and Figure 5a to Figure 5c Wearable devices and / or Figure 6a to Figure 6b , Figure 7a to Figure 7b and Figures 8a to 8c An example of a wearable device 101.

[0148] Figure 9a The wearable device exemplarily shows a state 910 in which the screen 911 is displayed in response to an input indicating selection of the external electronic device 121 and the external electronic device corresponding to the visual object 152. Figure 6a to Figure 6b and / or Figure 7a to Figure 7b In response to the input, the wearable device may display visual objects 901 and 902 indicating an external electronic device (e.g., electronic device 121 and an electronic device corresponding to visual object 152) selected by the user as being associated with the external electronic device.

[0149] exist Figure 9aIn state 910 of the wearable device, the wearable device may identify functions commonly applicable to the external electronic devices selected by the user. For example, the wearable device may identify functions commonly applicable to the external electronic devices based on metadata of the external electronic devices selected by the user and / or categories of each of the external electronic devices. The wearable device according to an embodiment may display a screen 911 for executing functions commonly applicable to the external electronic devices within the FoV 110.

[0150] Reference Figure 9a , the wearable device according to an embodiment may display visual objects 912, 913, 914, and 915 within screen 911 for executing functions that are commonly applicable to the external electronic devices selected by the user. For example, the wearable device may display a button (such as visual object 912) within screen 911 for commonly enabling or disabling the external electronic devices. Based on the fact that all external electronic devices selected by the user support adjustment of the amount of light by the user, the wearable device may display within screen 911 a visual object 913 including a slider for adjusting the amount of light of all external electronic devices. The wearable device may display within screen 911 a visual object 914 including a palette for adjusting the colors of all external electronic devices. The wearable device may display within visual object 914 a list of one or more colors commonly supported by the external devices. The wearable device may display within screen 911 a visual object 915 for adjusting the color temperature of all external electronic devices. Figure 9a In state 910, the wearable device may display visual objects 913, 914, and 915 for jointly adjusting the amount of light, color, and color temperature based on recognizing that all external electronic devices selected by the user support the function of adjusting the amount of light, color, and color temperature.

[0151] exist Figure 9a In state 910, the wearable device may control all external electronic devices selected by the user based on input associated with screen 911. In response to input to any one of visual objects 912, 913, 914, and 915 within screen 911, the wearable device may request execution of a function mapped to the visual object selected by the input to the external electronic device. For example, when an input for adjusting the amount of light based on visual object 913 is received, the wearable device may send a signal indicating the amount of light adjusted by the input to an external electronic device selected by the user (e.g., external electronic device 121 and an external electronic device corresponding to visual object 152). The external electronic device may operate based on the signal to have the amount of light adjusted by the user.

[0152] exist Figure 9aIn state 910 of the wearable device, the wearable device may change screen 911 based on recognizing that the external electronic device selected by the user is changed. For example, when the user additionally selects an external electronic device, the wearable device may stop displaying at least one of visual objects 912, 913, 914, and 915 within screen 911 based on a function applicable to the added external electronic device. For example, when the user excludes selection of a specific external electronic device, the wearable device may change screen 911 based on functions commonly applicable to one or more external electronic devices finally selected by the user.

[0153] Reference Figure 9b , showing by way of example Figure 9a In state 910, the wearable device changes state 920 of screen 911 in response to input indicating additional selection of the external electronic device 123. In state 920, in response to input indicating selection of the external electronic device 123, the wearable device may display a visual object 921 superimposed on the external electronic device 123 viewable through FoV 110. Each of the visual objects 901, 902, and 921 may be displayed as being associated with the external electronic device selected by the user. The wearable device may display screen 911 based on functions commonly applicable to the external electronic devices corresponding to the visual objects 901, 902, and 921. As the number of external electronic devices selected by the user increases, the number of functions commonly applicable to the external electronic devices may decrease. The wearable device may change screen 911 based on changes in functions commonly applicable to the external electronic devices.

[0154] exist Figure 9b In the example of , when the external electronic device 123 additionally selected by the user does not support adjusting the color and color temperature, the wearable device may stop displaying the visual objects 914 and 915 for adjusting the color and color temperature in the screen 911. The wearable device may keep displaying other visual objects 912 and 913 different from the visual objects 914 and 915 in the screen 911 regardless of stopping the display of the visual objects 914 and 915. For example, the wearable device may display the visual object 913 for adjusting the amount of light, and the visual object 912 for controlling the power of the external electronic device corresponding to the visual objects 901, 902, and 921.

[0155] As described above, in addition to jointly turning on or off the function of the external electronic device (such as visual object 912) selected by the user, the wearable device according to the embodiment may also display visual objects (e.g., visual objects 913, 914, and 915) for performing other functions commonly applicable to the external electronic devices. Since the wearable device lists the functions commonly applicable to the external electronic devices based on the metadata and / or categories of the external electronic devices, the user of the wearable device can simplify the common control of the external electronic devices.

[0156] In the following, reference will be made to Figures 10a to 10c An example of an operation in which a wearable device processes information received from an external electronic device selected by a user according to an embodiment is described.

[0157] Figures 10a to 10c FIG. 1 shows an example of a screen displayed by a wearable device based on an external electronic device selected by a user according to an embodiment. The wearable device 101 of FIG. 1 may include Figures 10a to 10c wearable device. Figures 10a to 10c A wearable device can be Figures 2 to 4 , Figure 5a to Figure 5c and Figure 9a to Figure 9b Wearable devices and / or Figure 6a to Figure 6b , Figure 7a to Figure 7b and Figures 8a to 8c An example of a wearable device 101.

[0158] Fig.10a The state 1010 may be an exemplary state in which the wearable device displays a screen 1011 based on an input indicating a selection of external electronic devices 141 and 142 for providing air-related functions. In the state 1010, the wearable device may display a screen 1011 for interacting with the external electronic devices 141 and 142 selected by the input within the FoV 110. The wearable device may display a visual object 1012 including a button for jointly turning on or off the external electronic devices 141 and 142 selected by the input within the screen 1011.

[0159] The wearable device according to the embodiment may obtain data measured from each of the external electronic devices 141 and 142 in response to an input indicating a selection of the external electronic devices 141 and 142. The wearable device may display the data obtained from the external electronic devices 141 and 142 within the screen 1011. For example, the wearable device may display visual objects 1014 and 1015 indicating data (e.g., air quality) measured from the external electronic device 141 as an air purifier within the screen 1011. An example in which the wearable device displays visual objects 1014 and 1015 indicating air quality measured at two points from the external electronic device 141 is shown, but the embodiment is not limited thereto.

[0160] The wearable device according to the embodiment may display a combination of data measured in each of the external electronic devices 141 and 142 in the screen 1011. For example, the wearable device may display a visual object 1013 indicating a combination of power consumption in the screen 1011 based on receiving data indicating power consumption of each of the external electronic devices 141 and 142. Fig.10a, the wearable device may display a combination of power consumption of the external electronic devices 141 and 142 based on a unit of power usage such as watts. Embodiments are not limited thereto, and for example, the wearable device may output a combination of power consumption in the form of an electricity bill (e.g., "7,950KRW" within the visual object 1013).

[0161] The operation performed by the wearable device based on the input indicating the selection of multiple external electronic devices is not limited to the common control of the external electronic devices or the visualization of data received from the external electronic devices. The wearable device may display a visual object for performing a collaborative function of multiple external electronic devices selected by the user. For example, a collaborative function may be performed based on the individual control of multiple external electronic devices.

[0162] Fig.10b State 1020 shows a state in which the wearable device displays a screen 1023 based on an input indicating selection of the external electronic device 123 as a lighting device and the external electronic device 161 as a game controller. In state 1020, in response to the input, the wearable device may display visual objects 1021 and 1022 as being associated with each of the external electronic devices 123 and 161. In state 1020, the wearable device may display a screen 1023 for visualizing functions commonly applicable to the external electronic devices 123 and 161 selected by the input. The wearable device may display a visual object 1025 for executing a function assigned to a group of external electronic devices 123 and 161 and a visual object 1024 for turning the external electronic devices 123 and 161 on or off collectively within the screen 1023.

[0163] For example, the wearable device may display a visual object 1025 within the screen 1023, the visual object 1025 being related to all external electronic devices 123 and 161 selected by the input and being used to individually control the external electronic devices 123 and 161. Fig.10b , the wearable device may display a visual object 1025 for performing a function of controlling light of an external electronic device 123 as a lighting device based on the external electronic device 161 as a game controller. In response to an input indicating selection of the visual object 1025, the wearable device may transmit information for controlling the external electronic device 123 to the external electronic device 161.

[0164] Reference Fig.10c, an exemplary state 1030 of a wearable device display screen 1034 is shown based on an input indicating a selection of the external electronic devices 121, 123, and 171. In state 1030, the wearable device may emphasize the external electronic devices 121, 123, and 171 selected by the input based on visual objects 1031, 1032, and 1033 superimposed on the external electronic devices 121, 123, and 171 viewable through the FoV 110. In state 1030, the wearable device may display visual objects 1036 and 1037 for executing a function (e.g., scheduling light off) assigned to a group of external electronic devices 121, 123, and 171, and an enabled visual object 1035 for jointly controlling the external electronic devices 121, 172, and 171 selected by the input. For example, in response to an input indicating a selection of the visual object 1036, the wearable device may perform a function of controlling another external electronic device (e.g., external electronic devices 121 and 123) included in the group during a time period identified by the external electronic device 171 as a watch. For example, in response to an input indicating a selection of the visual object 1037, the wearable device may perform a function of controlling the amount of light of the external electronic devices 121 and 123 during a time period identified by the external electronic device 171.

[0165] The operation of the wearable device controlling the external electronic device selected by the user according to the embodiment is not limited to the above-mentioned embodiment. For example, the wearable device may display the state of charge (SOC) of each battery included in the external electronic device individually or in combination based on the input of selecting the external electronic device including the battery. Figure 1a to Figure 1b In an embodiment of the present invention, in response to an input indicating selection of the external electronic device 131 as a surveillance camera and the external electronic device 132 as a television, the wearable device may display a visual object for performing a function for displaying an image and / or video obtained by the external electronic device 131 based on the external electronic device 132. When a function commonly applicable to the external electronic device selected by the user is not identified, the wearable device may display a message within the FoV 110 indicating that there is no function commonly applicable to the external electronic devices.

[0166] In the following, reference will be made to Figures 11 to 15 The operation of the above-mentioned wearable device is described based on an exemplary flowchart.

[0167] Fig.11 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown. Fig.11 The wearable device may include the wearable device described based on the above figures. For example, Fig.11 The operation can be performed from Figure 1 to Figure 2 The wearable device 101 and / or Figure 2 Executed by processor 210.

[0168] Reference Fig.11 In operation 1110, the wearable device according to the embodiment may recognize an input indicating a selection of at least one of a plurality of external electronic devices. The wearable device may recognize the external electronic device based on metadata received from the server and / or the external electronic device. The wearable device may establish a communication link for transmitting and / or receiving signals with the external electronic device based on the metadata. The wearable device may establish a communication link for transmitting and / or receiving signals with the external electronic device based on the metadata. Figure 6a and Figure 6b The wearable device may be configured to recognize input based on the movement of a user wearing the wearable device described above. Figure 7a and Figure 7b Describes the user's voice and / or the user's movements to recognize input.

[0169] In response to the input of operation 1110, in operation 1115, the wearable device according to the embodiment may determine whether two or more external electronic devices have been selected by the input. Based on operation 1110, when an input indicating the selection of an external electronic device is received (1115-No), in operation 1120, the wearable device according to the embodiment may display a screen for interacting with the external electronic device selected by the input of operation 1110. For example, similar to Figure 5c The wearable device may be configured to display a screen 534 of the wearable device at the FoV (e.g., Figure 1a to Figure 1b A screen for controlling an external electronic device selected by input is displayed within the FoV 110).

[0170] When an input indicating selection of a plurality of external electronic devices is received based on operation 1110 (1120-Yes), in operation 1125, the wearable device according to the embodiment may identify a category including all two or more external electronic devices selected by the input. The wearable device may identify a category including each of the external electronic devices based on metadata about the external electronic device selected by the input. Based on the category and / or a higher category in the category, the wearable device may identify a category including all external electronic devices selected by the input.

[0171] Reference Fig.11 In operation 1130, the wearable device according to the embodiment may display one or more visual objects for distinguishing the external electronic device included in the category of operation 1125 from a plurality of external electronic devices. The one or more visual objects may be displayed within the FoV of the wearable device as being associated with the external electronic device included in the category of operation 1125. The one or more visual objects may include a visual object (such as a cursor) for indicating the selection of the external electronic device by the input of operation 1110. Figure 8bThe one or more visual objects may include at least one visual object (such as a Figure 8b visual objects 822, 823 and 825).

[0172] Reference Fig.11 In operation 1135, the wearable device according to the embodiment may display a screen based on functions commonly applicable to two or more external electronic devices selected by the input of operation 1110. Functions commonly applicable to the external electronic devices selected by the input may be identified based on metadata of the external electronic devices and / or categories including the external electronic devices. For example, the screen may include Figure 9a to Figure 9b Screens 911 and 921 and / or Figures 10a to 10c 1011, 1023, and 1034 of operation 1110. In response to the input related to the screen, the wearable device can fully control the external electronic device selected by the input of operation 1110.

[0173] Figure 12a to Figure 12b An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown. Figure 12a to Figure 12b The wearable device may include the wearable device described based on the above figures. For example, Figure 12a to Figure 12b The operation can be performed from Figure 1 to Figure 2 The wearable device 101 and / or Figure 2 Executed by processor 210. Figure 12a to Figure 12b The operation can be Fig.11 operation related.

[0174] Reference Fig.12a In operation 1210, the wearable device according to the embodiment may obtain metadata indicating the external electronic device. The wearable device may obtain the metadata through a server connected to the wearable device. The wearable device may obtain the metadata based on direct communication between the external electronic device and the wearable device. The metadata may include information for use based on the above reference. Figure 4 The hierarchical categories described classify information about external electronic devices.

[0175] Reference Fig.12a In operation 1215, the wearable device according to the embodiment may determine whether at least one virtual object mapped to at least one external electronic device is recognized. When at least one virtual object is recognized (1215-Yes), Fig.12a In operation 1220, the wearable device according to the embodiment may display the recognized at least one virtual object in the display. The at least one virtual object may include Figure 1b151, 152, and 153. When at least one virtual object is not recognized (1215-No), the wearable device may bypass display of at least one virtual object based on operation 1220.

[0176] Reference Fig.12a In operation 1225, the wearable device according to the embodiment may determine whether an input indicating a selection of an external object including at least one virtual object and / or an external electronic device is received. Fig.12a The input received by operation 1225 may include Fig.11 The input of operation 1110. Fig.12a , in operation 1230, the wearable device according to the embodiment may determine whether the external object selected by the input corresponds to at least one of the external electronic devices. For example, the wearable device may determine whether the external object selected by the input is the external electronic device indicated by the metadata of operation 1210. For example, the wearable device may determine whether the external object selected by the input corresponds to the virtual object displayed based on operation 1220. When the external object selected by the input does not correspond to any external electronic device (1230-No), in Fig.12a In operation 1235, the wearable device according to an embodiment may display a visual object indicating that the external object does not correspond to the external electronic device. Fig.12a In operation 1235, the wearable device may display a screen for recognizing the external object selected by the input as an external electronic device.

[0177] When the external object selected by the input corresponds to at least one of the external electronic devices (1230-Yes), Figure 12b In operation 1240, the wearable device according to the embodiment may determine whether a plurality of external electronic devices are selected by input. Fig.11 Operation 1115 is performed Figure 12b When an external electronic device is selected by the input (1240-No), in operation 1245, the wearable device according to the embodiment may display a screen for interacting with the external electronic device selected by the input. Fig.11 Operation 1120 is performed Figure 12b Operation 1245.

[0178] When a plurality of external electronic devices are selected by input (1240-Yes), Figure 12b In operation 1250, the wearable device according to the embodiment may identify a category including all of the plurality of external electronic devices selected by the input based on the metadata. Fig.11The category is identified by operation 1125 of the wearable device. The category identified by the wearable device based on operation 1250 may also include another external electronic device connected to the wearable device and a plurality of external electronic devices selected by the input.

[0179] Reference Figure 12b In operation 1255, the wearable device according to the embodiment may display at least one virtual object as being associated with at least one visual object and / or external electronic device included in the category. For example, the wearable device may display at least one visual object superimposed on the virtual object and / or external electronic device selected by the input of operation 1225. Since the category identified based on operation 1250 also includes another external electronic device connected to the wearable device and multiple external electronic devices selected by the input of operation 1225, the wearable device may also display a visual object indicating other external electronic devices. Fig.11 Operation 1130 is similarly performed Figure 12b Operation 1255.

[0180] Reference Figure 12b In operation 1260, the wearable device according to the embodiment may display a screen based on functions applicable to all of the plurality of external electronic devices selected by the input. The function 1125 displayed on the screen may be identified based on the category of the operation. Fig.11 Operation 1135 is similarly performed Figure 12b Operation 1260.

[0181] Fig.13 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown. Fig.13 The wearable device may include the wearable device described based on the above figures. For example, Fig.13 The operation can be performed from Figure 1 to Figure 2 The wearable device 101 and / or Figure 2 Executed by processor 210. Fig.13 The operation can be Fig.11 and Figure 12a to Figure 12b operation related. Fig.13 Operation and Fig.11 and Figure 12a to Figure 12b The operations are not indivisible.

[0182] Reference Fig.13 In operation 1310, the wearable device according to the embodiment may use a microphone (e.g., Figure 2 The wearable device may initiate receiving an audio signal from the user based on a preset keyword (such as a microphone 270) that notifies the user of the start of a voice command. Figure 7bInitiating reception of an audio signal by the wearable device based on operation 1310 may include initiating an operation of obtaining text indicating a voice included in the audio signal.

[0183] Reference Fig.13 In operation 1320, the wearable device according to the embodiment may obtain a preset keyword for selecting one or more external electronic devices from the audio signal. The preset keyword may include a demonstrative pronoun (e.g., "this" and / or "that") for selecting a single external electronic device, or a demonstrative pronoun (e.g., "all of these", "these" and / or "those") for selecting multiple external electronic devices.

[0184] Reference Fig.13 In operation 1330, the wearable device according to the embodiment may obtain (for example, Figure 2 The wearable device may obtain data indicating the user's gaze (e.g., capturing an image of the user's eyes) detected by the sensor during a time period in which the preset keyword of operation 1320 is received. For example, the wearable device may detect the movement of the head of the user wearing the wearable device based on the data indicating the movement of the wearable device.

[0185] Reference Fig.13 In operation 1340, the wearable device according to the embodiment may identify one or more external electronic devices selected by the user based on the data obtained according to operation 1330. Fig.13 In operations 1320 and 1330 of the embodiment, the wearable device may receive an input indicating a selection of one or more external electronic devices based on a motion performed by the user within a time period when the user says a preset keyword in operation 1320. The input may be related to Fig.11 The input of operation 1110 and / or Fig.12a For example, the wearable device may perform the operation based on the one or more external electronic devices selected according to operation 1340. Fig.11 At least one of operations 1120, 1125, 1130, and 1140. For example, the wearable device may perform the operation based on the one or more external electronic devices selected according to operation 1340. Figure 12a to Figure 12b At least one of operations 1230, 1235, 1240, 1245, 1250, 1255 and 1260.

[0186] Fig.14 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown. Fig.14 The wearable device may include the wearable device described based on the above figures. For example, Fig.14The operation can be performed from Figure 1 to Figure 2 The wearable device 101 and / or Figure 2 Executed by processor 210. Fig.14 The operation can be Fig.11 and Figure 12a to Figure 12b operation related.

[0187] Reference Fig.14 , in operation 1410, the wearable device according to an embodiment may receive an input indicating selection of a first external electronic device and a second external electronic device. Fig.14 The input may include Fig.11 The input of operation 1110 and / or Fig.12a Input of operation 1225.

[0188] Reference Fig.14 , in operation 1420, the wearable device according to the embodiment may identify a third external electronic device in a category including both the first external electronic device and the second external electronic device. The third external electronic device may include the first external electronic device and the second external electronic device of operation 1410. For example, the wearable device may identify a first category into which the first external electronic device is classified from first metadata about the first external electronic device. The wearable device may identify a second category into which the second external electronic device is classified from second metadata about the second external electronic device. The wearable device may identify the third external electronic device based on a higher category including the first category and the second category. Fig.14 Operation 1420 may be performed with Fig.11 Operation 1125 and / or Figure 12b Operation 1250 is related.

[0189] Reference Fig.14 In operation 1430, the wearable device according to the embodiment may display the first visual object and the second visual object as being displayed through the display (eg, Figure 2 The wearable device may be associated with a first external electronic device and a second external electronic device viewable through a display 230. In response to the input of operation 1410, the wearable device may perform operation 1430. The wearable device may display the first visual object and the second visual object by overlaying each of the first external electronic device and the second external electronic device viewable through the display.

[0190] Reference Fig.14, in operation 1440, the wearable device according to the embodiment may display at least one third visual object as being associated with at least one fourth external electronic device in the third external electronic device that is different from the first external electronic device and the second external electronic device. The at least one fourth external electronic device may be another external electronic device in the third external electronic device other than the first external electronic device and the second external electronic device. The at least one third visual object may have a shape and / or color different from the first visual object and the second visual object of operation 1430. For example, the wearable device may guide the selection of an external electronic device (e.g., at least one fourth external electronic device) similar to the first external electronic device and the second external electronic device selected by the input of operation 1410 based on the third visual object of operation 1440. Fig.14 At least one of operations 1420, 1430, and 1440 may be performed with Fig.11 Operation 1130 and / or Fig.12a Operation 1225 is related.

[0191] Fig.15 An example of a flowchart for describing the operation of a wearable device according to an embodiment is shown. Fig.15 The wearable device may include the wearable device described based on the above figures. For example, Fig.15 The operation can be performed from Figure 1 to Figure 2 The wearable device 101 and / or Figure 2 Executed by processor 210. Fig.15 The operation can be Fig.11 and Figure 12a to Figure 12b operation related.

[0192] Reference Fig.15 In operation 1510, the wearable device according to the embodiment may receive a first input indicating a selection of one or more first external electronic devices among a plurality of external electronic devices. The first input may be received based on recognition of a user's motion and / or a user's voice by the wearable device, similar to Fig.11 The input of operation 1110 and Fig.12a Input of operation 1225.

[0193] Reference Fig.15 In operation 1520, the wearable device according to the embodiment may identify one or more first functions applicable to one or more first external electronic devices. The first function may be identified based on a category including all of the one or more first external electronic devices. The wearable device may identify the first function based on metadata about the one or more first external electronic devices.

[0194] Reference Fig.15In operation 1530, the wearable device according to the embodiment may display one or more visual objects for performing one or more first functions. The wearable device may display the one or more visual objects for performing one or more first functions on a display such as a Figure 9a The wearable device according to the embodiment may display one or more visual objects by using another portion of the display in addition to the portion in which the one or more first external electronic devices selected by the first input of operation 1510 are viewed.

[0195] While displaying one or more visual objects of operation 1530, in operation 1540, the wearable device according to the embodiment may determine whether a second input indicating selection of at least one second external electronic device is received. The at least one second external electronic device may be different from the one or more first external electronic devices of operation 1510. When the second input is not received (1540-No), in operation 1570, the wearable device may control the one or more first external electronic devices in response to a third input about the one or more visual objects. Before receiving the second input, the wearable device may keep displaying one or more visual objects for performing one or more first functions based on operation 1530.

[0196] When a second input indicating selection of at least one second external electronic device is received (1540-Yes), the wearable device according to the embodiment may identify at least one second function applicable to the at least one second external electronic device in operation 1550. Similar to operation 1520, identifying the at least one second function by the wearable device may be performed based on metadata about the at least one second external electronic device and / or a category including the at least one second external electronic device.

[0197] Reference Fig.15 In operation 1560, the wearable device according to the embodiment may stop displaying at least one visual object related to a function different from at least one second function among the one or more visual objects. For example, the wearable device may stop displaying at least one visual object different from at least one second function of operation 1550 among the one or more visual objects displayed in the display based on operation 1530. Display of at least one visual object for performing at least one second function of operation 1550 among the one or more visual objects of operation 1530 may be maintained through operation 1560.

[0198] As described above, the wearable device according to the embodiment may guide the selection of a homogeneous external electronic device among multiple external electronic devices connected to the wearable device. In response to an input indicating the selection of multiple external electronic devices, the wearable device may recommend another external electronic device similar to the multiple external electronic devices to the user. The wearable device may display a screen for executing one or more functions commonly applicable to the multiple external electronic devices in response to the input.

[0199] According to an embodiment, a method for providing a user interface to select a wearable device that is applicable to at least one function of a plurality of external electronic devices may be required.

[0200] As described above, the wearable device 101 may include a display 230 and a processor 210. The processor may be configured to receive an input indicating a selection of a first external electronic device and a second external electronic device among a plurality of external electronic devices viewable through the display. The processor may be configured to, in response to the input, identify a third external electronic device in a category that includes both the first external electronic device and the second external electronic device. The third external electronic device may be configured to include the first external electronic device and the second external electronic device. The processor may be configured to display a first visual object and a second visual object (e.g., visual objects 815 and 821) as being associated with the first external electronic device and the second external electronic device viewable through the display, the first visual object and the second visual object indicating that the first external electronic device and the second external electronic device are selected, respectively. The processor may be configured to display at least one third visual object (e.g., visual objects 822, 823, and 825) as being associated with at least one fourth external electronic device among the third external electronic devices that is different from the first external electronic device and the second external electronic device, the at least one third visual object guiding the selection of at least one fourth external electronic device. The processor may display the first visual object and the second visual object by superimposing on the first external electronic device and the second external electronic device viewable through the display. The wearable device according to an embodiment may provide the user with a UI (e.g., screens 534, 911, 1011, 1023, and 1034) for executing functions commonly applicable to the external electronic devices selected by the user. The wearable device according to an embodiment may provide a user interface for selecting at least one function applicable to multiple external electronic devices.

[0201] For example, the processor may be configured to: in response to an input, identify a first category of the first external electronic device and a second category of the second external electronic device from first metadata about the first external electronic device and second metadata about the second external electronic device. For example, the processor may be configured to: based on the first category and the second category, display executable objects for controlling the first external electronic device and the second external electronic device.

[0202] For example, the processor may be configured to identify one or more functions commonly included in a first function set applicable to an external electronic device in a first category and a second function set applicable to an external object in a second category. The processor may be configured to display an executable object for executing the one or more functions in the display.

[0203] For example, the processor may be configured to identify a category including the third external electronic device based on a higher category of each of the first category of the first external electronic device and the second category of the second external electronic device based on identifying the second category different from the first category.

[0204] For example, the wearable device may further include a communication circuit 250. The processor may be configured to obtain metadata indicating a function applicable to each of the plurality of external electronic devices and including first metadata and second metadata through the communication circuit.

[0205] For example, the input may be a first input. The processor may be configured to, in response to a second input indicating selection of one of the executable objects, request the first external electronic device and the second external electronic device to execute a function mapped to the object selected by the second input.

[0206] For example, the processor may be configured to: in response to an input, receive first data obtained by a first external electronic device and second data obtained by a second external electronic device. The processor may be configured to display a fourth visual object based on a combination of the first data and the second data on the display.

[0207] For example, the input may be a first input. The processor may be configured to identify one or more functions applicable to all third external electronic devices in response to a second input indicating a selection of at least one third visual object. The processor may be configured to display one or more fourth visual objects in the display for performing the one or more functions.

[0208] For example, the wearable device may include a sensor 260 for tracking eye gaze of a user wearing the wearable device. The wearable device may include a microphone 270 for receiving an audio signal. The processor may be configured to identify a preset keyword 716 from the audio signal received by the microphone. The processor may be configured to receive input based on the eye gaze detected by the sensor and the time period 724 during which the preset keyword is identified in response to identifying the preset keyword.

[0209] For example, the processor may be configured to request the first external electronic device and the second external electronic device to execute a function indicated by another keyword based on recognizing another keyword received after a preset keyword from the audio signal.

[0210] As described above, the method of the wearable device 101 may include an operation 1510 of receiving a first input indicating a selection of one or more first external electronic devices among a plurality of external electronic devices viewable through a display. The method may include an operation 1520 of identifying one or more first functions applicable to the one or more first external electronic devices in response to the first input. The method may include an operation 1530 of displaying one or more visual objects for performing the one or more first functions within the display. The method may include an operation 1550 of identifying at least one second function applicable to at least one second external electronic device in response to a second input indicating a selection of at least one second external electronic device in a state where the one or more visual objects are displayed. The method may include an operation 1560 of stopping displaying at least one visual object regarding a function different from the at least one second function in the one or more visual objects. The wearable device according to the embodiment may provide a UI in response to a change in the user's selection of the external electronic device.

[0211] For example, identifying the one or more first functions may include receiving metadata about the one or more first external electronic devices through the communication circuit of the wearable device. Identifying the one or more first functions may include identifying the one or more first functions based on the received metadata.

[0212] For example, identifying the one or more first functions may include: identifying a category including all of the one or more first external electronic devices based on the received metadata. Identifying the one or more first functions may include identifying the one or more first functions based on the identified category.

[0213] For example, displaying the one or more visual objects may include receiving a third input indicating a selection of one of the one or more visual objects. Displaying the one or more visual objects may include: in response to the third input, requesting the one or more first external electronic devices to perform a function of the one or more first functions corresponding to the visual object selected by the third input.

[0214] For example, the method may include: based on the identified at least one second function, in response to the second input, maintaining display of at least one visual object corresponding to the at least one second function.

[0215] For example, displaying one or more visual objects may include displaying a first visual object to adjust the power of one or more first external electronic devices. Displaying one or more visual objects may include: displaying one or more second visual objects, the one or more second visual objects corresponding to one or more first functions applicable to all first external electronic devices and different from the first visual object.

[0216] For example, receiving the first input may include identifying a preset keyword from an audio signal recognized by a microphone of the wearable device. Receiving the first input may include: identifying a movement of a user wearing the wearable device by using a sensor of the wearable device during a time period when the preset keyword is recognized. Receiving the first input may include selecting one or more first external electronic devices from a plurality of external electronic devices based on the recognized movement.

[0217] As described above, the method of the wearable device 101 may include an operation 1410 of receiving an input indicating a selection of a first external electronic device and a second external electronic device among a plurality of external electronic devices viewable through a display of the wearable device. The method may include an operation 1420 of identifying a third external electronic device in a category including both the first external electronic device and the second external electronic device in response to the input. The third external electronic device may include the first external electronic device and the second external electronic device. The method may include an operation 1430 of displaying a first visual object and a second visual object as being associated with the first external electronic device and the second external electronic device viewable through the display, the first visual object and the second visual object indicating that the first external electronic device and the second external electronic device are selected, respectively. The method may include an operation 1440 of displaying at least one third visual object as being associated with at least one fourth external electronic device among the third external electronic devices that is different from the first external electronic device and the second external electronic device, the at least one third visual object guiding the selection of the at least one fourth external electronic device.

[0218] For example, the method may include: in response to an input, identifying a first category of the first external electronic device and a second category of the second external electronic device from first metadata about the first external electronic device and second metadata about the second external electronic device. The method may include displaying executable objects for controlling the first external electronic device and the second external electronic device based on the first category and the second category.

[0219] For example, displaying the executable object may include identifying one or more functions commonly included in a first set of functions applicable to an external electronic device in a first category and a second set of functions applicable to an external object in a second category. Displaying the executable object may include displaying an executable object for executing the one or more functions within a display.

[0220] For example, displaying the executable object may include: based on identifying a second category different from the first category, identifying a category including a third external electronic device based on a higher category of each of the first category of the first external electronic device and the second category of the second external electronic device. Displaying the executable object may include: based on identifying the category including the third external electronic device, displaying the executable object based on the identified category.

[0221] For example, displaying executable objects may include requesting, in response to a second input indicating selection of one of the executable objects, to the first external electronic device and the second external electronic device to execute a function mapped to the object selected by the second input, wherein the second input is different from the input that is the first input.

[0222] For example, the method may include: in response to an input, receiving first data obtained by a first external electronic device and second data obtained by a second external electronic device. The method may include displaying a fourth visual object based on a combination of the first data and the second data on a display.

[0223] For example, displaying at least one third visual object may include identifying one or more functions applicable to all third external electronic devices in response to a second input indicating a selection of the at least one third visual object. Displaying at least one third visual object may include displaying one or more fourth visual objects in the display for performing the one or more functions.

[0224] For example, receiving the input may include identifying a preset keyword from an audio signal received from a microphone. Receiving the input may include, in response to identifying the preset keyword, receiving the input based on the eye gaze detected by the sensor and a time period during which the preset keyword is identified.

[0225] For example, receiving the input may include requesting the first and second external electronic devices to execute a function indicated by another keyword based on recognizing another keyword received after a preset keyword from the audio signal.

[0226] As described above, the wearable device 101 according to the embodiment may include a display 230 and a processor 210. The processor may be configured to receive a first input indicating a selection of one or more first external electronic devices among a plurality of external electronic devices viewable through the display. The processor may be configured to identify one or more first functions applicable to the one or more first external electronic devices in response to the first input. The processor may be configured to display one or more visual objects for performing the one or more first functions within the display. The processor may be configured to identify at least one second function applicable to at least one second external electronic device in response to a second input indicating a selection of at least one second external electronic device while displaying the one or more visual objects. The processor may be configured to stop displaying at least one visual object about a function different from the at least one second function in the one or more visual objects.

[0227] For example, the processor may be configured to receive metadata about one or more first external electronic devices through the communication circuit of the wearable device. The processor may be configured to identify one or more first functions based on the received metadata.

[0228] For example, the processor may be configured to identify a category including all of the one or more first external electronic devices based on the received metadata. The processor may be configured to identify the one or more first functions based on the identified category.

[0229] For example, the processor may be configured to receive a third input indicating a selection of one of the one or more visual objects. The processor may be configured to request the one or more first external electronic devices to perform a function corresponding to the visual object selected by the third input among the one or more first functions in response to the third input.

[0230] For example, the processor may be configured to maintain display of at least one visual object corresponding to the at least one second function based on the at least one second function recognized in response to the second input.

[0231] For example, the processor may be configured to display a first visual object for adjusting power of one or more first external electronic devices. The processor may be configured to display one or more second visual objects that correspond to one or more first functions applicable to all first external electronic devices and are different from the first visual object.

[0232] For example, the wearable device may be configured to further include a sensor 260 and a microphone 270. The processor may be configured to recognize a preset keyword from an audio signal recognized by the microphone. The processor may be configured to recognize a movement of a user wearing the wearable device by using the sensor during a time period in which the preset keyword is recognized. The processor may be configured to select one or more first external electronic devices from a plurality of external electronic devices based on the recognized movement.

[0233] The above-mentioned device can be implemented as a combination of hardware components, software components and / or hardware components and software components. For example, the device and components described in the embodiment can be implemented by using one or more general-purpose computers or special-purpose computers (such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions). The processing device can execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device can access, store, manipulate, process and generate data in response to the execution of software. For ease of understanding, there is a situation where a processing device is described as being used, but people with ordinary knowledge in the relevant technical field can see that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or a processor and a controller. In addition, another processing configuration such as a parallel processor is also possible.

[0234] Software may include computer programs, codes, instructions, or a combination of one or more thereof, and may configure a processing device to operate as desired or may command the processing device independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to interpret commands or data by or to provide commands or data to the processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

[0235] The method according to the embodiment can be implemented in the form of a program command, which can be executed and recorded on a computer-readable medium by various computer devices. In this case, the medium can continuously store a program executable by a computer or can temporarily store a program for execution or download. In addition, the medium can be a variety of recording devices or storage devices in the form of a combination of a single or several hardware, but is not limited to a medium directly connected to a certain computer system, and can be distributed on a network. Examples of media may include magnetic media (such as hard disks, floppy disks, and tapes), optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks), and those media (including ROM, RAM, flash memory, etc.) configured to store program instructions. In addition, examples of other media may include recording media or storage media managed by application stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0236] As described above, although the embodiments have been described using limited examples and drawings, a person with ordinary knowledge in the relevant technical field can make various modifications and variations from the above description. For example, even if the described techniques are performed in a different order than the described method, and / or the components of the described systems, structures, devices, circuits, etc. are coupled or combined in a different form than the described method, or replaced or substituted by other components or equivalents, appropriate results can also be achieved.

[0237] Therefore, other embodiments, other examples, and equivalents to the scope of the claims are within the scope of the following claims.

Claims

1. A wearable device, comprising: monitor; as well as processor, Wherein, the processor is configured to: receiving an input indicating selection of a first external electronic device and a second external electronic device among a plurality of external electronic devices viewable through the display; in response to the input, identifying a third external electronic device in a category that includes both the first external electronic device and the second external electronic device, wherein the third external electronic device includes the first external electronic device and the second external electronic device; displaying a first visual object and a second visual object in association with a first external electronic device and a second external electronic device viewable through the display, the first visual object and the second visual object respectively indicating that the first external electronic device and the second external electronic device are selected; and At least one third visual object is displayed to be associated with at least one fourth external electronic device among the third external electronic devices, which is different from the first external electronic device and the second external electronic device, and the at least one third visual object guides selection of the at least one fourth external electronic device.

2. The wearable device according to claim 1, wherein: The processor is configured to: identifying, in response to the input, a first category of the first external electronic device and a second category of the second external electronic device from first metadata about the first external electronic device and second metadata about the second external electronic device; as well as Based on the first category and the second category, executable objects for controlling the first external electronic device and the second external electronic device are displayed.

3. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: identifying one or more functions commonly included in a first set of functions applicable to external electronic devices in a first category and a second set of functions applicable to external objects in a second category; as well as The executable object for performing the one or more functions is displayed within the display.

4. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: Based on identifying the second category different from the first category, a category including a third external electronic device is identified based on a higher category of each of the first category of the first external electronic device and the second category of the second external electronic device.

5. A wearable device according to any one of the preceding claims, further comprising a communication circuit, in, The processor is configured to: Metadata indicating a function applicable to each of a plurality of external electronic devices and including first metadata and second metadata is obtained through the communication circuit.

6. A wearable device according to any one of the preceding claims, wherein: The input is a first input, Wherein, the processor is configured to: In response to a second input indicating selection of one of the executable objects, execution of a function mapped to the object selected by the second input is requested to the first external electronic device and the second external electronic device.

7. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: In response to the input, receiving first data obtained by a first external electronic device and second data obtained by a second external electronic device; A fourth visual object based on a combination of the first data and the second data is displayed on the display.

8. A wearable device according to any one of the preceding claims, wherein: The input is a first input, Wherein, the processor is configured to: identifying, in response to a second input indicating selection of the at least one third visual object, one or more functions applicable to all third external electronic devices; and One or more fourth visual objects are displayed within the display for performing the one or more functions.

9. A wearable device according to any one of the preceding claims, further comprising, a sensor for tracking eye gaze of a user wearing the wearable device; as well as A microphone for receiving audio signals, in, The processor is configured to: identifying a preset keyword from an audio signal received by the microphone; as well as In response to identifying the preset keyword, the input is received based on the eye gaze detected by the sensor and a time period during which the preset keyword is identified.

10. A wearable device according to any one of the preceding claims, wherein: The processor is configured to: Based on recognizing another keyword received after the preset keyword from the audio signal, executing a function indicated by the another keyword is requested to the first external electronic device and the second external electronic device.

11. A method for a wearable device, comprising: receiving input indicating selection of a first external electronic device and a second external electronic device from among a plurality of external electronic devices viewable through a display of the wearable device; in response to the input, identifying a third external electronic device in a category that includes both the first external electronic device and the second external electronic device, wherein the third external electronic device includes the first external electronic device and the second external electronic device; displaying a first visual object and a second visual object in association with a first external electronic device and a second external electronic device viewable through the display, the first visual object and the second visual object respectively indicating that the first external electronic device and the second external electronic device are selected; and At least one third visual object is displayed to be associated with at least one fourth external electronic device among the third external electronic devices, which is different from the first external electronic device and the second external electronic device, and the at least one third visual object guides selection of the at least one fourth external electronic device.

12. The method according to claim 11, further comprising: identifying, in response to the input, a first category of the first external electronic device and a second category of the second external electronic device from first metadata about the first external electronic device and second metadata about the second external electronic device; Based on the first category and the second category, executable objects for controlling the first external electronic device and the second external electronic device are displayed.

13. A method according to any one of the preceding claims, wherein: Displaying the executable object includes: identifying one or more functions commonly included in a first set of functions applicable to external electronic devices in a first category and a second set of functions applicable to external objects in a second category; The executable object for performing the one or more functions is displayed within the display.

14. A method according to any one of the preceding claims, wherein: Displaying the executable object includes: identifying a category including a third external electronic device based on a higher category of each of the first category of the first external electronic device and the second category of the second external electronic device based on identifying the second category different from the first category; Based on identifying the category including the third external electronic device, the executable object is displayed based on the identified category.

15. A method according to any one of the preceding claims, wherein: Displaying the executable object includes: In response to a second input indicating selection of one of the executable objects, executing a function mapped to the object selected by the second input is requested to the first external electronic device and the second external electronic device, wherein the second input is different from the input as the first input.

Images