Method for controlling external electronic device using remote control device and electronic device supporting the same

Abstract

According to an embodiment, an electronic device may include; a communication module; and at least one processor. The at least one processor may be configured to detect a first remote control device. The at least one processor may be configured to establish a communication connection with the first remote control device via the communication module. The at least one processor may be configured to determine a first external electronic device to be controlled using the first remote control device. The at least one processor may be configured to enable the communication module to establish a communication connection between the electronic device and a first external electronic device. The at least one processor may be configured to perform an operation for mapping at least one key of the first remote control device with at least one function of the first external electronic device. The at least one processor may be configured to control the first external electronic device to perform the at least one function by using the first remote control device and the at least one key of the first remote control device.

Description

Technical Field

[0001] The present disclosure relates to a method of controlling an external electronic device using a remote control device and an electronic device supporting the method. Background Art

[0002] With the development of communication technology, electronic devices are becoming small and light so that they do not cause great inconvenience when worn on the user's body. For example, wearable electronic devices such as head mounted devices (HMDs), smart watches (or bracelets), contact lens type devices, ring type devices, glove type devices, shoe type devices, or clothing type devices are being commercialized. Since wearable electronic devices are worn directly on the body, portability and user accessibility can be improved.

[0003] An HMD device is a device that is worn on the head or face of a user and can provide augmented reality (AR) to the user. For example, an HMD device that provides augmented reality can be implemented in the form of glasses, and can provide information about an object in the form of an image or text to the user in at least a portion of the user's field of view. The HMD device can provide virtual reality (VR) to the user. For example, independent images can be output to both eyes of the user, so that the content provided by the external input is output to the user in the form of video or sound, thereby providing an excellent sense of immersion. In addition, the HMD device can provide mixed reality (MR) and / or extended reality (XR). Summary of the invention

[0004] A user may control an electronic device (e.g., a home appliance) using a remote control device (e.g., a remote controller). A user may control an electronic device using a remote control device dedicated to the electronic device. For example, a user may control functions of the TV using a remote control device of a TV, and control functions of the air conditioner using a remote control device of an air conditioner. In this case, the user may experience inconvenience because the user must use the remote control device dedicated to the electronic device every time the user wants to use the electronic device.

[0005] The present disclosure relates to a method for controlling an external electronic device using a remote control device and an electronic device supporting the method, which enables a user wearing an electronic device (e.g., an HMD device) to control an electronic device (e.g., a home appliance) that the user wants to control using a single remote control device.

[0006] The technical problems that the present disclosure seeks to solve are not limited to the above-mentioned technical problems, and other technical problems not mentioned can be clearly understood by those skilled in the art to which this document relates according to the following description.

[0007] The electronic device according to the embodiment may include a communication module and at least one processor. The at least one processor may be configured to detect a first remote control device. The at least one processor may be configured to establish a communication connection with the first remote control device through the communication module. The at least one processor may be configured to determine a first external electronic device to be controlled using the first remote control device. The at least one processor may be configured to enable the communication module to establish a communication connection between the electronic device and the first external electronic device. The at least one processor may be configured to perform an operation for mapping at least one button of the first remote control device to at least one function of the first external electronic device. The at least one processor may be configured to control the first external electronic device to perform the at least one function using the first remote control device and at least one button of the first remote control device.

[0008] A method for controlling an external electronic device using a remote control device in an electronic device according to an embodiment may include detecting a first remote control device. The method may include establishing a communication connection with the first remote control device through a communication module of the electronic device. The method may include determining a first external electronic device to be controlled using the first remote control device. The method may include establishing a communication connection between the electronic device and the first external electronic device through a communication module. The method may include performing an operation for mapping at least one key of the first remote control device to at least one function of the first external electronic device. The method may include using the first remote control device to control the first external electronic device to perform at least one function.

[0009] The electronic device according to the embodiment may include a communication module and at least one processor. The at least one processor may be configured to detect a first external electronic device. The at least one processor may be configured to enable the communication module to establish a communication connection between the electronic device and the external electronic device. The at least one processor may be configured to detect a first remote control device. The at least one processor may be configured to establish a communication connection with the first remote control device through the communication module. The at least one processor may be configured to perform an operation for mapping at least one key of the first remote control device to at least one function of the first external electronic device. The at least one processor may be configured to use the first remote control device to control the first external electronic device to perform the at least one function.

[0010] In an embodiment, a non-transitory computer-readable medium is provided, on which computer executable instructions are recorded, wherein the computer executable instructions can be configured to cause an electronic device including at least one processor to obtain at least one image when executed. The computer executable instructions can be configured to cause an electronic device including at least one processor to obtain at least one region of interest associated with an object within the at least one image when executed. The computer executable instructions can be configured to cause an electronic device including at least one processor to detect a first remote control device when executed. The computer executable instructions can be configured to cause an electronic device including at least one processor to establish a communication connection with the first remote control device through a communication module when executed. The computer executable instructions can be configured to cause an electronic device including at least one processor to determine a first external electronic device to be controlled using the first remote control device when executed. The computer executable instructions can be configured to cause an electronic device including at least one processor to cause a communication module to establish a communication connection between the electronic device and the first external electronic device when executed. The computer executable instructions can be configured to cause an electronic device including at least one processor to perform an operation for mapping at least one key of the first remote control device to at least one function of the first external electronic device when executed. The computer executable instructions may be configured to, when executed, cause an electronic device including at least one processor to control the first external electronic device to perform the at least one function using the first remote control device and at least one key of the first remote control device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an electronic device in a network environment according to an embodiment.

[0012] Figure 2 is a diagram showing a structure of an electronic device according to an embodiment.

[0013] Figure 3 is a block diagram of an electronic device according to an embodiment.

[0014] Figure 4 is a flowchart illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0015] Figure 5a and Figure 5b is a diagram showing a method for detecting a first remote control device according to an embodiment.

[0016] Figure 6 is a flow chart illustrating a method for establishing a connection with a first remote control device according to an embodiment.

[0017] Figure 7 is a diagram showing a method for establishing a connection with a first remote control device according to an embodiment.

[0018] Figure 8 is a flowchart illustrating a method for determining a first external electronic device according to an embodiment.

[0019] Fig. 9 is a diagram illustrating a method for determining a first external electronic device according to an embodiment.

[0020] Fig.10 is a flowchart illustrating a method for determining a first external electronic device according to an embodiment.

[0021] Fig.11 is a diagram illustrating a method for determining a first external electronic device according to an embodiment.

[0022] Fig.12a is a flow chart illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0023] Figure 12b is a diagram showing a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0024] Fig.13a is a flow chart illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0025] Fig.13b is a diagram showing a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0026] Fig.14a , Fig.14b and Fig.14c is a diagram illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device using a second mapping method according to an embodiment.

[0027] Fig.15 is a diagram illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device using a third mapping method according to an embodiment.

[0028] Fig.16 is a diagram illustrating a method for controlling a first external electronic device to perform at least one function using a first remote control device according to an embodiment.

[0029] Fig.17is a flowchart illustrating a mode related to switching of a first external electronic device according to an embodiment.

[0030] Fig.18 is a diagram showing a mode associated with switching of a first external electronic device according to an embodiment.

[0031] Fig.19 is a system diagram illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0032] Fig. 20 is a flowchart illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0033] Fig.21 is a diagram illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0034] Fig. 22 is a diagram illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0035] Fig.23a is a flowchart illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0036] Figure 23b is a diagram illustrating a method for controlling an external electronic device using a remote control device according to an embodiment. DETAILED DESCRIPTION

[0037] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to an embodiment.

[0038] refer to Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above-described components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) may be implemented as a single component (eg, display module 160 ).

[0039] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resulting data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0040] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning can be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or alternatively, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0041] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). For example, the various data may include input data or output data of software (e.g., the program 140) and commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0042] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0043] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0044] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as playing multimedia or playing records. The receiver may be used to receive incoming calls. Depending on the embodiment, the receiver may be implemented as a separate part from the speaker, or as part of the speaker.

[0045] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display module 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0046] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0047] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0048] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0049] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0050] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0051] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0052] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0053] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0054] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth™, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0055] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low latency communication (URLLC). The wireless communication module 192 can support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high frequency bands, such as, for example, beamforming, massive multiple input multiple output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0056] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0057] According to an embodiment, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a specified high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the specified high frequency band.

[0058] At least some of the above components may be coupled to each other and transmit signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

[0059] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0060] The electronic device according to the embodiment may be one of various types of electronic devices. For example, the electronic device may include a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the above-mentioned devices.

[0061] It should be understood that the embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or alternatives of the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each phrase such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together in a corresponding phrase. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding components from other components, and do not limit the components in other aspects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “coupled”, “coupled to”, “connected to” or “connected to” another element (e.g., a second element) by using or not using the terms “operably” or “communicatively”, it means that the element can be coupled to the other element directly (e.g., wired), wirelessly or via a third element.

[0062] As used in conjunction with the embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0063] The embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform the at least one function according to the at least one instruction called. One or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between the case where data is semi-permanently stored in the storage medium and the case where data is temporarily stored in the storage medium.

[0064] According to an embodiment, the method according to an embodiment of the present disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be published in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or the computer program product may be published online (e.g., downloaded or uploaded) via an application store (e.g., PlayStoreTM), or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., a manufacturer's server, an application store's server, or a memory of a relay server).

[0065] According to an embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be detachably arranged in different components. According to an embodiment, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to an embodiment, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

[0066] Figure 2 is a diagram showing a structure of an electronic device 201 according to an embodiment.

[0067] refer to Figure 2 In an embodiment, the electronic device 201 (e.g., the electronic device 101) may include one or more first cameras 211-1 and 211-2, one or more second cameras 212-1 and 212-2, and one or more third cameras 213. In an embodiment, images obtained by the one or more first cameras 211-1 and 211-2 may be used to detect a user's gesture, track the user's head, and / or recognize a space. In an embodiment, the one or more first cameras 211-1 and 211-2 may be global shutter (GS) cameras.

[0068] In an embodiment, the one or more first cameras 211-1 and 211-2 may perform a simultaneous localization and mapping (SLAM) operation through depth capture. In an embodiment, the one or more first cameras 211-1 and 211-2 may perform 6DoF (degrees of freedom) spatial recognition.

[0069] In an embodiment, the image obtained by the one or more second cameras 212-1 and 212-2 may be used to detect and track the pupil of the user. In an embodiment, the one or more second cameras 212-1 and 212-2 may be GS cameras. In an embodiment, the one or more second cameras 212-1 and 212-2 may correspond to the left eye and the right eye, respectively, and the one or more second cameras 212-1 and 212-2 may have the same performance.

[0070] In an embodiment, the one or more third cameras 213 may be a high-resolution camera. In an embodiment, the one or more third cameras 213 may perform an auto focus (AF) function and an image stabilization function. In an embodiment, the one or more third cameras 213 may be a GS camera or a rolling shutter (RS) camera.

[0071] In an embodiment, the electronic device 201 may include one or more light emitting elements 214-1 and 214-2. In an embodiment, the light emitting elements 214-1 and 214-2 may be different from the light source described below (which emits light to the screen output area of ​​the display). In an embodiment, when the pupil of the user is detected and tracked by one or more second cameras 212-1 and 212-2, the light emitting elements 214-1 and 214-2 may emit light to facilitate the detection of the pupil.

[0072] In an embodiment, each of the light emitting elements 214-1 and 214-2 may include a light emitting diode (LED). In an embodiment, the light emitting elements 214-1 and 214-2 may emit light in an infrared range. In an embodiment, the light emitting elements 214-1 and 214-2 may be attached to the periphery of the frame of the electronic device 201. In an embodiment, the light emitting elements 214-1 and 214-2 may be located around one or more first cameras 211-1 and 211-2, and when the electronic device 201 is used in a dark environment, the light emitting elements 214-1 and 214-2 may assist in gesture detection, head tracking, and spatial recognition performed by the one or more first cameras 211-1 and 211-2. In an embodiment, the light emitting elements 214-1 and 214-2 may be located around one or more third cameras 213, and when the electronic device 201 is used in a dark environment, the light emitting elements 214-1 and 214-2 may assist in obtaining images by the one or more third cameras 213.

[0073] In an embodiment, the electronic device 201 may include batteries 235 - 1 and 235 - 2 . The batteries 235 - 1 and 235 - 2 may store power to operate the remaining elements of the electronic device 201 .

[0074] In an embodiment, the electronic device 201 may include a first display 251, a second display 252, one or more optical input members 253-1 and 253-2, one or more transparent elements 290-1 and 290-2, and one or more screen display areas 254-1 and 254-2.

[0075] In an embodiment, the first display 251 and the second display 252 may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0076] In an embodiment, when the first display 251 and the second display 252 are configured as one of a liquid crystal display, a digital mirror display, or a silicon-based liquid crystal, the electronic device 201 may include a light source that emits light to a screen output area of ​​the display. In an embodiment, when the first display 251 and the second display 252 are configured to emit light by themselves (for example, configured as one of an organic light emitting diode or a micro LED), the electronic device 201 may provide a better quality virtual image to the user even without a separate light source.

[0077] In an embodiment, when the user wears the electronic device 201, one or more transparent members 290-1 and 290-2 may be arranged to face the user's eyes. In an embodiment, one or more transparent members 290-1 and 290-2 may include at least one of a glass plate, a plastic plate, or a polymer. In an embodiment, when the electronic device 201 is worn, the user can view the outside through the one or more transparent members 290-1 and 290-2. In an embodiment, one or more optical input members 253-1 and 253-2 may guide the light generated from the first display 251 and the second display 252 to the user's eyes. In an embodiment, an image based on the light generated from the first display 251 and the second display 252 may be formed on one or more screen display areas 254-1 and 254-2 on the one or more transparent members 290-1 and 290-2, and the user may view the image formed on the one or more screen display areas 254-1 and 254-2.

[0078] In an embodiment, the electronic device 201 may include one or more optical waveguides (not shown). The optical waveguide may transmit light generated from the first display 251 and the second display 252 to the eyes of the user. The electronic device 201 may include one optical waveguide corresponding to the left eye and one optical waveguide corresponding to the right eye, respectively. In an embodiment, the optical waveguide may include at least one of glass, plastic, or polymer. In an embodiment, the optical waveguide may include a nano pattern formed on one of the inner surface or the outer surface, such as a grating structure having a polygonal or curved shape. In an embodiment, the optical waveguide may include a free-form prism, and in this case, the optical waveguide may provide incident light to the user through a reflector. In an embodiment, the optical waveguide may include at least one of a diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a reflector), and the display light emitted from the light source may be guided to the eyes of the user using at least one diffractive element or reflective element included in the optical waveguide. In an embodiment, the diffractive element may include an optical input / output member. In an embodiment, the reflective element may include an element that causes total reflection.

[0079] In an embodiment, the electronic device 201 may include one or more voice input devices 262-1, 262-2, and 262-3 and one or more voice output devices 263-1 and 263-2.

[0080] In an embodiment, the electronic device 201 may include a first PCB 270-1 and a second PCB 270-2. The first PCB 270-1 and the second PCB 270-2 may be configured to send electrical signals to elements (e.g., one or more first cameras 211-1 and 211-2, one or more second cameras 212-1 and 212-2, one or more third cameras 213, a display, an audio module, and a sensor) included in the electronic device 201. In an embodiment, the first PCB 270-1 and the second PCB 270-2 may include a flexible printed circuit board (FPCB). In an embodiment, each of the first PCB 270-1 and the second PCB 270-2 may include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate.

[0081] Figure 3 is a block diagram of an electronic device 301 according to an embodiment.

[0082] refer to Figure 3 In an embodiment, the electronic device 301 may be included in Figure 1 The electronic device 101 or Figure 2 For example, the electronic device 301 may be an HMD device capable of providing augmented reality, virtual reality, mixed reality and / or extended reality.

[0083] In an embodiment, the electronic device 301 may include a communication module 310 , a camera module 320 , a display module 330 , a sensor 340 , an audio module 341 , a microphone 342 , a speaker 343 , a memory 350 , and / or a processor 360 .

[0084] In an embodiment, the communication module 310 may be included in Figure 1 In the communication module 190.

[0085] In an embodiment, the communication module 310 can establish a communication connection between the electronic device 301 and a remote control device (e.g., a remote controller), a server, and / or at least one external electronic device. For example, the communication module 310 can establish a communication connection between the electronic device 301 and the remote control device. For example, the communication module 310 can establish a communication connection between the electronic device 301 and a server (e.g., an IoT server) that can use Internet of Things (IoT) technology to control an external electronic device. For example, the communication module 310 can establish a communication connection between the electronic device 301 and at least one external electronic device (e.g., an IoT device registered to an IoT server). However, the device that can be connected to the electronic device 301 through the communication module 310 is not limited to the aforementioned remote control device, server, and / or at least one external electronic device.

[0086] In an embodiment, the communication module 310 may receive a command from a remote control device. For example, a short-range communication module (e.g., Bluetooth or Wi-Fi) may receive a command corresponding to a user input to the remote control device (e.g., a command for controlling a function of an external electronic device) from a remote control device in communication connection with the electronic device 301. For example, an infrared (IR) communication (infrared communication) module (also referred to as an "IR sensor") may receive a command from a remote control device in communication connection with the electronic device 301. However, the communication module 310 for receiving a command from a remote control device is not limited to the above-mentioned short-range communication module and IR communication module.

[0087] In an embodiment, the camera module 320 may be included in Figure 1 In the camera module 180.

[0088] In an embodiment, the camera module 320 may include a plurality of cameras. For example, the camera module 320 may include a camera for detecting a user's gesture, tracking a user's head, and / or recognizing a space (e.g., Figure 2 For example, the camera module 320 may include a camera for detecting and tracking the user's eyes (eg, Figure 2 For example, the camera module 320 may include a camera for performing an auto focus function and an image stabilization function (eg, Figure 2 However, the cameras included in the camera module 320 are not limited to the above examples, and the camera module 320 may exclude at least some of the above cameras.

[0089] In an embodiment, the display module 330 may be included in Figure 1 In the display module 160.

[0090] In an embodiment, the sensor 340 may obtain information about the motion (and direction) of the electronic device. In an embodiment, the sensor 340 may include a six-axis sensor, a magnetic sensor, a proximity sensor, and / or an optical sensor.

[0091] In an embodiment, the audio module 341 may be included in Figure 1 In the audio module 170 in the embodiment. In an embodiment, the microphone 342 may be included in Figure 1 In an embodiment, the speaker 343 may be included in the input module 150. Figure 1 In the audio output module 155.

[0092] In an embodiment, the memory 350 may be included in Figure 1 In the memory 130 in.

[0093] In an embodiment, the memory 350 may store information for performing an operation of controlling an external electronic device using a remote control device. The information stored in the memory 350 will be described later.

[0094] In an embodiment, processor 360 may be included in Figure 1 In the processor 120 in.

[0095] In an embodiment, the processor 360 may control the overall operation of controlling the external electronic device using the remote control device. In an embodiment, the processor 360 may include one or more processors for performing the operation of controlling the external electronic device using the remote control device. Figure 4 The operation of the processor 360 controlling the external electronic device using the remote control device is described with reference to the following figures.

[0096] although Figure 3 The electronic device 301 is shown to include a communication module 310, a camera module 320, a display module 330, a sensor 340, a memory 350, an audio module 341, a microphone 342, a speaker 343, and a processor 360, but the present disclosure is not limited thereto. For example, the electronic device 301 may also include Figure 1 or Figure 2 For example, the electronic device 301 may exclude Figure 3 At least one of the configurations shown.

[0097] The electronic device 301 according to the embodiment may include a communication module 310 and at least one processor 360. The at least one processor 360 may be configured to detect a first remote control device. The at least one processor 360 may be configured to establish a communication connection with the first remote control device through the communication module 310. The at least one processor 360 may be configured to determine a first external electronic device to be controlled using the first remote control device. The at least one processor 360 may be configured to control the communication module 310 to establish a communication connection between the electronic device 301 and the first external electronic device. The at least one processor 360 may be configured to perform an operation for mapping at least one key of the first remote control device to at least one function of the first external electronic device. The at least one processor 360 may be configured to control the first external electronic device to perform the at least one function using the first remote control device and at least one key of the first remote control device.

[0098] In an embodiment, the electronic device 301 may further include a camera module 320, and the at least one processor 360 may be configured to obtain an image through the camera module 320 and, based on the image, detect a first remote control device located within a range corresponding to the line of sight of a user wearing the electronic device 301.

[0099] In an embodiment, at least one processor 360 may be configured to: detect movement of a first remote control device; based on detecting movement of the first remote control device, request a second external electronic device corresponding to the first remote control device to forward a command of the first remote control device to the electronic device 301; and establish a communication connection with the first remote control device.

[0100] In an embodiment, at least one processor 360 may be configured to: receive user input input to the first remote control device from the first remote control device via the second external electronic device and the server through the communication module 310, the user input being input for a communication connection with the first remote control device; and establish a communication connection with the first remote control device based on the user input.

[0101] In an embodiment, the at least one processor 360 may be configured to detect the first external electronic device and control the communication module 310 such that the communication module 310 establishes a communication connection between the first external electronic device and the electronic device 301 .

[0102] In an embodiment, at least one processor 360 may be configured to: output a list of external electronic devices that can be controlled by using the first remote control device based on a communication connection being established with the first remote control device; determine the first external electronic device from among the external electronic devices based on user input to the list; and control the communication module 310 so that the communication module 310 establishes a communication connection between the first external electronic device and the electronic device 301.

[0103] In an embodiment, at least one processor 360 can be configured to: display at least one object corresponding to at least one function of a first external electronic device through a display module 330 of the electronic device 301, so that the at least one object is seen by a user wearing the electronic device (301) and is set in a three-dimensional space around the electronic device (301); select an object corresponding to the user's line of sight from among the at least one object; receive information about a key of the first remote control device, a user input input to the key, and a function corresponding to the object to be mapped to the key from the first remote control device through a communication module 310; and map the function corresponding to the selected object to the key of the first remote control device.

[0104] In an embodiment, at least one processor 360 may be configured to: classify at least one function of the first external electronic device into groups with priorities based on a user's usage history of each of the at least one function, and display at least one object through the display module 330 based on the priority of the group so that the user sees the at least one object arranged in a three-dimensional space around the electronic device 301.

[0105] In an embodiment, at least one processor 360 may be configured to: if at least one function of the first external electronic device is mapped to at least one key of the first remote control device, display an object to be overlapped with at least one key of the first remote control device through the display module 330 of the electronic device 301, the object indicating the at least one mapped function.

[0106] The electronic device 301 according to the embodiment may include a communication module 310 and at least one processor 360. The at least one processor 360 may be configured to detect a first external electronic device. The at least one processor 360 may be configured to: control the communication module 310 so as to establish a communication connection between the electronic device 301 and the first external electronic device. The at least one processor 360 may be configured to detect a first remote control device. The at least one processor 360 may be configured to establish a communication connection with the first remote control device through the communication module 310. The at least one processor 360 may be configured to perform an operation for mapping at least one key of the first remote control device to at least one function of the first external electronic device. The at least one processor 360 may be configured to use the first remote control device to control the first external electronic device to perform the at least one function.

[0107] In an embodiment, the at least one processor 360 may be configured to display information about at least one remote control device capable of controlling the first external electronic device through the display module 330 of the electronic device 301 when the first external electronic device and the electronic device 301 are in communication connection.

[0108] Figure 4 is a flow chart 400 illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0109] refer to Figure 4 In operation 401, in an embodiment, the processor 360 may detect a first remote control device. For example, the processor 360 may detect (or identify) a remote control device for controlling an external electronic device (hereinafter, referred to as a "first remote control device").

[0110] In an embodiment, the processor 360 may detect the first remote control device based on an image obtained through the camera module 320 .

[0111] In an embodiment, the processor 360 may use the camera module 320 (e.g., Figure 2 The processor 360 can detect the sight of the user wearing the electronic device 301 through the camera module 320 (for example, Figure 2 The processor 360 may detect the first remote control device by comparing the obtained image with a plurality of images stored in the memory 350 and including remote control devices. The processor 360 may detect the first remote control device to obtain information about the first remote control device (e.g., the type, name, and model name of the first remote control device, functions performed by the first remote control device, external electronic devices dedicated to the remote control device, and operation status) from the memory 350.

[0112] In an embodiment, the processor 360 may use the camera module 320 (e.g., Figure 2 The processor 360 may detect the first remote control device by comparing the obtained image with a plurality of images including remote control devices stored in the memory 350. The processor 360 may obtain information about the first remote control device from the memory 350 by detecting the first remote control device.

[0113] Although the above example shows that the processor 360 detects the first remote control device based on the image obtained by the camera module 320, the present disclosure is not limited thereto. For example, the processor 360 can detect the first remote control device located in the direction the electronic device faces through the communication module 310 (e.g., ultra-wideband (UWB) communication module 310).

[0114] The following will refer to Figure 5a and Figure 5b A method for detecting a first remote control device is described.

[0115] Figure 5a and Figure 5b is a diagram showing a method for detecting a first remote control device according to an embodiment.

[0116] refer to Figure 5a and Figure 5b In an embodiment, the processor 360 may detect that the electronic device 301 is worn on the user's body (e.g., the user's face) through a sensor 340 (e.g., a proximity sensor). In an embodiment, in a state where the electronic device 301 is worn on the user's body, the processor 360 may cause the electronic device 301 to enter a mode for controlling an external electronic device (hereinafter, also referred to as a "home control mode") based on a user input. In an embodiment, the electronic device 301 and the external electronic device may be registered in an IoT network. For example, the electronic device 301 may register an external electronic device that can be controlled by the electronic device 301 in association with a location (e.g., a place) and / or a place (e.g., a room) specified in the server through an IoT server (hereinafter, referred to as a "server") providing IoT services by using an account of a user of the electronic device 301. In an embodiment, if the external electronic device is registered in the IoT network, the processor 360 may cause the electronic device 301 to enter a mode for controlling the functions of the registered external electronic device based on a user input.

[0117] In an embodiment, if the electronic device 301 enters a mode for controlling a function of an external electronic device, the processor 360 may obtain information about an external electronic device registered in the IoT server in association with a designated location and / or a designated place, and information about a remote control device for controlling the external electronic device. For example, if the electronic device 301 enters a mode for controlling a function of an external electronic device, the processor 360 may receive information about the external electronic device and information about a remote control device for controlling the external electronic device from the server through the communication module 310. The processor 360 may store the information about the external electronic device and the information about the remote control device received from the server in the memory 350.

[0118] In an embodiment, the information of the external electronic device may include the type of the external electronic device, the name, the model name, the function performed by the external electronic device (e.g., the function of the external electronic device that can be performed by the remote controller), the operation state, the information about the remote control device corresponding to the external electronic device, and / or the image of the external electronic device. In an embodiment, the image of the external electronic device may be used to detect the external electronic device.

[0119] In an embodiment, the information of the remote control device may include the type, name and model name of the remote control device, functions that can be performed by the remote control device, information about an external electronic device dedicated to the remote control device, an operation state, and / or an image of the remote control device. In an embodiment, the image of the remote control device may be used to detect the remote control device. For example, the image of the remote control device may be at least one image used for comparison with the image of the first remote control device in operation 401.

[0120] In an embodiment, the processor 360 may cause the electronic device to enter a mode (home control mode) for controlling functions of an external electronic device, as indicated by reference numeral 501. In an embodiment, reference numeral 501 may indicate that a user controls the functions of an external electronic device through the electronic device 301 (e.g., Figure 2 The view 511 is seen by one or more transparent members 290 - 1 and 290 - 2 in the figure. The view may include remote control device 1 ( 513 ) and remote control device 515 placed on table 517 .

[0121] In an embodiment, reference numeral 502 may represent a view 521 that the user sees through the electronic device 301 when the line of sight of the user wearing the electronic device 301 moves (eg, when the user wearing the electronic device moves his head to the left).

[0122] In an embodiment, in reference numeral 503, the processor 360 may detect the remote control device 1 (513) located within a range corresponding to the user's line of sight. For example, the processor 360 may recognize that the remote control device 1 (513) exists within the range corresponding to the user's line of sight for a specified period of time (for example, when the user's line of sight is fixed on the remote control device 513 for the specified period of time). The processor 360 may perform an operation for detecting the remote control device 1 (513) based on recognizing that the remote control device 1 (513) exists within the range corresponding to the user's line of sight for the specified period of time. The processor 360 may display an indication 533 (for example, a virtual line indicating an outer boundary of the remote control device 1 (513)) through the display module 330, the indication 533 indicating that the remote control device 1 (513) is detected (or indicating that an operation of detecting the remote control device 1 (513) is being performed). However, the present disclosure is not limited thereto, and as a supplement or alternative to the indication 533, the processor 360 may output audio indicating that the remote control device 1 (513) is detected (or indicating that an operation of detecting the remote control device 1 (513) is being performed) through a speaker.

[0123] In an embodiment, in reference numeral 504, if the remote control device 1 (513) located within the range corresponding to the user's line of sight is detected, the processor 360 may display information 543 about the remote control device 1 (513) together with an indication 533 on the display module 330. However, the present disclosure is not limited thereto, and in addition to or alternatively to the information 543, the processor 360 may output audio indicating the information 543 about the remote control device 1 (513) through a speaker.

[0124] In an embodiment, reference numeral 505 may indicate a view 551 that the user sees through the electronic device 301 when the line of sight of the user wearing the electronic device 301 moves (eg, when the user wearing the electronic device 301 moves his head to the right).

[0125] In an embodiment, in reference numeral 506, the processor 360 may detect the remote control device 2 (515) located within a range corresponding to the user's line of sight. For example, the processor 360 may recognize that the remote control device 2 (515) exists within the range corresponding to the user's line of sight within a specified time period. The processor 360 may perform an operation for detecting the remote control device 2 (515) based on recognizing that the remote control device 2 (515) exists within the range corresponding to the user's line of sight within a specified time period. The processor 360 may display an indication 563 (e.g., a virtual line indicating an outer boundary of the remote control device 2 (515)) through the display module 330, the indication 563 indicating that the remote control device 2 (515) is detected (or indicating that an operation of detecting the remote control device 2 (515) is being performed). However, the present disclosure is not limited thereto, and as a supplement or alternative to the indication 533, the processor 360 may output audio indicating that the remote control device 2 (515) is detected (or indicating that an operation of detecting the remote control device 2 (515) is being performed) through a speaker. In an embodiment, if the remote control device 2 (515) located within the range corresponding to the user's line of sight is detected, the processor 360 may display information 565 about the remote control device 2 (515) together with the indication 563 on the display module 330. However, the present disclosure is not limited thereto, and in addition to or alternatively to the information 565, the processor 360 may output audio indicating the information 565 about the remote control device 2 (515) through the speaker.

[0126] Return to reference Figure 4 In operation 403, in an embodiment, the processor 360 may establish a connection with the first remote control device through the communication module 310. For example, the processor 360 may establish a connection to receive a command inputted to the first remote control device by a user from the first remote control device. Figure 6 and Figure 7 A method for establishing a connection with a first remote control device is described.

[0127] Figure 6 is a flow chart 600 illustrating a method for establishing a connection with a first remote control device according to an embodiment.

[0128] Figure 7 FIG700 is a diagram illustrating a method for establishing a connection with a first remote control device according to an embodiment.

[0129] refer to Figure 6 and Figure 7 , in operation 601, in an embodiment, the processor 360 may detect movement of a first remote control device.

[0130] In an embodiment, the processor 360 may detect movement of the first remote control device based on a plurality of images sequentially obtained through the camera module 320 .

[0131] In an embodiment, if the electronic device 301 and the first remote control device are in communication connection, the processor 360 may detect the movement of the first remote control device based on the information received from the first remote control device. For example, the first remote control device may detect the movement of the first remote control device through a motion sensor (e.g., an acceleration sensor) included in the first remote control device. The processor 360 may obtain information about the movement of the first remote control device detected in the first remote control device from the first remote control device through the communication module 310, and detect the movement of the first remote control device.

[0132] In an embodiment, the processor 360 may detect a gesture of the user holding the first remote control device. For example, the processor 360 may detect a gesture of the user holding the first remote control device based on an image obtained through the camera module 320.

[0133] In an embodiment, the processor 360 may detect a gesture of the user holding the first remote control device and a movement of the first remote control device.

[0134] In an embodiment, if the movement of the first remote control device is detected (and / or a gesture of the user holding the first remote control device is detected), the processor 360 may output information indicating the first remote control device. For example, as shown in reference numeral 701, if the first remote control device 711 is moved, the processor 360 may display an indication 722 indicating the first remote control device through the display module 330. For example, as shown in reference numeral 701, if the first remote control device 711 is moved, the processor 360 may output audio indicating that the first remote control device 711 is moved through a speaker.

[0135] In operation 603, in an embodiment, the processor 360 may request the second external electronic device corresponding to the first remote control device to forward the command of the first remote control device to the electronic device 301. For example, if the movement of the first remote control device is detected, the processor 360 may request the second external electronic device corresponding to the first remote control device to forward the command of the first remote control device to the electronic device 301.

[0136] In an embodiment, the second external electronic device corresponding to the first remote control device may be an external electronic device to which the first remote control device is configured to be dedicated (hereinafter, referred to as the "second external electronic device"). For example, if the first remote control device is configured to be dedicated to TV control, the second external electronic device may be a TV. However, the present disclosure is not limited thereto. For example, the second external electronic device may include an external electronic device currently controlled by a user input to the first remote control device.

[0137] In an embodiment, if a user input for selecting a button of the first remote control device is input to the first remote control device, the processor 360 may send a request (hereinafter, also referred to as a "command transmission request") to the second external electronic device so that the second external electronic device receiving the command corresponding to the user input transmits the command to the electronic device 301. For example, the processor 360 may send the command transmission request to a server (e.g., an IoT server to which the electronic device 301 and the second external electronic device are registered) through the communication module 310. The server may send the received command transmission request to the second external electronic device.

[0138] In an embodiment, if the second external electronic device receives a command corresponding to the user input from the first remote control device when the command transmission request is sent to the second external electronic device, the processor 360 may cause the second external electronic device not to perform a function corresponding to the received command. In an embodiment, in the case of receiving a command transmission request from the server, the second external electronic device may be in a standby state, in which the second external electronic device does not perform a function corresponding to the received command even if the second external electronic device receives a command corresponding to the user input from the first remote control device.

[0139] In operation 605, in an embodiment, the processor 360 may establish a communication connection with a first remote control device.

[0140] In an embodiment, the processor 360 may establish a communication connection with the first remote control device based on a user input to the first remote control device. In an embodiment, the user input to the first remote control device may include a specified user input for establishing a communication connection between the electronic device and the first remote control device.

[0141] In an embodiment, the first remote control device may obtain a specified user input, which includes a user input for selecting a specified button of the first remote control device (e.g., a button associated with an identifier of a function), a specified movement of the first remote control device, and / or a specified voice input of a user. The first remote control device may send a command corresponding to the specified user input to the second external electronic device. The second external electronic device may send a command corresponding to the specified user input received from the first remote control device to the server. The server may send a command corresponding to the specified user input received from the second external electronic device to the electronic device 301. The processor 360 may receive a command corresponding to the specified user input from the server through the communication module 310. The processor 360 may establish a communication connection between the electronic device and the first remote control device based on receiving the command corresponding to the specified user input.

[0142] In an embodiment, the processor 360 may control the communication module 310 so that the communication module 310 establishes a communication connection between the electronic device 301 and the first remote control device. For example, in a case where the first remote control device is a device capable of connecting to the electronic device 301 using short-range wireless communication (e.g., Bluetooth or Wi-Fi) (e.g., in a case where the first remote control device includes a short-range wireless communication module 310), the processor 360 may control the short-range wireless communication module based on receiving a command corresponding to a specified user input so that the short-range wireless communication module establishes a communication connection between the electronic device 301 and the first remote control device. In an embodiment, when a communication connection is established between the electronic device and the first remote control device, the processor 360 may control the first remote control device and / or the second external electronic device so that the communication connection established between the first remote control device and the second external electronic device is released.

[0143] In an embodiment, in the case where the first remote control device is a device (e.g., a remote control device) configured to control the second external electronic device using infrared (IR) communication or radio frequency (RF) communication, the processor 360 may perform configuration so that the electronic device 301 receives a command corresponding to a user input to the first remote control device via the second external electronic device and the server. For example, if a user input of a command is input to the first remote control device, the processor 360 may perform configuration so that the first remote control device sends the command to the second external electronic device, so that the second external electronic device sends the command to the server, and so that the server sends the command to the electronic device 301.

[0144] In an embodiment, if a communication connection with the first remote control device is established, the processor 360 may output information indicating that a communication connection with the first remote control device is established. For example, as shown in reference numeral 702, the processor 360 may display a virtual image 721 indicating the first remote control device in a designated area 723 of the transparent member (e.g., the upper right area of ​​the transparent member) through the display module 330 as information indicating that a communication connection with the first remote control device is established. For example, in addition to the virtual image 721, the processor 360 may also output audio through a speaker, the audio indicating that a communication connection with the first remote control device is established.

[0145] In an embodiment, if a communication connection with the first remote control device is established, the processor 360 may output information indicating a second external electronic device corresponding to the first remote control device. For example, as shown in reference numeral 703, the processor 360 may display a virtual image 723 in a designated area 723 of the transparent member through the display module 330, and display an indication 732, the virtual image 723 indicating a second external electronic device 712 corresponding to the first remote control device 711, the indication 732 indicating that the second external electronic device 712 is within the field of view within a designated period of time (or temporarily). For example, as shown in reference numeral 703, if the second external electronic device 712 exists within the range corresponding to the user's line of sight, the processor 360 may display a virtual image 731 and an indication 732 within the designated area 723 of the transparent member through the display module 330 to indicate that the second external electronic device 712 can be controlled by the first remote control device 711 (for example, indicating that the second external electronic device 712 can be determined as the first external electronic device to be described later), the virtual image 731 indicating the second external electronic device 712 corresponding to the first remote control device 711, and the indication 732 indicating that the second external electronic device 712 is within the field of view.

[0146] Return to reference Figure 4 In operation 405, in an embodiment, the processor 360 may determine a first external electronic device to be controlled using a first remote control device. For example, the processor 360 may determine that the external electronic device (hereinafter, referred to as "the first external electronic device") performs a function based on a command corresponding to a user input to the first remote control device.

[0147] In operation 407, in an embodiment, the processor 360 may control the communication module 310 so that the communication module 310 establishes a communication connection between the electronic device 301 and the first external electronic device. However, if the communication connection between the electronic device 301 and the first external electronic device has already been established, operation 407 may not be performed.

[0148] In the following, reference will be made to Figures 8 to 11A method of determining an operation of a first external electronic device in operation 405 and establishing a communication connection with the first external electronic device in operation 407 is described.

[0149] Figure 8 is a flowchart 800 illustrating a method for determining a first external electronic device according to an embodiment.

[0150] Fig. 9 is a diagram 900 illustrating a method for determining a first external electronic device according to an embodiment.

[0151] refer to Figure 8 and Fig. 9 , in operation 801, in an embodiment, the processor 360 may detect a first external electronic device.

[0152] In an embodiment, the processor 360 may use the camera module 320 (e.g., Figure 2 The processor 360 can detect the sight of the user wearing the electronic device through the camera module 320 (for example, Figure 2 The processor 360 may detect the first external electronic device by comparing the obtained image with a plurality of images stored in the memory 350 and including external electronic devices. The processor 360 may detect the first external electronic device to obtain information about the first external electronic device (e.g., the type, name, model name, function performed by the first external electronic device, and operation status) from the memory 350.

[0153] In an embodiment, the processor 360 may use the camera module 320 (e.g., Figure 2 The processor 360 may detect the first external electronic device by comparing the obtained image with a plurality of images including external electronic devices stored in the memory 350. The processor 360 may obtain information about the first external electronic device from the memory 350 by detecting the first external electronic device.

[0154] Although the above example shows that the processor 360 detects the first external electronic device based on the image obtained by the camera module 320, the present disclosure is not limited thereto. For example, the processor 360 may detect the first external electronic device located in the direction facing the electronic device through the communication module 310 (e.g., an ultra-wideband (UWB) communication module 310).

[0155] In an embodiment, if the first external electronic device is detected, the processor 360 may output information indicating that the first external electronic device is detected. Fig. 9 In the embodiment, if the air conditioner 911 as the first external electronic device is detected, the processor 360 may display an indication 912 through the display module 330, the indication 912 indicating that the air conditioner 911 is detected. Fig. 9 In the embodiment, if the air conditioner 911 as the first external electronic device is detected, the processor 360 may display information 915 about the air conditioner 911 (e.g., the name and model name of the air conditioner 911) and an indication 912 indicating that the air conditioner 911 is detected through the display module 330. For example, if the air conditioner 911 as the first external electronic device is detected, the processor 360 may output audio indicating that the air conditioner 911 is detected through a speaker.

[0156] In an embodiment, if the first external electronic device is detected, the processor 360 may output information indicating that the first external electronic device can be controlled using the first remote control device. Fig. 9 , if the air conditioner 911 is detected as the first external electronic device, the processor 360 can display a virtual image 913 indicating the air conditioner 911 in a designated area 723 of the transparent member (for example, the upper right area of ​​the transparent member) to indicate that the air conditioner 911 can be controlled using the first external electronic device 711.

[0157] In an embodiment, the processor 360 may output audio through a speaker in different time periods based on whether the first external electronic device is currently a control target using the electronic device 301 and the first remote control device. For example, if the first external electronic device is currently a control target using the first remote control device, the processor 360 may output audio through the speaker in a first time (e.g., a connection sound to be output in the first time). If the first external electronic device detected in operation 801 is the air conditioner 911 (e.g., if the first external electronic device is not currently a control target using the first remote control device), the processor 360 may output audio through the speaker in a second time longer than the first time (e.g., a connection sound to be output in the second time).

[0158] In operation 803, in an embodiment, the processor 360 may control the communication module 310 so that the communication module 310 establishes a communication connection between the electronic device and the first external electronic device. However, if the communication connection between the electronic device and the first external electronic device has already been established, the operation of establishing a communication connection between the electronic device and the first external electronic device in operation 803 may not be performed. In this case, the processor 360 may determine the first external electronic device detected in operation 801 as a control target using the first remote control device.

[0159] In an embodiment, in a state where the first external electronic device is detected, the processor 360 may control a short-range communication module (e.g., a Bluetooth module or a Wi-Fi module) based on a designated user input to the electronic device 301 (or the first remote control device), so that the short-range communication module establishes a communication connection between the electronic device 301 and the first external electronic device. For example, in a state where the first external electronic device is detected, the processor 360 may perform configuration based on a designated user input to the electronic device 301 (or the first remote control device), so that a command sent from the electronic device 301 is sent to the first external electronic device through the server.

[0160] In an embodiment, although the above example shows that the communication connection between the electronic device 301 and the first external electronic device is established based on a specified user input to the electronic device 301 (or the first remote control device), the present disclosure is not limited thereto. For example, based on detecting that the first external electronic device exists within a range corresponding to the user's line of sight within a specified time period after the first external electronic device is detected in operation 801, the processor 360 may control the communication module 310 so that the communication module 310 establishes a communication connection between the electronic device 301 and the first external electronic device.

[0161] Fig.10 is a flowchart 1000 illustrating a method for determining a first external electronic device according to an embodiment.

[0162] Fig.11 is a diagram 1100 illustrating a method for determining a first external electronic device according to an embodiment.

[0163] refer to Fig.10 and Fig.11 In operation 1001, in an embodiment, the processor 360 may display a list of external electronic devices that can be controlled using the first remote control device through the display module 330. For example, in a case where a communication connection with the first remote control device is established, the processor 360 may display a list of external electronic devices that can be controlled using the first remote control device through the display module 330.

[0164] In an embodiment, the external electronic devices included in the list may include external electronic devices capable of executing a function using the first remote control device through mapping in operation 409 described below.

[0165] In an embodiment, Fig.11 In the embodiment, if a communication connection with the first remote control device 1111 is established, the processor 360 can display information 1121 indicating the TV (for example, the name of the TV and an image indicating the TV), information 1131 indicating the air conditioner (for example, the name of the air conditioner and an image indicating the air conditioner), and information 1141 indicating the air purifier (for example, the name of the air purifier and an image indicating the air purifier) ​​through the display module 330, which can be controlled using the first remote control device 1111.

[0166] In an embodiment, Fig.11 In the embodiment, if a communication connection with the first remote control device 1111 is established and the first remote control device 1111 exists within a range corresponding to the user's line of sight, the processor 360 can display information 1121 indicating the TV, information 1131 indicating the air conditioner, and information 1141 indicating the air purifier through the display module 330, which can be controlled using the first remote control device 1111.

[0167] In an embodiment, the processor 360 may display information 1112 about the first remote control device 1111 in communication connection with the electronic device 301 through the display module 330, such as Fig.11 shown.

[0168] In operation 1003, in an embodiment, the processor 360 may determine the first external electronic device based on the user input when displaying the list. For example, the processor 360 may select one of the external electronic devices included in the list based on the user input. The processor 360 may determine the selected external electronic device as the first external electronic device.

[0169] In an embodiment, Fig.11 In the embodiment, the processor 360 may select the air conditioner as the first external electronic device based on the user's input of the information 1131 indicating the air conditioner among the information 1121 indicating the TV, the information 1131 indicating the air conditioner, and the information 1141 indicating the air purifier. Fig.10 Not shown, if the first external electronic device is selected, the communication module 310 may be controlled such that the communication module 310 establishes a communication connection between the electronic device 301 and the first external electronic device.

[0170] although Fig.11The first external electronic device is selected based on the user's input to the screen displayed on the display 330, but the present disclosure is not limited thereto. For example, the processor 360 may output a voice related to the screen displayed by the display 330 through the speaker 343 and receive the user's voice through the microphone 342, thereby selecting the first external electronic device.

[0171] Return to reference Figure 4 In operation 409, in an embodiment, the processor 360 may perform an operation of mapping at least one button of the first remote control device with at least one function of the first external electronic device.

[0172] In an embodiment, if the electronic device 301 is in communication connection with the first external electronic device, the processor 360 may identify at least one function of the first external electronic device stored in the memory 350. The processor 360 may map the at least one function of the first external electronic device with at least one key of the first remote control device (e.g., a button of the first remote control device).

[0173] In an embodiment, if the electronic device 301 is in communication connection with the first external electronic device, the processor 360 may identify information stored in the memory 350 about at least one key of a remote control device corresponding to the first external electronic device (e.g., a remote control device dedicated to the first external electronic device). For example, the processor 360 may identify functions of the first external electronic device corresponding to a plurality of keys of the remote control device dedicated to the first external electronic device, respectively. For example, the processor 360 may identify information about a mapping relationship between a plurality of keys of the remote control device dedicated to the first external electronic device and a function of the first external electronic device to be executed for the corresponding key if a user input is received (hereinafter, referred to as a "mapping relationship"). In an embodiment, the processor 360 may map at least one function of the first external electronic device to at least one key of the first remote control device based on information about at least one key of the remote control device corresponding to the first external electronic device. For example, the processor 360 may remap (hereinafter, also referred to as "remapping") a plurality of keys of the remote control device dedicated to the first external electronic device (which are mapped to the functions of the first external electronic device) to a plurality of keys of the first remote control device based on the mapping relationship. In an embodiment, the remapping operation may be as follows: when an input to a key of the first remote control device is received, a command corresponding to a key of the remote control device corresponding to the first external electronic device (which is mapped to an input key of the first remote control device) is sent to the first external electronic device.

[0174] In the following, reference will be made to Figures 12a to 15A method for mapping at least one key of a first remote control device with at least one function of a first external electronic device is described.

[0175] Fig.12a is a flowchart 1200a illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0176] Figure 12b FIG1200 b is a diagram showing a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0177] refer to Fig.12a and Figure 12b In operation 1201, in an embodiment, the processor 360 may display an image indicating a plurality of mapping methods through the display module 330. For example, in reference numeral 1201-1, if a communication connection with a first external electronic device is established, the processor 360 may display a virtual image 1210 for selecting a plurality of mapping methods through the display module 330. The virtual image 1210 may include information 1221, 1222 indicating the first external electronic device, information 1211, 1212 indicating the first remote control device 1213, and information 1231 indicating a mapping operation to be performed, and includes an object 1241 corresponding to a first mapping method (hereinafter, also referred to as "custom mapping"), an object 1242 corresponding to a second mapping method (hereinafter, also referred to as "direct mapping"), and an object 1243 corresponding to a third mapping method (hereinafter, also referred to as "simple mapping"). However, the present disclosure is not limited to this, and the virtual image 1210 may include only an object 1241 corresponding to the first mapping method, an object 1242 corresponding to the second mapping method, and an object 1243 corresponding to the third mapping method, without including information 1222 indicating the first external electronic device, information 1211 indicating the first remote control device, and information 1231 indicating that a mapping operation is to be performed.

[0178] In operation 1202, in an embodiment, the processor 360 may determine whether to select a first mapping method from among a plurality of mapping methods.

[0179] In an embodiment, the processor 360 may select the first mapping method based on a user input of the object 1243 corresponding to the first mapping method, the object 1242 corresponding to the second mapping method, and the object 1241 corresponding to the third mapping method.

[0180] In an embodiment, the first mapping method may be a method of mapping at least one key of the first remote control device with at least one function of the first external electronic device based on a user input.

[0181] If the first mapping method is not selected from among the plurality of mapping methods in operation 1202, in an embodiment, the processor 360 may perform a mapping operation by the second mapping method or the third mapping method in operation 1203. Fig.15 A mapping operation performed by the second mapping method or the third mapping method is described.

[0182] If the first mapping method is selected from among the plurality of mapping methods in operation 1202, in an embodiment, the processor 360 may output an object corresponding to the function in the 3D space in operation 1204. For example, the processor 360 may display at least one object corresponding to at least one function of the first external electronic device through the display module 330 so that the user sees that the at least one object is set in the three-dimensional space around the electronic device. For example, the processor 360 may display at least one object 1251, 1252, 1253, 1254, 1255, or 1256 corresponding to at least one function of the first external electronic device through the display module 330, as shown in reference numeral 1202.

[0183] In operation 1205 , in an embodiment, the processor 360 may select an object corresponding to the line of sight of the user.

[0184] In an embodiment, the processor 360 may select an object corresponding to the user's line of sight (e.g., an object set in a direction pointed to by the user's line of sight) from among at least one object corresponding to at least one function of the first external electronic device. For example, in reference numeral 1202, the processor 360 may select object 1254 corresponding to the user's line of sight.

[0185] In operation 1206, in an embodiment, the processor 360 may recognize a key input of the first remote control device.

[0186] In an embodiment, after selecting an object corresponding to the user's line of sight, the processor 360 may receive, through the communication module 310, information from the first remote control device about a key of the first remote control device to which user input is input (which is mapped to a function of the first external electronic device corresponding to the object corresponding to the user's line of sight).

[0187] In operation 1207 , in an embodiment, the processor 360 may map the object and the button.

[0188] Based on the information, the processor 360 may map the function of the first external electronic device corresponding to the object corresponding to the user's line of sight with the key of the first remote control device to which the user input is input. For example, in reference numeral 1202-1, if a user input to a key (e.g., a key configured to increase the volume of the second external electronic device dedicated to the first remote control device before the first remote control device is detected) is received from the first remote control device 1213 after the object 1254 corresponding to the user's line of sight is selected, the function of the first external electronic device corresponding to the selected object 1243 (e.g., a function of lowering the configured temperature of the air conditioner) may be mapped to the key to which the user input is received. In an embodiment, as shown in reference numeral 1202-1, if a user input to the key is received, the processor 360 may display information 1258 indicating the function configured for the key to which the user input is received (e.g., a function configured to increase the volume of the second external electronic device dedicated to the first remote control device before the first remote control device is detected) through the display module 330.

[0189] In an embodiment, after mapping the function of the first external electronic device corresponding to the selected object and the key receiving the user input, the processor 360 may re-execute the mapping operation based on the user input. For example, in reference numeral 1202-1, the processor 360 may map the function of the first external electronic device corresponding to the selected object 1243 (e.g., a function of lowering the configuration temperature of the air conditioner) to the key receiving the user input, and then, if an object 1253 other than the object 1243 is selected and a key of the first remote control device is selected, the function corresponding to the selected object 1253 is mapped to the selected key. In this case, the processor 360 may display information 1257 through the display module 330, indicating that the function corresponding to the selected object 1253 is mapped to the selected key, as shown in reference numeral 1202.

[0190] although Fig.12a and Figure 12b It is shown that the mapping operation is performed based on the user's input to the screen displayed by the display 330 (for example, images indicating a plurality of mapping methods or objects corresponding to functions), but the present disclosure is not limited thereto. For example, the processor 360 may output a voice related to the screen displayed by the display 330 through the speaker 343 and receive the user's voice through the microphone 342, thereby performing the mapping operation.

[0191] Fig.13a is a flowchart 1300a illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0192] Fig.13b FIG1300 b is a diagram showing a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device according to an embodiment.

[0193] refer to Fig.13a and Fig.13b , in operation 1301, in an embodiment, the processor 360 may classify a plurality of functions into groups based on usage history.

[0194] In an embodiment, the processor 360 may classify the multiple functions of the first external electronic device into groups with priorities (e.g., assigned priorities) based on the usage history of the multiple functions of the first external electronic device. The processor 360 may receive information about the usage history of the multiple functions of the first external electronic device (e.g., the number of times each of the multiple functions of the first external electronic device is used within a specified time period) from the server through the communication module 310. The processor 360 may classify the multiple functions of the first external electronic device into groups with priorities in order of high usage history of the multiple functions of the first external electronic device based on the received information.

[0195] In operation 1302 , in an embodiment, the processor 360 may output an object corresponding to the function in a 3D space.

[0196] In an embodiment, the processor 360 may display a plurality of objects corresponding to a plurality of functions of the first external electronic device through the display module 330 according to the priority of the group, so that the user sees that the plurality of objects are set in a three-dimensional space around the electronic device. For example, the processor 360 may display a plurality of objects through the display module 330 based on the user (or electronic device) of the electronic device, so that objects with higher priority are set in a space closer to the user. For example, as shown in reference numeral 1301-1, the processor 360 may configure a first space 1312 (e.g., a first priority function setting area), a second space 1313 (e.g., a second priority function setting area), and a third space 1314 (e.g., a third priority function setting area) based on the user 1311. The processor 360 may display a plurality of objects through the display module 330, so that objects corresponding to functions of a group belonging to a first priority (as the highest priority), objects corresponding to functions of a group belonging to a second priority (as the next priority of the first priority), and objects corresponding to functions of a group belonging to a third priority (as the next priority of the second priority) are set to be visible to the user.

[0197] In an embodiment, as shown in the figure mark 1302-1, in the case where the first external electronic device is an air conditioner, the processor 360 can display objects 1321, 1322, 1323, 1324 corresponding to functions belonging to the first priority group, objects 1331, 1332, 1333, 1334 corresponding to functions belonging to the second priority group, and objects corresponding to functions belonging to the third priority group through the display module 330 based on the priority of the groups into which the functions of the air conditioner are classified.

[0198] In operation 1303 , in an embodiment, the processor 360 may select an object using a virtual laser pointer.

[0199] In an embodiment, the processor 360 may select an object located at a position pointed to by a virtual laser pointer emitted from the first remote control device 1316 visible to the user from among a plurality of objects. For example, in reference numeral 1302-1, when the direction pointed to by the first remote control device 1316 changes, the display of the virtual laser pointer 1317 emitted from the first remote control device 1316 visible to the user may change. The processor 360 may select an object 1322 corresponding to a virtual cylinder 1322-1 located at a position pointed to by the virtual laser pointer 1317, as shown in reference numeral 1302-1. However, although reference numeral 1302-1 shows that a virtual cylinder 1322-1 corresponding to the object 1322 is displayed and the virtual laser pointer 1317 points to the virtual cylinder 1322-1, the present disclosure is not limited thereto. For example, the object 1322 located at a position pointed to by the virtual laser pointer 1317 may be selected without displaying the virtual cylinder 1322-1.

[0200] In operation 1304, in an embodiment, the processor 360 may recognize a key input of the first remote control device.

[0201] In operation 1305, in an embodiment, the processor 360 may map the object and the button.

[0202] In an embodiment, if a user input to a key of the first remote control device is received after an object is selected from among a plurality of objects, the processor 360 may map a function corresponding to the selected object and the key of the first remote control device to which the user input is received. For example, in reference numeral 1303-1, if a user input to a key of the first remote control device (e.g., a key configured to increase the volume of a second external electronic device dedicated to the first remote control device before the first remote control device is detected) is received after the object 1322 is selected, the processor 360 may map the object 1322 to the key. If the object 1322 and the key are mapped, the processor 360 may display information 1351 about the mapping between the object 1322 and the key through the display module 330.

[0203] In an embodiment, after mapping the function of the first external electronic device corresponding to the selected object and the key receiving the user input, the processor 360 may re-perform the mapping operation based on the user input. For example, in reference numeral 1304-1, after mapping the function of the first external electronic device corresponding to the selected object 1322 (e.g., a function of lowering the configured temperature of the air conditioner) and the key receiving the user input, if a key of the first remote control device is selected and if an object 1323 other than the object 1322 (or a virtual cylinder 1323-1 corresponding to another object 1323) is selected using the virtual laser pointer 1317, the processor 360 may map the function corresponding to the selected object 1323 and the selected key. In this case, the processor 360 may display the function corresponding to the selected object 1323 (e.g., a function of increasing the configured temperature of the air conditioner) and information 1361 indicating that the selected key is mapped through the display module 330, as shown in reference numeral 1304-1.

[0204] although Fig.13a and Fig.13b It is shown that the mapping operation is performed based on the user's input to the screen displayed by the display 330 (e.g., the object corresponding to the function and the virtual laser pointer), but the present disclosure is not limited thereto. For example, the processor 360 can perform the mapping operation by outputting a voice related to the screen displayed by the display 330 through the speaker 343 and receiving the user's voice through the microphone 342.

[0205] Figures 14a to 14c 1401 , 1402 , and 1403 are diagrams illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device using a second mapping method according to an embodiment.

[0206] refer to Figures 14a to 14cIn an embodiment, the second mapping method may be a method of mapping at least one button of the first remote control device to at least one function of the first external electronic device using a specified mapping (or a specified mapping relationship). For example, in the second mapping method, the mapping between at least one button of the first remote control device and at least one function of the first external electronic device may be pre-specified (or configured) by a developer of an application for controlling an external electronic device using a remote control device. In an embodiment, the specified mapping between at least one button of the first remote control device and at least one function of the first external electronic device may be stored in a server. The processor 360 may receive a specified mapping between at least one button of the first remote control device and at least one function of the first external electronic device from the server through the communication module 310. The processor 360 may store the specified mapping between at least one button of the first remote control device and at least one function of the first external electronic device received from the server in the memory 350.

[0207] In an embodiment, Fig.14a In the case where the first remote control device is a TV remote controller 1410, a virtual image 1410-1 including a plurality of buttons of an air conditioner remote controller (which are mapped to a plurality of buttons of the TV remote controller 1410) may be displayed on the TV remote controller 1410 through a second mapping method. Fig.14a In the example, the volume control key 1411 of the TV remote controller 1410 can be remapped to the temperature control key of the air conditioner remote controller (or mapped to the temperature control function of the air conditioner). Fig.14a In the embodiment, the channel switching key 1413 of the TV remote controller 1410 can be remapped to the wind speed control key of the air conditioner remote controller (or mapped to the wind speed control function of the air conditioner). Fig.14a In the embodiment, if mapping is performed between at least one button of the first remote control device and at least one function of the first external electronic device, the processor 360 can display an object corresponding to at least one function of the first external electronic device (for example, object 1412 indicating a temperature control key of the air conditioner remote control or object 1414 indicating a wind speed control key of the air conditioner remote control) on at least one button of the first remote control device through the display module 330.

[0208] In an embodiment, Fig.14b In the case where the first remote control device is an air conditioner remote controller 1420 (e.g., an air conditioner remote controller supporting Bluetooth), a virtual image 1420-1 including a plurality of buttons of a TV remote controller (which are mapped to a plurality of buttons of the air conditioner remote controller 1420) may be displayed on the air conditioner remote controller 1420 through a second mapping method. Fig.14b In the example, the temperature control key 1421 of the air conditioner remote controller 1420 can be remapped to the volume control key 1422 of the TV remote controller. Fig.14b In the embodiment, the wind speed control key 1423 of the air conditioner remote controller 1420 can be remapped to the channel switching key 1424 of the TV remote controller. Fig.14b In the embodiment, if mapping is performed between at least one button of the first remote control device and at least one function of the first external electronic device, the processor 360 can display an object corresponding to at least one function of the first external electronic device (for example, object 1422 indicating a volume control key of a TV remote control or object 1424 indicating a channel switching key of a TV remote control) on at least one button of the first remote control device through the display module 330.

[0209] In an embodiment, Fig.14c In the case where the first remote control device is an air conditioner remote controller 1430 (for example, an air conditioner remote controller using IR communication), a virtual image 1430-1 including a plurality of buttons of a TV remote controller (which are mapped to the plurality of buttons of the air conditioner remote controller 1430) may be displayed on the air conditioner remote controller 1430 through a second mapping method. Fig.14c , if remapping is performed between at least one button of the air conditioner remote control 1430 and the TV remote control, the processor 360 can display an object corresponding to at least one function of the TV remote control (for example, object 1432 indicating a volume control key of the TV remote control 1432 or object 1434 indicating a channel switching key of the TV remote control 1434) on at least one button of the air conditioner remote control 1430 (for example, a temperature control key 1431 of the air conditioner remote control 1430 or a wind speed control key 1433 of the air conditioner remote control 1420) through the display module 330.

[0210] Fig.15 is a diagram illustrating a method for mapping at least one key of a first remote control device with at least one function of a first external electronic device using a third mapping method according to an embodiment.

[0211] refer to Fig.15 The third mapping method may include a mapping method in which, if the first remote control device is a designated remote control, at least one function of the first external electronic device can be controlled using at least one key of the designated remote control.

[0212] In an embodiment, the designated remote controller may be a remote controller capable of NUI (Natural User Interface) processing (hereinafter, referred to as a "designated remote controller" or a "simple remote controller"). In an embodiment, the designated remote controller may be an integrated intelligent remote controller capable of communicating with a server and controlling a plurality of external electronic devices in communication connection with the designated remote controller, rather than being dedicated to a specific external electronic device.

[0213] In an embodiment, specifying a remote control may include: Fig.15 1501 in the figure includes only a minimum number of keys (e.g., a power key 1511, a movement key 1512 (e.g., a key for moving the selection), a confirmation key 1513, and a cancel key 1514) for selecting and executing the function of the external electronic device. However, the keys included in the designated remote controller are not limited to the above examples, and the designated remote controller may also include additional keys not described above.

[0214] In an embodiment, reference numerals 1502 and 1503 may indicate an example showing an operation for selecting a function of a first external electronic device so as to control the operation of the first external electronic device through a designated remote controller. In reference numerals 1502 and 1503, in a case where the first external electronic device is a robot cleaner and the type of the first remote control device is a designated remote controller, a virtual image 1520 for controlling the function of the robot cleaner may be displayed by the display module 330. In reference numeral 1502, the image 1520 may include information 1521 indicating that the first external electronic device is a robot cleaner, an object 1522 for turning on / off the robot cleaner, an object 1522 for controlling a cleaning function of the robot cleaner, an object 1523 for controlling a function of a battery of the robot cleaner, and an object 1525 for controlling the suction strength of the robot cleaner. Reference numeral 1502 shows that the object 1522 for turning on / off the robot cleaner is selected (or activated). For example, in order to indicate that the object 1522 for turning on / off the robot cleaner is selected, the object 1522 for turning on / off the robot cleaner may be highlighted. If the user inputs the confirmation key 1513 of the designated remote controller in a state where the object 1522 for turning on / off the robot cleaner is selected, the processor 360 may control the robot cleaner to turn on or off the robot cleaner. In an embodiment, if the user inputs the movement key 1512 of the designated remote controller in a state where the object 1522 for turning on / off the robot cleaner is selected, the processor 360 may move the selection from the object 1522 for turning on / off the robot cleaner to the object 1522 for controlling the cleaning function of the robot cleaner. If the selection is moved to the object 1522 for controlling the cleaning function of the robot cleaner, as shown in reference numeral 1503, the processor 360 may select an object 1532 indicating the details of the cleaning function of the object 1522 for controlling the cleaning function of the robot cleaner. For example, if the selection is moved to the object 1522 for controlling the cleaning function of the robot cleaner, the highlight may be moved from the object 1522 for turning on / off the robot cleaner to the object 1532 indicating the details of the cleaning function.

[0215] In an embodiment, the objects included in the virtual image for controlling the function of the first external electronic device (e.g., the virtual image 1520 for controlling the function of the robot cleaner) may vary depending on the type of the first external electronic device. For example, in the case where the first external electronic device is an air purifier, as shown in reference numeral 1504, the virtual image 1540 may include information 1541 indicating that the first external electronic device is an air purifier, an object 1542 for turning on / off the air purifier, and an object 1543 for controlling the function of the purification intensity of the air purifier. For example, in the case where the first external electronic device is an air conditioner, as shown in reference numeral 1505, the virtual image 1550 may include information 1551 indicating that the first external electronic device is an air conditioner, an object 1552 for turning on / off the air conditioner, and an object 1553 for controlling the function of the air conditioner temperature.

[0216] Return to reference Figure 4 , in operation 411, in an embodiment, the processor 360 may control the first external electronic device to perform at least one function using the first remote control device.

[0217] In an embodiment, if at least one function of the first external electronic device is mapped to at least one key of the first remote control device (or, if at least one key of the remote control device corresponding to the first external electronic device is remapped to at least one key of the first remote control device), the processor 360 may receive a command corresponding to at least one key of the first remote control device receiving the user input through the communication module 310 based on the user input to the at least one key of the first remote control device. The processor 360 may send the command to the first external electronic device through the communication module 310. If the command is received by the first external electronic device, the first external electronic device may execute the function of the first external electronic device corresponding to the command.

[0218] Fig.16 is a diagram 1600 illustrating a method for controlling a first external electronic device to perform at least one function using a first remote control device according to an embodiment.

[0219] refer to Fig.16In an embodiment, after mapping at least one function of the first external electronic device 912 to at least one key of the first remote control device 711, the user input may be input to the first remote control device 711 using the user's finger 1612. The processor 360 may receive a command corresponding to the key of the first remote control device 711 to which the user input is input from the first remote control device 711 (or from the first remote control device 711 via an external electronic device corresponding to the first remote control device 711, and a server) through the communication module 310. If the command is received, the processor 360 may control the first external electronic device to perform the function of the first external electronic device corresponding to the command. For example, if the command is to control the cooling temperature of the first external electronic device (e.g., an air conditioner), the processor 360 may control the first external electronic device so that the first external electronic device controls the cooling temperature. In an embodiment, the processor 360 may display information indicating the function of the first external electronic device corresponding to the key of the first remote control device to which the user input is input through the display module 330. For example, as Fig.16 As shown, the processor 360 may display information 1611 indicating the function of the first external electronic device 912 corresponding to the key of the first remote control device 711 to which the user input is input, through the display module 330, to overlap with the position where the first external electronic device 912 is displayed. However, the present disclosure is not limited thereto, and the processor 360 may output audio indicating that the function of the first external electronic device 912 corresponding to the key of the first remote control device 711 to which the user input is input (or the user input is input to the key of the first remote control device 711) is executed through the speaker.

[0220] Return to reference Figure 4 In an embodiment, if at least one function of the first external electronic device is mapped to at least one key of the first remote control device, the processor 360 may display, through the display module 330, an object indicating at least one function of the first external electronic device mapped to at least one key of the first remote control device to overlap with at least one key of the first remote control device.

[0221] In an embodiment, if at least one key of the remote control device corresponding to the first external electronic device is remapped to at least one key of the first remote control device, the processor 360 may display an object indicating at least one remapped key of the remote control device corresponding to the first external electronic device through the display module 330 to overlap with at least one key of the first remote control device.

[0222] In an embodiment, in a case where the first remote control device is a designated remote control (e.g., a simple remote control), the processor 360 may display, through the display module 330, an image including an object for controlling the first external electronic device at the location of the first external electronic device or the location of the first remote control device (or at a location spaced a designated distance from the location of the first external electronic device or the location of the first remote control device) (e.g., Fig.15 1520, image 1540 or image 1550).

[0223] Fig.17 is a flowchart 1700 illustrating a mode related to switching of a first external electronic device according to an embodiment.

[0224] Fig.18 is a diagram showing a mode associated with switching of a first external electronic device according to an embodiment.

[0225] refer to Fig.17 and Fig.18 , in operation 1701, in an embodiment, in a state where a connection with a first remote control device is established and a first external electronic device is detected (or a communication connection with the first external electronic device is established), the processor 360 may receive a user input for selecting a mode related to switching of the first external electronic device.

[0226] In an embodiment, after establishing a connection with the first remote control device, the processor 360 may establish a communication connection between the electronic device and the first external electronic device by detecting the first external electronic device present in the range corresponding to the user's line of sight. For example, after establishing a connection with the first remote control device, the processor 360 may establish a communication connection between the electronic device 301 and the external electronic device 1 by detecting the external electronic device 1 present in the first range corresponding to the user's line of sight. If the range corresponding to the user's line of sight changes from the first range to the first range, the processor 360 may establish a communication connection between the electronic device 301 and the external electronic device 2 by detecting the external electronic device 2 present in the first range. Hereinafter, a mode for changing the communication connection with the first external electronic device present in the range corresponding to the user's line of sight depending on a change in the range corresponding to the user's line of sight will be referred to as a "first mode (or first mode of the electronic device)" or a "soft link".

[0227] In an embodiment, after the connection with the first remote control device is established, even if another external electronic device other than the first external electronic device is detected to exist in the range corresponding to the current line of sight of the user due to a change in the line of sight of the user, the processor 360 may maintain the communication connection between the electronic device 301 and the first external electronic device. Hereinafter, the mode of maintaining the communication connection with the first external electronic device connected to the electronic device 301 regardless of the change in the range corresponding to the line of sight of the user will be referred to as the "second mode (or the second mode of the electronic device)", "hard link" or "hard lock".

[0228] In an embodiment, the mode related to the switching of the first external electronic device may include a first mode and a second mode.

[0229] In an embodiment, if a designated user input (e.g., a long press input) for selecting a mode (e.g., the first mode or the second mode) related to switching of the first external electronic device is received in the first remote control device, the processor 360 may receive the designated user input from the first remote control device through the communication module 310. However, the present disclosure is not limited thereto, and the processor 360 may receive the designated user input for selecting a mode related to switching of the first external electronic device through an input device (e.g., a microphone or a touch pad) of the electronic device 301.

[0230] In operation 1702, in an embodiment, the processor 360 may select a mode based on a user input. For example, the processor 360 may select a first mode or a second mode.

[0231] In the following, reference will be made to Fig.18 The modes related to the switching of the first external electronic device are described in more detail.

[0232] In an embodiment, in reference numeral 1801, if the first remote control device 1811 and the first external electronic device 1821 are detected, the processor 360 may display indications 1812 and 1822 indicating that the first remote control device 1811 and the first external electronic device 1821 are detected through the display module 330. In an embodiment, if a communication connection with the first remote control device 1811 and the first external electronic device 1821 is established, the processor 360 may display images 1832 and 1833 on a designated area 1831 of the transparent member through the display module 330 indicating that a communication connection with the first remote control device 1811 and the first external electronic device 1821 is established.

[0233] In an embodiment, in reference numeral 1801, if the electronic device is operating in the first mode (soft link), the processor 360 may display information 1842 through the display, which indicates that the electronic device 301 is operating in the first mode (soft link). In an embodiment, in reference numeral 1801, the processor 360 may display information 1841 through the display module 330, which guides the user input for switching the mode of the electronic device 301 from the first mode to the second mode.

[0234] In an embodiment, if a user input for switching the mode of the electronic device 301 from the first mode to the second mode is received from the first remote control device 1811 (for example, if information indicating the user input for switching the mode of the electronic device 301 from the first mode to the second mode is received from the first remote control device 1811), the processor 360 may switch the mode of the electronic device 301 from the first mode to the second mode. In an embodiment, if the mode of the electronic device 301 is switched from the first mode to the second mode, as indicated by reference numeral 1802, the processor 360 may display information 1851 indicating the second mode (for example, information indicating that the electronic device 301 is operating in the second mode) in the designated area 1831 through the display module 330.

[0235] In an embodiment, even if the first external electronic device and another external electronic device are detected to exist in the range corresponding to the current sight of the user due to the change of the sight of the user in the second mode, the processor 360 may maintain the communication connection between the electronic device 301 and the first external electronic device. For example, as shown in reference numeral 1803, even if the first external electronic device 1821 and another external electronic device exist in the range corresponding to the sight of the user due to the change of the range corresponding to the sight of the user, the communication connection between the electronic device 301 and the first external electronic device 1821 may be maintained. In the second mode, even if the first external electronic device 1821 does not exist in the range corresponding to the sight of the user, the processor 360 may control the first external electronic device 1821 using the first remote control device 1811. For example, even if the first external electronic device 1821 does not exist in the range corresponding to the sight of the user, a command for controlling the cooling temperature may be sent to the first external electronic device 1821 (e.g., air conditioner) using the first remote control device 1811. The processor 360 may display information 1881 indicating that a user input corresponding to a command for controlling the cooling temperature of the first external electronic device 1821 (eg, air conditioner) is input to the first remote control device 1811 through the display module 330 .

[0236] Fig.19 is a system diagram 1900 illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0237] refer to Fig.19 In an embodiment, the electronic device 301, the first external electronic device 1950, and the second external electronic device 1930 may register with a server 1940 providing IoT services.

[0238] In operation 1901 , in an embodiment, the electronic device 301 may detect a first remote control device 1920 .

[0239] In operation 1903 , in an embodiment, the electronic device 301 may detect movement of the first remote control device 1920 after detecting the first remote control device 1920 .

[0240] In operation 1903 , in an embodiment, the electronic device 301 may detect movement of the first remote control device 1920 after detecting the first remote control device 1920 .

[0241] In operation 1905 , in an embodiment, if movement of the first remote control device 1920 is detected, the electronic device 301 may send a command transmission request to the server 1940 .

[0242] In operation 1907 , in an embodiment, the server 1940 may send a command transmission request to the second external electronic device 1930 corresponding to the first remote control device 1920 .

[0243] In operation 1909 , in an embodiment, the first remote control device 1920 may transmit a designated command for communication connection with the electronic device 301 to the second external electronic device 1930 .

[0244] In operation 1911 , in an embodiment, the second external electronic device 1930 may transmit a prescribed command to the server 1940 .

[0245] In operation 1913 , in an embodiment, the server 1940 may transmit a designated command to the electronic device 301 .

[0246] In operation 1915 , in an embodiment, the electronic device 301 may establish a communication connection with the first remote control device 1920 in response to receiving the designated command.

[0247] In operation 1917 , in an embodiment, the electronic device 301 may determine a first external electronic device 1950 to perform a function using the first remote control device 1920 .

[0248] In operation 1919 , in an embodiment, the electronic device 301 may establish a communication connection with the first external electronic device 1950 .

[0249] In operation 1921 , in an embodiment, the electronic device 301 may perform a mapping operation for controlling the first external electronic device 1950 using the first remote control device 1920 .

[0250] In operation 1923 , in an embodiment, the first remote control device 1920 may transmit a command corresponding to the user input.

[0251] In operation 1925 , in an embodiment, the electronic device 301 may send a command to the second external electronic device 1930 based on the command from the first remote control device 1920 , so that the second external electronic device 1930 performs a function corresponding to the command.

[0252] Despite the above Figures 4 to 19 It is shown that after establishing a communication connection with the first remote control device 1920, the first external electronic device 1950 is controlled using the first remote control device 1920 by establishing a communication connection with the first external electronic device 1950, but the present disclosure is not limited thereto. Fig. 20 and Fig.21 An example is described in which, after a communication connection with the first external electronic device 1950 is established, the first external electronic device 1950 is controlled using the first remote control device 1920 by establishing a communication connection with the first remote control device 1920 .

[0253] Fig. 20 is a flowchart 2000 illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0254] Fig.21 is a diagram 2100 illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0255] refer to Fig. 20 and Fig.21 , in operation 2001 , in an embodiment, the processor 360 may detect a first external electronic device through the communication module 310 .

[0256] In an embodiment, the processor 360 may use the camera module 320 (e.g., Figure 2 The processor 360 can detect the sight of the user wearing the electronic device through the camera module 320 (for example, Figure 2 The processor 360 may detect the first external electronic device by comparing the obtained image with a plurality of images stored in the memory 350 and including the external electronic device.

[0257] In an embodiment, the processor 360 may use the camera module 320 (e.g., Figure 2 The processor 360 may use one or more first cameras 211-1 and 211-2 in the memory 350 to obtain an image of a first external electronic device within the field of view of the camera module 320 (e.g., a specified area from the center of the field of view of the camera module 320). The processor 360 may compare the obtained image with a plurality of images including external electronic devices stored in the memory 350, thereby detecting the first external electronic device.

[0258] Although the above example shows that the processor 360 detects the first external electronic device based on the image obtained by the camera module 320, the present disclosure is not limited thereto. For example, the processor 360 may detect the first external electronic device located in the direction facing the electronic device through the communication module 310 (e.g., an ultra-wideband (UWB) communication module 310).

[0259] In an embodiment, if the first external electronic device is detected, the processor 360 may output information indicating that the first external electronic device is detected.

[0260] In an embodiment, if a first external electronic device is detected, the processor 360 may request information about the first external electronic device from the server through the communication module 310. The processor 360 may send the request through the communication module 310 and then receive information about the first external electronic device from the server. The information about the first external electronic device may include the type, name and model name of the first external electronic device, the function performed by the first external electronic device, the operating state, and / or information of the communication connection with the first external electronic device. However, the information about the first external electronic device is not limited to the above examples.

[0261] In operation 2003, in an embodiment, the processor 360 may control the communication module 310 so that the communication module 310 establishes a communication connection between the electronic device 301 and the first external electronic device. For example, in a state where the first external electronic device is detected, the processor 360 may establish a communication connection between the electronic device 301 and the first external electronic device through a short-range communication module (e.g., a Bluetooth module or a Wi-Fi module) based on a designated user input to the electronic device 301 (or the first remote control device).

[0262] In an embodiment, if a first external electronic device is detected or a communication connection is established between the electronic device 301 and the first external electronic device, the processor 360 may obtain information about at least one remote control device capable of controlling the first external electronic device. For example, if a first external electronic device is detected or a communication connection is established between the electronic device 301 and the first external electronic device, the processor 360 may request information about at least one remote control device capable of controlling the first external electronic device from the server through the communication module 310. After sending the request, the processor 360 may receive information about at least one remote control device capable of controlling the first external electronic device from the server through the communication module 310. The information about at least one remote control device received from the server may be a list of protocols, manufacturer codes, and data codes of at least one remote control device. If the remote control device uses IR communication, the date code is a command code for identifying an IR command, and may be an IR signal value corresponding to each of at least one key of the remote control device.

[0263] In an embodiment, if information about at least one remote control device capable of controlling the first external electronic device is received, the processor 360 may display the received information through the display module 330. Fig.21 In the embodiment, the processor 360 may display information 2102 about at least one remote control device capable of controlling the first external electronic device at a position adjacent to (or at least partially overlapping with) an area on the transparent member where the first external electronic device 2101 is visible through the display module 330. The information 2102 may include an image 2110 and information 2111 of the remote control device 1, an image 2120 and information 2121 of the remote control device 2, and an image 2130 and information 2131 of the remote control device 3. However, the information 2102 of at least one remote control device capable of controlling the first external electronic device is not limited to the above examples.

[0264] Return to reference Fig. 20 In operation 2005, in an embodiment, the processor 360 may detect a first remote control device.

[0265] Since Operation 2005 is at least partially related to Figure 4 The operation 401 in FIG. 4 is the same or similar, so its repeated description will be omitted.

[0266] In an embodiment, if the first external electronic device is detected or a user input is received after a communication connection is established between the electronic device and the first external electronic device, the processor 360 may perform an operation for detecting the first remote control device.

[0267] In operation 2007, in an embodiment, the processor 360 may establish a connection with the first remote control device.

[0268] Since Operation 2007 is at least partially related to Figure 4 The operation 403 in FIG. 4 is the same or similar, so its repeated description will be omitted.

[0269] In operation 2009, in an embodiment, the processor 360 may perform an operation for mapping at least one key of the first remote control device with at least one function of the first external electronic device.

[0270] Since Operation 2009 was at least partially related to Figure 4 The operation 409 in FIG. 4 is the same or similar, so its repeated description will be omitted.

[0271] In operation 2011, in an embodiment, the processor 360 may control the first external electronic device to perform at least one function using the first remote control device.

[0272] Since Operation 2011 was at least partially related to Figure 4 The operation 411 in is the same or similar, so its repeated description will be omitted.

[0273] although Fig. 20 and Fig.21 It is shown that the operation for controlling the remote control device is performed based on the user's input to the screen displayed by the display 330, but the present disclosure is not limited thereto. For example, the processor 360 can output a voice related to the screen displayed on the display 330 through the speaker 343 and receive the user's voice through the microphone 342, thereby performing the operation for controlling the remote control device.

[0274] Fig. 22 is a diagram 2200 illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0275] refer to Fig. 22 In an embodiment, when the user wears the electronic device 301, the processor 360 may cause the electronic device 301 to enter a mode for controlling an external electronic device (home control mode) based on user input.

[0276] In an embodiment, if the electronic device 301 enters a mode for controlling an external electronic device, the processor 360 may detect at least one external electronic device located within a range corresponding to the user's line of sight. Fig. 22In the case where the bathroom is registered in the server as a designated place (e.g., a room), the processor 360 may detect at least one external electronic device (e.g., a toilet seat 2201, a bathtub shower 2202, a cabinet 2203, a washstand 2204, or a locker 2205) provided inside the bathroom within a range corresponding to the user's line of sight. If at least one external electronic device is detected, the processor 360 may display an indication (or a highlighted mark) indicating that at least one external electronic device is detected and information 2211, 2212, 2213, 2214, and 2215 about the at least one external electronic device through the display module 330. If at least one external electronic device is detected, the processor 360 may output audio through a speaker, the audio indicating that at least one external electronic device is detected.

[0277] Fig.23a is a flowchart 2300a illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0278] Figure 23b FIG2300 b is a diagram illustrating a method for controlling an external electronic device using a remote control device according to an embodiment.

[0279] refer to Fig.23a and Figure 23b In operation 2301, in an embodiment, the processor 360 may detect a first external electronic device. Figure 12b In the reference numeral 2301 - 1 in FIG. 1 , a cabinet 2312 existing within a range corresponding to a user's line of sight may be detected as a first external electronic device to be controlled using a first remote control device 2311 .

[0280] In an embodiment, if the cabinet 2312 is detected, the processor 360 may output an indication and audio indicating that the cabinet 2312 is detected and information 2313 indicating that the designated location is a bathroom.

[0281] In operation 2302, in an embodiment, the processor 360 may display a function of the first external electronic device.

[0282] In an embodiment, if the type of the first remote control device 2311 is a simple remote controller, the processor 360 may display, through the display module 330, a virtual image 2321 or 2322 for controlling the function of the cabinet 2312 to be displayed adjacent to the first remote control device 2311 or the cabinet 2312. The virtual image 2321 or 2322 may include a plurality of objects indicating a plurality of functions of the cabinet 2312 (e.g., a light function (a function for controlling the type and intensity of light), a screen function (a function related to the screen of the cabinet 2312), and a digital device (DA) function).

[0283] In operation 2303 , in an embodiment, the processor 360 may control a function of the first external electronic device using the first remote control device.

[0284] In an embodiment, the processor 360 may control the function of the cabinet 2312 using the first remote control device 2311 when displaying the virtual image 2321 or 2322. For example, the processor 360 may move the highlight for selecting an object indicating the function of the cabinet 2312 included in the virtual image 2321 or 2322, or select the object using the first remote control device 2311, as shown in reference numerals 2302-1 and 2303-1. For example, the processor 360 may select (and execute) the light function of the cabinet 2312, as shown in reference numeral 2302-1. For example, the processor 360 may select (and execute) the power on / off function of the cabinet 2312, as shown in reference numeral 2302-1.

[0285] The method for controlling an external electronic device using a remote control device in an electronic device 301 according to an embodiment may include detecting a first remote control device. The method may include establishing a communication connection with the first remote control device through a communication module 310 of the electronic device 301. The method may include determining a first external electronic device to be controlled using the first remote control device. The method may include establishing a communication connection between the electronic device 301 and the first external electronic device through the communication module 310. The method may include performing an operation for mapping at least one key of the first remote control device to at least one function of the first external electronic device. The method may include using the first remote control device to control the first external electronic device to perform at least one function.

[0286] In an embodiment, detecting the first remote control device may include: obtaining an image through the camera module 320 of the electronic device 301 , and based on the image, detecting whether the first remote control device exists within a range corresponding to a line of sight of the user wearing the electronic device 301 .

[0287] In an embodiment, establishing a communication connection with a first remote control device may include: detecting movement of the first remote control device; based on detecting movement of the first remote control device, requesting a second external electronic device corresponding to the first remote control device to forward a command of the first remote control device to the electronic device 301; and establishing a communication connection with the first remote control device.

[0288] In an embodiment, establishing a communication connection with a first remote control device may include: receiving, through a communication module 310, user input input to the first remote control device from the first remote control device via a second external electronic device and a server, the user input being used for a communication connection with the first remote control device; and establishing a communication connection with the first remote control device based on the user input.

[0289] In an embodiment, determining the first external electronic device may include: detecting the first external electronic device, and establishing a communication connection between the first external electronic device and the electronic device 301 .

[0290] In an embodiment, determining the first external electronic device may include: outputting a list of external electronic devices that can be controlled using the first remote control device based on establishing a communication connection with the first remote control device; determining the first external electronic device from the external electronic devices based on user input to the list; and establishing a communication connection between the first external electronic device and the electronic device 301.

[0291] In an embodiment, performing a mapping operation may include: displaying at least one object corresponding to at least one function of a first external electronic device through a display module 330 of the electronic device 301, so that the at least one object is seen by a user wearing the electronic device (301) and is set in a three-dimensional space around the electronic device (301); selecting an object corresponding to the user's line of sight from among the at least one object; receiving information about a key of the first remote control device, a user input input to the key, and a function corresponding to the object to be mapped to the key from the first remote control device through a communication module 310; and mapping the function corresponding to the selected object to the key of the first remote control device.

[0292] In an embodiment, displaying at least one object through the display module 330 of the electronic device 301 may include: classifying at least one function of the first external electronic device into groups with priorities based on a user's usage history of each of the at least one function, and displaying the at least one object through the display module 330 based on the priority of the group so that the user sees the at least one object arranged in a three-dimensional space around the electronic device 301.

[0293] In an embodiment, using a first remote control device to control a first external electronic device to perform at least one function may include: if at least one function of the first external electronic device is mapped to at least one key of the first remote control device, an object to be overlapped with the at least one key of the first remote control device is displayed through the display module 330 of the electronic device 301, and the object indicates at least one mapped function.

[0294] In addition, the data structures used in the embodiments of the present document described above can be recorded on a computer-readable recording medium in various ways. Computer-readable recording media include storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.), optical reading media (e.g., CD-ROM, DVD, etc.), etc.

Claims

1. An electronic device (301), comprising: Communication module (310); as well as at least one processor (360); Wherein, the at least one processor (360) is configured to: detecting a first remote control device; establishing a communication connection with the first remote control device via the communication module (310); determining a first external electronic device to be controlled using the first remote control device; Controlling the communication module (310) so that the communication module (310) establishes a communication connection between the electronic device (301) and the first external electronic device; performing an operation of mapping at least one button of the first remote control device to at least one function of the first external electronic device; and The first external electronic device is controlled to perform the at least one function using the first remote control device.

2. The electronic device according to claim 1, further comprising a camera module (320), in, The at least one processor (360) is configured to: acquiring an image through the camera module (320); and Based on the image, the first remote control device existing in a range corresponding to the line of sight of a user wearing the electronic device (301) is detected.

3. The electronic device according to claim 1 or 2, wherein: The at least one processor (360) is configured to: detecting movement of the first remote control device; Based on detecting the movement of the first remote control device, requesting a second external electronic device corresponding to the first remote control device to forward a command of the first remote control device to the electronic device (301); as well as A communication connection is established with the first remote control device.

4. The electronic device according to any one of claims 1 to 3, wherein: The at least one processor (360) is configured to: receiving, by the communication module (310), a user input input to the first remote control device from the first remote control device via the second external electronic device and a server, the user input being input for a communication connection with the first remote control device; and Based on the user input, a communication connection is established with the first remote control device.

5. The electronic device according to any one of claims 1 to 4, wherein: The at least one processor (360) is configured to: detecting the first external electronic device; as well as The communication module (310) is controlled so that the communication module (310) establishes a communication connection between the first external electronic device and the electronic device (301).

6. The electronic device according to any one of claims 1 to 5, wherein: The at least one processor (360) is configured to: outputting a list of external electronic devices that can be controlled using the first remote control device based on the communication connection with the first remote control device being established; determining the first external electronic device from among the external electronic devices based on a user input to the list; as well as The communication module (310) is controlled so that the communication module (310) establishes a communication connection between the first external electronic device and the electronic device (301).

7. The electronic device according to any one of claims 1 to 6, wherein: The at least one processor (360) is configured to: Displaying at least one object corresponding to at least one function of the first external electronic device through a display module (330) of the electronic device (301), so that a user wearing the electronic device (301) sees that the at least one object is arranged in a three-dimensional space around the electronic device (301); selecting an object corresponding to the line of sight of the user from among the at least one object; receiving, through the communication module (310), information about keys of the first remote control device, user inputs input to the keys, and functions corresponding to objects to be mapped to the keys from the first remote control device; and A function corresponding to the selected object is mapped to a key of the first remote control device.

8. The electronic device according to any one of claims 1 to 7, wherein: The at least one processor (360) is configured to: classifying the at least one function of the first external electronic device into a group having a priority based on a usage history of each function of the at least one function of the first external electronic device by the user; as well as Based on the priority of the group, the at least one object is displayed by the display module (330) so that the user sees that the at least one object is arranged in a three-dimensional space around the electronic device (301).

9. The electronic device according to any one of claims 1 to 8, wherein: The at least one processor (360) is configured to: In a case where the at least one function of the first external electronic device is mapped to at least one key of the first remote control device, an object to be overlapped with the at least one key of the first remote control device is displayed through the display module (330) of the electronic device (301), the object indicating the at least one mapped function.

10. A method for controlling an external electronic device using a remote control device in an electronic device (301), the method comprising: detecting a first remote control device; Establishing a communication connection with the first remote control device through a communication module (310) of the electronic device; determining a first external electronic device to be controlled using the first remote control device; Controlling the communication module (310) so that the communication module (310) establishes a communication connection between the electronic device (301) and the first external electronic device; Performing an operation of mapping at least one button of the first remote control device to at least one function of the first external electronic device; as well as The first external electronic device is controlled to perform the at least one function using the first remote control device.

11. The method according to claim 10, wherein: Detecting the first remote control device includes: obtaining an image through a camera module (320) of the electronic device (301); and Based on the image, the first remote control device existing in a range corresponding to the line of sight of a user wearing the electronic device (301) is detected.

12. The method according to claim 10 or 11, wherein: Establishing a communication connection with the first remote control device includes: detecting movement of the first remote control device; Based on detecting the movement of the first remote control device, requesting a second external electronic device corresponding to the first remote control device to forward a command of the first remote control device to the electronic device (301); and A communication connection is established with the first remote control device.

13. The method according to any one of claims 10 to 12, wherein: Establishing a communication connection with the first remote control device includes: receiving, by the communication module (310), a user input input to the first remote control device from the first remote control device via the second external electronic device and a server, the user input being for a communication connection with the first remote control device; and Based on the user input, a communication connection is established with the first remote control device.

14. The method according to any one of claims 10 to 13, wherein: Determining the first external electronic device includes: detecting the first external electronic device; and A communication connection is established between the first external electronic device and the electronic device (301).

15. The method according to any one of claims 10 to 14, wherein: Determining the first external electronic device includes: outputting a list of external electronic devices that can be controlled using the first remote control device based on the communication connection with the first remote control device being established; determining the first external electronic device from among the external electronic devices based on a user input to the list; and A communication connection is established between the first external electronic device and the electronic device (301).

Images