Electronic device and control method thereof

Abstract

According to an embodiment, an electronic device may comprise: a display; a communication module; at least one processor comprising a processing circuit; and a memory comprising at least one storage medium storing instructions. The instructions, when executed individually or collectively by the at least one processor, may cause the electronic device to: on the basis that one first home appliance is selected from among at least one home appliance, display, on the display, augmented reality displaying the first home appliance and the surrounding environment of the first home appliance; on the basis of information corresponding to the operation intensity of the first home appliance, display, within the augmented reality, a first object corresponding to the operation intensity of the first home appliance; and on the basis of a user input for enlarging or reducing the size or volume of the first object, transmit, to the first home appliance, a signal for increasing or decreasing the operation intensity of the first home appliance.

Description

Electronic device and method of controlling the same

[0001] Various embodiments of the present disclosure relate to an electronic device and a method for controlling the same.

[0002] As technology advances and smart home environments become commonplace, the level of user experience is rising. The accuracy and performance of indoor spatial computing technology have been significantly improved through the use of LiDAR scanners integrated into electronic devices, and the structure of a home can also be identified through AI image recognition.

[0003] A single electronic device is connected to communicate with various home appliances placed within the house, allowing the user to remotely control various types of home appliances through the electronic device.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment, the electronic device may include at least one processor comprising a display, a communication module, and a processing circuit, and a memory comprising at least one storage medium for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the first appliance to display an augmented reality on the display that displays the first appliance and the surrounding environment of the first appliance based on the selection of one of at least one appliance, display a first object corresponding to the operating intensity of the first appliance in the augmented reality based on information corresponding to the operating intensity of the first appliance, and transmit a signal to the first appliance to increase or decrease the operating intensity of the first appliance based on user input that enlarges or reduces the size or volume of the first object.

[0006] A control method for an electronic device according to one embodiment may include the steps of: displaying augmented reality based on the selection of a first home appliance; displaying a first object in the augmented reality based on the operating intensity of the first home appliance; enlarging or reducing the size or volume of the first object based on user input; and transmitting a signal to the first home appliance to increase or decrease the operating intensity of the first home appliance based on the enlarging or reducing of the size or volume of the first object.

[0007] The effects obtainable in the exemplary embodiments of the present disclosure, which cause to reduce signals, are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure.

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0009] FIG. 2 is a perspective view looking toward the front of an electronic device according to one embodiment of the present disclosure.

[0010] FIG. 3 is a perspective view looking toward the rear of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating a method of operation of an electronic device according to one embodiment.

[0012] FIG. 5 is a diagram illustrating the operation flow between an electronic device and a first home appliance according to one embodiment.

[0013] FIGS. 6a, FIGS. 6b, and FIGS. 6c are drawings for explaining a method of controlling the operating intensity of a first home appliance using an electronic device according to one embodiment.

[0014] FIGS. 7a and 7b are drawings for explaining a method of controlling the direction of operation of a first home appliance using an electronic device according to one embodiment.

[0015] FIGS. 8a and FIGS. 8b are drawings showing coordinates corresponding to the location of the second object shown in FIGS. 7a and FIGS. 7b, respectively.

[0016] FIGS. 9a and FIGS. 9b are exemplary drawings for explaining a method of controlling the direction of operation of a first home appliance based on the position of a second object shown in FIGS. 7a and FIGS. 7b, respectively.

[0017] FIG. 10 is a flowchart illustrating a method of operation of an electronic device according to one embodiment.

[0018] FIG. 11 is a diagram illustrating the operation flow between an electronic device and a first home appliance according to one embodiment.

[0019] In the following description, the attached drawings are referenced, and specific examples of implementation are illustrated within the drawings. Additionally, other examples may be used and structural modifications may be made without departing from the scope of the various examples.

[0020] The various embodiments used to illustrate the principles of the present disclosure in FIGS. 1 through 11 disclosed below and in this patent document are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure in any way. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any system or device appropriately arranged.

[0021] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0022] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0023] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0024] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0025] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0026] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0027] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0028] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0029] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0030] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0031] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0032] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0033] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0034] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0035] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0036] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0037] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0038] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0039] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0040] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0041] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197). According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0042] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0043] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0044] FIG. 2 is a perspective view looking toward the front of an electronic device according to one embodiment of the present disclosure.

[0045] FIG. 3 is a perspective view looking toward the rear of an electronic device according to one embodiment of the present disclosure.

[0046] The components of the electronic device (101) of FIGS. 2 and 3 may be some or all identical to the components of the electronic device (101) of FIG. 1.

[0047] Referring to FIGS. 2 and 3, an electronic device (101) according to one embodiment (e.g., the electronic device (101) of FIG. 1) may include a housing (110) comprising a first surface (or front) (110A), a second surface (or rear) (110B), and a side (110C) surrounding the space between the first surface (110A) and the second surface (110B). In one embodiment (not shown), the housing (110) may refer to a structure forming some of the first surface (110A) of FIG. 2, the second surface (110B) of FIG. 3, and the side (110C).

[0048] According to one embodiment, the first surface (110A) may be formed by a front plate (101a) in which at least a portion is substantially transparent (e.g., a glass plate containing various coating layers, or a polymer plate). The second surface (110B) may be formed by a rear plate (111) that is substantially opaque. The rear plate (111) may be formed by, for example, coated or colored glass, ceramic, polymer, metal (e.g., aluminum, stainless steel (STS), or magnesium), or a combination of at least two of the above materials. The side surface (110C) may be formed by a side structure (or "side bezel structure") (118) that combines with the front plate (101a) and the rear plate (111) and comprises metal and / or polymer. In one embodiment, the rear plate (111) and the side structure (118) may be formed integrally and may comprise the same material (e.g., a metallic material such as aluminum).

[0049] According to one embodiment, the front plate (101a) may include region(s) that are curved and seamlessly extended toward the rear plate (111) at least a portion of the edge. For example, the front plate (101a) (or rear plate (111)) may include only one of the regions that are curved and extended toward the rear plate (111) (or front plate (101a)) at one edge of the first surface (110A). According to one embodiment, the front plate (101a) or the rear plate (111) may be substantially flat, in which case it may not include the curved and extended region. If the front plate (101a) or the rear plate (111) includes the curved and extended region, the thickness of the electronic device (101) in the portion including the curved and extended region may be smaller than the thickness of the other portion.

[0050] According to one embodiment, the electronic device (101) may include at least one of a display (115), an audio module (e.g., a microphone hole (103), an external speaker hole (107), a call receiver hole (114)), a sensor module (e.g., a first sensor module (101b), a second sensor module (not shown), a third sensor module (119)), a camera module (e.g., a first camera device (105), a second camera device (112), a flash (113)), a key input device (117), a light-emitting element (106), and a connector hole (e.g., a first connector hole (101c), a second connector hole (109)). In one embodiment, the electronic device (101) may omit at least one of the components (e.g., a key input device (117), or a light-emitting element (106)) or additionally include other components.

[0051] The display (115) may output a screen or be visually exposed through a significant portion of, for example, the first surface (110A) (e.g., the front plate (101a)). In one embodiment, at least a portion of the display (115) may be visually exposed through the front plate (101a) forming the first surface (110A) or through a portion of the side (110C). In one embodiment, the corners of the display (115) may be formed to be generally identical to the adjacent outer shape of the front plate (101a). In one embodiment (not shown), to expand the area where the display (115) is visually exposed, the gap between the outer edge of the display (115) and the outer edge of the front plate (101a) may be formed to be generally identical.

[0052] According to one embodiment, a recess or opening is formed in a part of the screen display area of ​​the display (115), and the electronic device (101) may include at least one of an audio module (e.g., a call receiver hole (114)), a sensor module (e.g., a first sensor module (101b)), a camera module (e.g., a first camera device (105)), and a light-emitting element (106) that are aligned with the recess or the opening. According to one embodiment, at least one of an audio module (e.g., a call receiver hole (114)), a sensor module (e.g., a first sensor module (101b)), a camera module (e.g., a first camera device (105)), a fingerprint sensor (not shown), and a light-emitting element (106) may be disposed on the back surface of the screen display area of ​​the display (115). According to one embodiment, the display (115) may be combined with or placed adjacent to a touch detection circuit, a pressure sensor capable of measuring the intensity (pressure) of the touch, and / or a digitizer capable of detecting a magnetic field type stylus pen.

[0053] According to one embodiment, the audio module (103, 107, 114) may include a microphone hole (103) and a speaker hole (e.g., an external speaker hole (107), a call receiver hole (114)). A microphone for acquiring external sound may be placed inside the microphone hole (103), and in one embodiment, a plurality of microphones may be placed to detect the direction of sound. The speaker hole may include an external speaker hole (107) and a call receiver hole (114). In one embodiment, the speaker hole (e.g., an external speaker hole (107), a call receiver hole (114)) and the microphone hole (103) may be implemented as a single hole, or a speaker may be included without a speaker hole (e.g., an external speaker hole (107), a call receiver hole (114)) (e.g., a piezo speaker).

[0054] According to one embodiment, the sensor module may generate an electrical signal or data value corresponding to an internal operating state of the electronic device (101) or an external environmental state. The sensor module may include, for example, a first sensor module (101b) (e.g., a proximity sensor) and / or a second sensor module (not shown) (e.g., a fingerprint sensor) disposed on a first surface (110A) of the housing (110), and / or a third sensor module (119) disposed on a second surface (110B) of the housing (110). The second sensor module (not shown) (e.g., a fingerprint sensor) may be disposed on the second surface (110B) or side (110C) as well as on the first surface (110A) (e.g., a display (115)) of the housing (110). The electronic device (101) may further include at least one of, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor (104).

[0055] According to one embodiment, the camera module may include a first camera device (105) disposed on a first surface (110A) of the electronic device (101), a second camera device (112) disposed on a second surface (110B), and / or a flash (113). The camera devices (e.g., the first camera device (105), the second camera device (112)) may include one or more lenses, an image sensor, and / or an image signal processor. The flash (113) may include, for example, a light-emitting diode or a xenon lamp. In one embodiment, one or more lenses (infrared camera, wide-angle and telephoto lenses) and image sensors may be disposed on one surface of the electronic device (101). In one embodiment, the flash (113) may emit infrared light, and the infrared light emitted by the flash (113) and reflected by the subject may be received through a third sensor module (119). The electronic device (101) or the processor of the electronic device (101) (e.g., the processor (120) of FIG. 1) can detect depth information of the subject based on the time when infrared light is received from the third sensor module (119).

[0056] According to one embodiment, a key input device (117) may be disposed on a side (110C) of the housing (110). In one embodiment, the electronic device (101) may not include some or all of the aforementioned key input devices (117), and the key input devices (117) that are not included may be implemented in other forms, such as soft keys, on the display (115). In one embodiment, the key input device may include a sensor module disposed on a second side (110B) of the housing (110).

[0057] According to one embodiment, the light-emitting element (106) may be disposed, for example, on a first surface (110A) of a housing (110). The light-emitting element (106) may, for example, provide state information of an electronic device (101) in the form of light. In one embodiment, the light-emitting element (106) may, for example, provide a light source that is coupled with the operation of a camera module (e.g., a first camera device (105)). The light-emitting element (106) may include, for example, an LED, an IR LED, and a xenon lamp.

[0058] According to one embodiment, the connector hole (e.g., first connector hole (101c), second connector hole (109)) may include a first connector hole (101c) capable of receiving a connector (e.g., a USB connector) for transmitting and receiving power and / or data with an external electronic device (e.g., the electronic device (1002) of FIG. 1), and a second connector hole (e.g., an earphone jack) (109) capable of receiving a connector for transmitting and receiving audio signals with an external electronic device.

[0059] According to one embodiment, the electronic device (101) may include a touch panel. The touch panel may detect a user's finger gesture input and output a touch event value corresponding to the detected touch signal. The touch panel may be implemented as either a capacitive method or a resistive method. Here, the capacitive method is a method that calculates touch coordinates by detecting minute electric currents generated by the user's body. Here, the resistive method is a method that calculates touch coordinates by detecting that current flows when the upper and lower plates of the touch point come into contact, including two electrode plates embedded in the touch panel. The touch panel may be included in the display (115). Alternatively, the touch panel may be placed on the lower side of the display (115) as a separate configuration from the display (115).

[0060] FIG. 4 is a flowchart illustrating a method of operation of an electronic device according to one embodiment.

[0061] FIG. 4 illustrates a flowchart of the operation method of the electronic device of FIG. 1 to 3 (e.g., the electronic device (101) of FIG. 1 to 3).

[0062] The flowchart illustrated in FIG. 4 is exemplary, and the method of operation of the electronic device (101) is not limited to the order of the flowchart illustrated.

[0063] The embodiment of FIG. 4 can be optionally combined with at least one of the embodiments of FIG. 1 to 3 and the embodiments of FIG. 5 to 11.

[0064] According to one embodiment, the electronic device (101) may include a processor (e.g., the processor (120) of FIG. 1) and a memory (e.g., the memory (130) of FIG. 1). The processor (120) may include a processing circuit. The memory (130) may include at least one storage medium for storing instructions.

[0065] Referring to FIG. 4, according to one embodiment, the method of operation of an electronic device (101) may include an operation (410) of selecting a first appliance among a plurality of appliances capable of communicating with the electronic device (101).

[0066] Here, the first household appliance may include an air conditioner, an air purifier, a lighting device, a humidifier, or a dehumidifier.

[0067] According to one embodiment, the air conditioner may include an indoor unit including a blower, an outdoor unit, and a refrigerant circulation device configured to circulate refrigerant between the indoor unit and the outdoor unit.

[0068] According to one embodiment, the air purifier may include a convection fan that forms air convection.

[0069] According to one embodiment, the lighting device may include a light source.

[0070] According to one embodiment, the humidifier may include a water tank and a humidifying blower fan arranged to supply moisture to the air using the water contained in the water tank.

[0071] According to one embodiment, the dehumidifier may include a heat pump arranged to dehumidify moisture in the air and a dehumidifying blower fan arranged to diffuse the dehumidified air into the air.

[0072] According to one embodiment, the processor (120) can recognize or identify at least one home appliance placed in a house using a camera module (e.g., the camera module (180) of FIG. 1). One of the at least one home appliance may be selected as the first home appliance.

[0073] According to one embodiment, the processor (120) can select one of at least one home appliance as the first home appliance using a signal received from a touch panel. The first home appliance can be selected by a user touching one of the home appliances on an image captured by a camera module (180) displayed on a display (115).

[0074] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to recognize or identify at least one home appliance placed in the house using the camera module (180).

[0075] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to select one of at least one home appliance as the first home appliance using a signal received from a touch panel.

[0076] According to one embodiment, the method of operation of the electronic device (101) may include an operation (420) of displaying an augmented reality displaying a selected first home appliance and the surrounding environment of the first home appliance on a display (e.g., a display (115) of FIG. 2).

[0077] According to one embodiment, the processor (120) may perform the operation of displaying augmented reality on the display (115) that displays the first home appliance and the surrounding environment of the first home appliance in response to the selection of the first home appliance. Here, the augmented reality may be stored in advance in memory (130). The augmented reality may reflect the indoor structure of the house and the arrangement of surrounding furniture. The augmented reality may be determined by sensors included in the electronic device (101) or the home appliance and stored in memory (130). The augmented reality may be set to be modified directly by the user.

[0078] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to perform an operation of displaying the first home appliance and the surrounding environment of the first home appliance on the display (115) in response to the selection of the first home appliance.

[0079] According to one embodiment, the method of operation of the electronic device (101) may include an operation (430) of requesting information corresponding to the operating intensity or operating direction from the first appliance.

[0080] For example, if the first home appliance is an air conditioner, the information corresponding to the operating intensity may be information corresponding to the rotational speed of a blower. For example, if the first home appliance is an air purifier, the information corresponding to the operating intensity may be information corresponding to the rotational speed of a convection fan. For example, if the first home appliance is a lighting device, the information corresponding to the operating intensity may be information corresponding to the output of a light source. For example, if the first home appliance is a humidifier, the information corresponding to the operating intensity may be information corresponding to the rotational speed of a humidifying blower fan. For example, if the first home appliance is a dehumidifier, the information corresponding to the operating intensity may be information corresponding to the rotational speed of a dehumidifying blower fan.

[0081] According to one embodiment, the processor (120) can request information corresponding to the operating intensity from the first home appliance through a communication module (e.g., the communication module (190) of FIG. 1).

[0082] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to request information corresponding to the operating intensity from the first home appliance through the communication module (190).

[0083] According to one embodiment, the processor (120) can request information corresponding to the direction of operation from the first home appliance through the communication module (190).

[0084] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to request information corresponding to the direction of operation from the first home appliance through the communication module (190).

[0085] According to one embodiment, the method of operation of the electronic device (101) may include an operation (440) of displaying a first object (e.g., the first object (620) of FIG. 6a) or a second object (e.g., the second object (720) of FIG. 7a) in augmented reality based on the operating intensity or operating direction of the first home appliance.

[0086] According to one embodiment, the processor (120) may display a first object (620) in augmented reality based on information corresponding to the operating intensity of the first home appliance. The first object (620) may be set to have a variable volume by touch panel input. The size of the first object (620) may be proportional to the operating intensity of the first home appliance. Through the size of the first object (620) displayed in augmented reality, the user can intuitively understand the operating intensity of the first home appliance.

[0087] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to display the first object (620) in augmented reality based on information corresponding to the operating intensity of the first home appliance.

[0088] According to one embodiment, the processor (120) may display a second object (720) in augmented reality based on information corresponding to the direction of operation of the first home appliance. The second object (720) may be set to move its position in augmented reality by touch panel input. The position of the second object (720) displayed in augmented reality may be determined based on the direction of operation of the first home appliance. In augmented reality, the relative position of the second object (720) to the first home appliance may be determined based on information regarding the direction of operation of the first home appliance.

[0089] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to display a second object (720) in augmented reality based on information corresponding to the direction of operation of the first home appliance.

[0090] According to one embodiment, the processor (120) may be configured to display only one of the first object (620) or the second object (720) in augmented reality. For example, if the user selects an option to control the operating intensity of the first home appliance, the processor (120) may display the first object (620) in augmented reality. For example, if the user selects an option to control the operating direction of the first home appliance, the processor (120) may display the second object (720) in augmented reality.

[0091] According to one embodiment, the processor (120) may request one of the information corresponding to the operating intensity or the operating direction from the first home appliance according to the user's input using the communication module (190). When the processor (120) requests information corresponding to the operating intensity from the first home appliance, it may display the first object (620) in augmented reality, and when it requests information corresponding to the operating direction from the first home appliance, it may display the second object (720) in augmented reality.

[0092] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to request one of the information corresponding to the operating intensity or operating direction from the first home appliance according to the user's input using the communication module (190).

[0093] According to one embodiment, the method of operation of the electronic device (101) may include an operation (450) of enlarging or shrinking a first object (620) or moving a second object within augmented reality using a drag signal received from a touch panel.

[0094] According to one embodiment, the processor (120) can enlarge or reduce the size or volume of the first object (620) in response to receiving a drag signal from a touch panel. When a user touches and drags the first object (620) on a display (115) showing augmented reality, the processor (120) can perform an operation to enlarge or reduce the first object (620) in response.

[0095] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to perform an operation to enlarge or reduce the size or volume of the first object (620) in response to receiving a drag signal from a touch panel.

[0096] According to one embodiment, the processor (120) can move the position of the second object (720) in response to receiving a drag signal from a touch panel. When a user touches and drags the second object (720) on a display (115) showing augmented reality, the processor (120) can perform an operation to move the second object (720) in response.

[0097] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to move the position of the second object (720) in response to receiving a drag signal from the touch panel.

[0098] According to one embodiment, the method of operation of the electronic device (101) may include an operation (460) of adjusting the operating intensity or operating direction of the first home appliance based on the change in size of the first object (620) or the degree of movement of the second object (720).

[0099] According to one embodiment, the processor (120) may transmit a signal to the first home appliance through the communication module (190) to increase or decrease the operating intensity of the first home appliance by the ratio in which the first object (620) is enlarged or reduced. When the first object (620) is enlarged, the processor (120) may transmit a signal to the first home appliance to increase the operating intensity of the first home appliance. When the first object (620) is reduced, the processor (120) may transmit a signal to the first home appliance to decrease the operating intensity of the first home appliance.

[0100] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to transmit a signal to the first home appliance through the communication module (190) to increase or decrease the operating intensity of the first home appliance by the ratio in which the first object (620) is enlarged or reduced.

[0101] According to one embodiment, the processor (120) can transmit a signal to the first home appliance to control the direction of operation of the first home appliance in response to the movement of the second object (720) via the communication module (190). For example, if the second object (720) is moved to the left or right, the processor (120) can transmit a signal to the first home appliance to change the direction of operation of the first home appliance to the left or right in the horizontal direction. For example, if the second object (720) is moved upward or downward, the processor (120) can transmit a signal to the first home appliance to change the direction of operation of the first home appliance to the upward or downward in the horizontal direction.

[0102] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to transmit a signal to the first home appliance to control the direction of operation of the first home appliance in response to the movement of the second object (720) through the communication module (190).

[0103] FIG. 5 is a diagram illustrating the operation flow between an electronic device and a first home appliance according to one embodiment.

[0104] FIG. 5 illustrates a flowchart of the operation method between the electronic device of FIG. 1 to 3 (e.g., the electronic device of FIG. 1 to 3 (101)) and a selected first home appliance (200).

[0105] The flowchart illustrated in FIG. 5 is exemplary, and the method of operation of the electronic device (101) is not limited to the order of the flowchart illustrated.

[0106] The embodiment of FIG. 5 can be optionally combined with at least one of the embodiments of FIG. 1 to 4 and the embodiments of FIG. 6a to 11.

[0107] Referring to FIG. 5, an electronic device (101) according to one embodiment may include an operation (510) for displaying augmented reality for a first home appliance (200) and its surrounding environment. The operation 510 may correspond to the operation 420 of FIG. 4.

[0108] According to one embodiment, the electronic device (101) may perform an operation (520) of transmitting a signal to the first home appliance (200) requesting information corresponding to at least one of an operating intensity or an operating direction. The operation 520 may correspond to the operation 430 of FIG. 4. The electronic device (101) may communicate with the first home appliance (200) using a communication module (e.g., the communication module (190) of FIG. 1).

[0109] According to one embodiment, the first home appliance (200) can perform an operation (530) of transmitting a signal to the electronic device (101) that includes information corresponding to at least one of an operating intensity or an operating direction in response to a request signal from the electronic device (101).

[0110] According to one embodiment, the electronic device (101) may perform an operation (540) of displaying a first object (620) or a second object (720) in augmented reality based on the operating intensity or operating direction of the first home appliance (200). The operation 540 may correspond to the operation 440 of FIG. 4.

[0111] According to one embodiment, the electronic device (101) may perform an operation (550) of changing the size of the first object (620) or the position of the second object (720) based on a drag signal received from a touch panel. The operation 550 may correspond to the operation 450 of FIG. 4. The electronic device (101) may perform an operation of changing the size of the first object (620) or the position of the second object (720) based on user input.

[0112] According to one embodiment, the electronic device (101) may perform an operation (560) of transmitting a signal to the first home appliance (200) to change the operating intensity or the direction of operation based on a change in the size of the first object (620) or a change in the position of the second object (720). The operation 560 may correspond to the operation 460 of FIG. 4. The first home appliance (200) may adjust the operating intensity or the direction of operation in response to the signal received from the electronic device (101). For example, if the first home appliance (200) is an air conditioner, it may adjust the airflow or the direction of the cold air diffusing from the indoor unit in response to the signal from the electronic device (101).

[0113] FIGS. 6a, FIGS. 6b, and FIGS. 6c are drawings for explaining a method of controlling the operating intensity of a first home appliance using an electronic device according to one embodiment.

[0114] FIGS. 6a, 6b, and 6c illustrate a case where an air conditioner (or indoor unit) is selected as the first appliance among a plurality of appliances for convenience of explanation, but the present disclosure is not limited thereto.

[0115] The embodiments of FIGS. 6a, 6b, and 6c can be optionally combined with at least one of the embodiments of FIGS. 1 to 5 and the embodiments of FIGS. 7a to 11.

[0116] FIGS. 6a, 6b, and 6c illustrate the case where an air conditioner (or indoor unit) is selected as the first appliance, but this is for convenience of explanation and the following operation may also be applied when adjusting the operating intensity of other types of appliances.

[0117] FIGS. 6a, FIGS. 6b, and FIGS. 6c illustrate augmented reality (610) for a first home appliance (200) and its surrounding environment. In augmented reality (610), the user can select whether to control the operating intensity (e.g., operating intensity button (640)) and the operating direction (e.g., operating direction button (650)) of the first home appliance. For example, if the user touches or clicks the operating intensity button (640) displayed on the display, the electronic device (101) may, in response, display a first object (620) in augmented reality (610) for controlling the operating intensity of the first home appliance (200).

[0118] According to one embodiment, the electronic device (101) can display a first object (620) in augmented reality as shown in FIG. 6a. The first object (620) may be, for example, a three-dimensional shape. The first object (620) may be, for example, a cube or a rectangular shape. FIG. 6a illustrates a 6 by 6 cube as an example, but it may have a different volume based on the operating intensity of the first appliance (200) at the time of control. The user can intuitively understand the operating intensity of the first appliance (200) through the size of the first object (620) displayed in augmented reality as shown in FIG. 6a.

[0119] According to one embodiment, the first object (620) may be positioned to cover the first home appliance in augmented reality. The first object (620) may be displayed superimposed on the first home appliance (200). The first object (620) may be positioned to superimpose on the first home appliance. The first object (620) may be positioned on the first home appliance. Even if the first object (620) is positioned to cover the first home appliance, the first object (620) may have a transparent or translucent shape so that the first home appliance superimposed on the first object (620) may also be displayed in augmented reality. For example, only the corners or perimeters of the first object (620) may be displayed in color.

[0120] FIG. 6b illustrates the action of a user touching and dragging the first object (620) to enlarge the size of the first object (620). If the user wishes to increase the operating intensity (or wind intensity) of the first home appliance (200), the user may drag and enlarge the first object (620) as shown in FIG. 6b. The electronic device (101) may transmit a signal to the first home appliance (200) to increase the operating intensity by the increased volume ratio in response to the change in size of the first object (620). For example, if the first home appliance (200) is operating at level 6 intensity and a 6 by 6 cube is displayed as the first object (620), the user may enlarge the first object (620) to a size of 9 by 9 to increase the operating intensity of the first home appliance (200) to level 9.

[0121] FIG. 6c illustrates the action of a user touching and dragging the first object (620) to reduce the size of the first object (620). If the user wishes to reduce the operating intensity (or wind intensity) of the first home appliance (200), the user may drag and reduce the first object (620) as shown in FIG. 6b. The electronic device (101) may transmit a signal to the first home appliance (200) to reduce the operating intensity by the reduced volume ratio in response to the change in size of the first object (620). For example, if the first home appliance (200) is operating at level 6 intensity and a 6 by 6 cube is displayed as the first object (620), the user may reduce the first object (620) to a size of 2 by 2 to reduce the operating intensity of the first home appliance (200) to level 2.

[0122] FIGS. 7a and 7b are drawings for explaining a method of controlling the direction of operation of a first home appliance using an electronic device according to one embodiment. FIGS. 8a and 8b are drawings illustrating coordinates corresponding to the position of a second object shown in FIGS. 7a and 7b, respectively. FIGS. 9a and 9b are exemplary drawings for explaining a method of controlling the direction of operation of a first home appliance based on the position of a second object shown in FIGS. 7a and 7b, respectively.

[0123] FIGS. 7a, FIGS. 7b, FIGS. 8a, FIGS. 8b, FIGS. 9a, and FIGS. 9b illustrate a case where an air conditioner (or indoor unit) is selected as the first appliance among a plurality of appliances for convenience of explanation, but the present disclosure is not limited thereto.

[0124] The embodiments of FIGS. 7a, 7b, FIGS. 8a, 8b, FIGS. 9a and 9b can be optionally combined with at least one of the embodiments of FIGS. 1 to 6c and the embodiments of FIGS. 10 to 11.

[0125] FIGS. 7a, FIGS. 7b, FIGS. 8a, FIGS. 8b, FIGS. 9a, and FIGS. 9b illustrate the case where an air conditioner (or indoor unit) is selected as the first home appliance, but this is for convenience of explanation and the following operation may also be applied when controlling the direction of operation of other types of home appliances.

[0126] FIGS. 7a and 7b illustrate augmented reality (710) of a first home appliance (200) and its surrounding environment. In augmented reality (710), the user can select whether to control the operating intensity (e.g., operating intensity button (740)) and the operating direction (e.g., operating direction button (750)) of the first home appliance. For example, if the user touches or clicks the operating direction button (750) displayed on the display, the electronic device (101) may, in response, display a second object (720) in augmented reality (710) for controlling the operating intensity of the first home appliance (200).

[0127] According to one embodiment, the second object (720) may include an x-axis, a y-axis, and a z-axis. By displaying the x-axis, y-axis, and z-axis in augmented reality, the second object (720) can provide the user with a visual effect that the second object (720) appears to be located in three-dimensional space.

[0128] Referring to FIG. 7a, a second object (720) can be displayed in augmented reality by reflecting the current operating direction of the first appliance (200) of the electronic device (101).

[0129] FIG. 7b illustrates the action of a user touching and dragging the second object (720) to move the position of the second object (720). If the user wishes to change the direction of operation (or cold air blowing direction) of the first home appliance (200), the user can move the second object (720) by dragging it as shown in FIG. 7b. The electronic device (101) may transmit a signal to the first home appliance (200) to change the direction of operation by the moved direction and distance in response to the change in the position of the second object (720).

[0130] FIGS. 8A and 8B are drawings showing the location of a second object (720) in augmented reality as coordinates. An electronic device (101) according to one embodiment may divide each of the horizontal and vertical regions into multiple parts. For example, as illustrated, the electronic device (101) may divide the horizontal into 10 parts and the vertical into 10 parts. FIG. 8A is a drawing for showing the location of the second object (720) of FIG. 7A. FIG. 8B is a drawing for showing the location of the second object (720) of FIG. 7B. The coordinate notation shown in FIGS. 8A and 8B is exemplary for convenience of explanation and does not limit the scope of the present disclosure.

[0131] In FIGS. 8a and 8b, the horizontal axis may correspond to the horizontal movement direction of the first home appliance (200). The vertical axis may correspond to the vertical movement direction of the first home appliance (200). For example, when a user moves the second object (720) in a vertical direction, the movement direction of the first home appliance (200) may be moved in a vertical direction. For example, when a user moves the second object (720) in a vertical direction, the cold air blowing direction of the air conditioner, which is the first home appliance (200), may be moved vertically. The horizontal axis may correspond to the horizontal movement direction of the first home appliance (200). For example, when a user moves the second object (720) in a horizontal direction, the movement direction of the first home appliance (200) may be moved in a horizontal direction. For example, when the user moves the second object (720) in a horizontal direction, the direction of cold air blowing of the air conditioner, which is the first home appliance (200), can be moved horizontally (or left and right).

[0132] According to one embodiment, the size of the coordinates shown in FIGS. 8a and 8b may vary depending on the type of operation direction that the selected first home appliance (200) can control. For example, if the selected first home appliance (200), which is an air conditioner, can control the airflow direction by dividing it into 10 horizontal steps, the horizontal axis in augmented reality may be divided into 10 regions. For example, if the selected first home appliance (200), which is an air conditioner, can control the airflow direction by dividing it into 10 vertical steps, the vertical axis in augmented reality may be divided into 10 regions.

[0133] Referring to FIG. 8a, the electronic device (101) can display a second object (720) at coordinates (2,2) based on information regarding the current operating direction of the first appliance (200). Referring to FIG. 8b, the user can move the second object (720) to coordinates (6,8) by dragging it. For example, the second object (720) can be moved from step 2 to step 6 in the horizontal direction by the user's drag, and the second object (720) can be moved from step 2 to step 8 in the vertical direction.

[0134] FIGS. 9a and 9b are drawings for explaining the process of changing the direction of operation by dragging the second object (720) by the user. FIG. 9a illustrates a side view and a top view for explaining the direction of operation of the first home appliance (200) before the user moves the second object (720). FIG. 9b illustrates a side view and a top view for explaining the direction of operation of the first home appliance (200) after the user moves the second object (720). The side view indicates the vertical direction of operation of the first home appliance (200). The top view indicates the horizontal direction of operation of the first home appliance (200).

[0135] FIG. 9a may correspond to FIG. 8a. For example, the first appliance (200) may operate the airflow direction in two vertical steps and two horizontal steps before controlling the second object (720) as illustrated. These are exemplary figures for the example of explanation.

[0136] Figure 9b may correspond to Figure 8b. For example, in response to a user moving the second object (720) to coordinates (6, 8), the electronic device (101) may transmit a signal to the first home appliance (200) to change the airflow direction of the first home appliance (200) to 6 steps in the vertical direction and 8 steps in the horizontal direction.

[0137] In this way, the user can intuitively change the direction of operation of the first home appliance (200) by changing the position of the second object (720) in augmented reality.

[0138] FIG. 10 is a flowchart illustrating a method of operation of an electronic device according to one embodiment.

[0139] FIG. 4 illustrates a flowchart of the operation method of the electronic device of FIG. 1 to 3 (e.g., the electronic device (101) of FIG. 1 to 3).

[0140] The flowchart illustrated in FIG. 4 is exemplary, and the method of operation of the electronic device (101) is not limited to the order of the flowchart illustrated.

[0141] The embodiment of FIG. 10 can be optionally combined with at least one of the embodiments of FIG. 1 to FIG. 9b and the embodiment of FIG. 11.

[0142] According to one embodiment, the electronic device (101) may include a processor (e.g., the processor (120) of FIG. 1) and a memory (e.g., the memory (130) of FIG. 1). The processor (120) may include a processing circuit. The memory (130) may include at least one storage medium for storing instructions.

[0143] Referring to FIG. 10, according to one embodiment, the method of operation of an electronic device (101) may include an operation (1010) of selecting a first appliance among a plurality of appliances capable of communicating with the electronic device (101).

[0144] Here, the first household appliance may include an air conditioner, an air purifier, a lighting device, a humidifier, or a dehumidifier.

[0145] According to one embodiment, the air conditioner may include an indoor unit including a blower, an outdoor unit, and a refrigerant circulation device configured to circulate refrigerant between the indoor unit and the outdoor unit.

[0146] According to one embodiment, the air purifier may include a convection fan that forms air convection.

[0147] According to one embodiment, the lighting device may include a light source.

[0148] According to one embodiment, the humidifier may include a water tank and a humidifying blower fan arranged to supply moisture to the air using the water contained in the water tank.

[0149] According to one embodiment, the dehumidifier may include a heat pump arranged to dehumidify moisture in the air and a dehumidifying blower fan arranged to diffuse the dehumidified air into the air.

[0150] According to one embodiment, the processor (120) can recognize or identify at least one home appliance placed in a house using a camera module (e.g., the camera module (180) of FIG. 1). One of the at least one home appliance may be selected as the first home appliance.

[0151] According to one embodiment, the processor (120) can select one of at least one home appliance as the first home appliance using a signal received from a touch panel. The first home appliance can be selected by a user touching one of the home appliances on an image captured by a camera module (180) displayed on a display (115). The processor (120) can select one of at least one home appliance as the first home appliance based on user input.

[0152] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to recognize or identify at least one home appliance placed in the house using the camera module (180).

[0153] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to select one of at least one home appliance as the first home appliance using a signal received from a touch panel. The memory (130) may include a command that causes one of at least one home appliance to select the first home appliance based on user input.

[0154] According to one embodiment, the method of operation of the electronic device (101) may include an operation (1020) of displaying an augmented reality displaying a selected first home appliance and the surrounding environment of the first home appliance on a display (e.g., a display (115) of FIG. 2).

[0155] According to one embodiment, the processor (120) may perform the operation of displaying augmented reality on the display (115) that displays the first home appliance and the surrounding environment of the first home appliance in response to the selection of the first home appliance. Here, the augmented reality may be stored in advance in memory (130). The augmented reality may reflect the indoor structure of the house and the arrangement of surrounding furniture. The augmented reality may be determined by sensors included in the electronic device (101) or the home appliance and stored in memory (130). The augmented reality may be set to be modified directly by the user.

[0156] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to perform an operation of displaying the first home appliance and the surrounding environment of the first home appliance on the display (115) in response to the selection of the first home appliance.

[0157] According to one embodiment, the method of operation of the electronic device (101) may include an operation (1030) of displaying a first object (e.g., the first object (620) of FIG. 6a) or a second object (e.g., the second object (720) of FIG. 7a) having a specific size within augmented reality.

[0158] According to one embodiment, the processor (120) can display a first object (620) of a specific size or volume within augmented reality. The volume of the first object (620) may be set to vary by touch panel input. Regardless of the operating intensity of the first home appliance at the time of control, the processor (120) can display a first object (620) of a fixed size within augmented reality. The fixed size of the first object (620) may be stored in advance in memory (130).

[0159] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to display a first object (620) of a specific size or volume in augmented reality.

[0160] According to one embodiment, the processor (120) can display the second object (720) at a specific location within augmented reality. The second object (720) can be configured to move its position within augmented reality by touch panel input. Regardless of the operating direction of the first home appliance at the time of control, the processor (120) can display the second object (720) at a predetermined location within augmented reality. The predetermined location of the second object (720) can be stored in memory (130) in advance.

[0161] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to display the second object (720) at a specific location within augmented reality.

[0162] According to one embodiment, the processor (120) may be configured to display only one of the first object (620) or the second object (720) in augmented reality. For example, if the user selects an option to control the operating intensity of the first home appliance, the processor (120) may display the first object (620) in augmented reality. For example, if the user selects an option to control the operating direction of the first home appliance, the processor (120) may display the second object (720) in augmented reality.

[0163] According to one embodiment, the processor (120) may request one of the information corresponding to the operating intensity or the operating direction from the first home appliance according to the user's input using the communication module (190). When the processor (120) requests information corresponding to the operating intensity from the first home appliance, it may display the first object (620) in augmented reality, and when it requests information corresponding to the operating direction from the first home appliance, it may display the second object (720) in augmented reality.

[0164] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to request one of the information corresponding to the operating intensity or operating direction from the first home appliance according to the user's input using the communication module (190).

[0165] Unlike the method of operation of the electronic device (101) of FIG. 4, the method of operation of the electronic device (101) of FIG. 10 can display the first object (620) or the second object (720) at a predetermined size or location without requesting information corresponding to the operating intensity or operating direction of the first home appliance after the first home appliance is selected.

[0166] According to one embodiment, the method of operation of the electronic device (101) may include an operation (1040) of enlarging or shrinking a first object (620) or moving a second object within augmented reality using a drag signal received from a touch panel.

[0167] According to one embodiment, the processor (120) can enlarge or reduce the size or volume of the first object (620) in response to receiving a drag signal from a touch panel. When a user touches and drags the first object (620) on a display (115) showing augmented reality, the processor (120) can perform an operation to enlarge or reduce the first object (620) in response.

[0168] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to perform an operation to enlarge or reduce the size or volume of the first object (620) in response to receiving a drag signal from a touch panel.

[0169] According to one embodiment, the processor (120) can move the position of the second object (720) in response to receiving a drag signal from a touch panel. When a user touches and drags the second object (720) on a display (115) showing augmented reality, the processor (120) can perform an operation to move the second object (720) in response.

[0170] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to move the position of the second object (720) in response to receiving a drag signal from the touch panel.

[0171] According to one embodiment, the method of operation of the electronic device (101) may include an operation (1050) of adjusting the operating intensity or operating direction of the first home appliance based on the change in size of the first object (620) or the degree of movement of the second object (720).

[0172] According to one embodiment, the processor (120) may transmit a signal to the first home appliance through the communication module (190) to increase or decrease the operating intensity of the first home appliance by the ratio in which the first object (620) is enlarged or reduced. When the first object (620) is enlarged, the processor (120) may transmit a signal to the first home appliance to increase the operating intensity of the first home appliance. When the first object (620) is reduced, the processor (120) may transmit a signal to the first home appliance to decrease the operating intensity of the first home appliance.

[0173] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to transmit a signal to the first home appliance through the communication module (190) to increase or decrease the operating intensity of the first home appliance by the ratio in which the first object (620) is enlarged or reduced.

[0174] According to one embodiment, the processor (120) can transmit a signal to the first home appliance to control the direction of operation of the first home appliance in response to the movement of the second object (720) via the communication module (190). For example, if the second object (720) is moved to the left or right, the processor (120) can transmit a signal to the first home appliance to change the direction of operation of the first home appliance to the left or right in the horizontal direction. For example, if the second object (720) is moved upward or downward, the processor (120) can transmit a signal to the first home appliance to change the direction of operation of the first home appliance to the upward or downward in the horizontal direction.

[0175] According to one embodiment, the memory (130) may include a command that causes the electronic device (101) to transmit a signal to the first home appliance to control the direction of operation of the first home appliance in response to the movement of the second object (720) through the communication module (190).

[0176] FIG. 11 is a diagram illustrating the operation flow between an electronic device and a first home appliance according to one embodiment.

[0177] FIG. 11 illustrates a flowchart of the operation method between the electronic device of FIG. 1 to 3 (e.g., the electronic device of FIG. 1 to 3 (101)) and a selected first home appliance (200).

[0178] The flowchart illustrated in FIG. 11 is exemplary, and the method of operation of the electronic device (101) is not limited to the order of the flowchart illustrated.

[0179] The embodiment of FIG. 11 can be optionally combined with at least one of the embodiments of FIG. 1 to FIG. 10.

[0180] Referring to FIG. 11, an electronic device (101) according to one embodiment may include an operation (1110) for displaying augmented reality for a first home appliance (200) and its surrounding environment. The operation 1110 may correspond to the operation 1020 of FIG. 10.

[0181] According to one embodiment, the electronic device (101) may perform an operation (1120) of displaying a first object (620) having a specific size or a second object (720) formed at a specific location within augmented reality. The operation 1120 may correspond to the operation 1030 of FIG. 10.

[0182] According to one embodiment, the electronic device (101) may perform an operation (1130) of changing the size of a first object (620) or the position of a second object (720) based on a drag signal received from a touch panel. The operation 1130 may correspond to the operation 1040 of FIG. 10.

[0183] According to one embodiment, the electronic device (101) may perform an operation (1140) of transmitting a signal to the first home appliance (200) to change the operating intensity or operating direction based on a change in the size of the first object (620) or a change in the position of the second object (720). The operation 1140 may correspond to the operation 1050 of FIG. 10. The first home appliance (200) may adjust the operating intensity or operating direction in response to the signal received from the electronic device (101). For example, if the first home appliance (200) is an air conditioner, it may adjust the airflow or direction of the cold air diffusing from the indoor unit in response to the signal from the electronic device (101). The electronic device (101) may communicate with the first home appliance (200) using a communication module (e.g., the communication module (190) of FIG. 1).

[0184] According to one embodiment, the electronic device (101) may include a display (115), a communication module (190), at least one processor (120) including a processing circuit, and a memory (130) including at least one storage medium for storing instructions. When the above commands are executed individually or collectively by the at least one processor (120), the electronic device may cause the first appliance (200) among at least one appliance to be selected and, based on the selection of the first appliance (200), to display an augmented reality on the display (115) that displays the first appliance (200) and the surrounding environment of the first appliance (200), display a first object (620) corresponding to the operating intensity of the first appliance in the augmented reality based on information corresponding to the operating intensity of the first appliance (200), and transmit a signal to the first appliance (200) to increase or decrease the operating intensity of the first appliance (200) based on user input that increases or decreases the size or volume of the first object (620).

[0185] According to one embodiment, the electronic device (101) may further include a camera module (180). When the instructions are executed individually or collectively by the at least one processor (120), the electronic device may cause at least one home appliance to identify in an image captured by the camera module (180) and, based on user input, identify one of the at least one home appliance as the first home appliance (200).

[0186] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device may cause a second object (720) corresponding to at least one of the operating direction or operating distance of the first appliance (200) to be displayed in the augmented reality based on information corresponding to at least one of the operating direction or operating distance of the first appliance.

[0187] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), they may cause the electronic device to transmit a signal to the first appliance (200) to adjust at least one of the direction of operation or distance of operation of the first appliance (200) based on user input that moves the second object (720).

[0188] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor (120), the electronic device may cause the display (115) to display a side view and a top view corresponding to the location of the second object (720).

[0189] According to one embodiment, the first appliance (200) is an indoor unit including a blower, and the information corresponding to the operating intensity may be information corresponding to at least one of the rotational speed of the blower and the refrigerant temperature.

[0190] According to one embodiment, the first appliance (200) includes a fan that forms air convection, and information corresponding to the operating intensity may be information corresponding to the rotational speed of the fan.

[0191] According to one embodiment, the first household appliance is a lighting device including a light source, and information corresponding to the operating intensity may be information corresponding to the output of the light source.

[0192] According to one embodiment, the first object (620) may be displayed superimposed on the first appliance (200).

[0193] A control method for an electronic device (101) according to one embodiment may include the steps of: displaying augmented reality based on the selection of a first home appliance (200); displaying a first object (620) in the augmented reality based on the operating intensity of the first home appliance (200); increasing or decreasing the size or volume of the first object (620) based on user input; and transmitting a signal to the first home appliance (200) to increase or decrease the operating intensity of the first home appliance (200) based on the increase or decrease in the size or volume of the first object (620).

[0194] According to one embodiment, the control method of the electronic device (101) may further include the step of identifying at least one home appliance in an image captured by the camera module (180) of the electronic device.

[0195] According to one embodiment, the control method of the electronic device (101) may further include the step of requesting information corresponding to the operating intensity from the first appliance (200) to determine the size of the first object (620).

[0196] According to one embodiment, the control method of the electronic device (101) may further include the step of displaying a first object (620) in the augmented reality based on the operating intensity of the first home appliance (200), and the step of determining the size or volume of the first object (620) based on the operating intensity of the first home appliance (200).

[0197] According to one embodiment, the control method of the electronic device (101) may further include the step of displaying a second object (720) in the augmented reality based on information corresponding to at least one of the operating direction or operating distance of the first home appliance (200).

[0198] According to one embodiment, the control method of the electronic device (101) may further include the step of transmitting a signal to the first home appliance (200) to adjust at least one of the operating direction or operating distance of the first home appliance (200) based on user input that moves the second object (720).

[0199] According to one embodiment, the control method of the electronic device (101) may further include the step of displaying a side view and a top view corresponding to the position of the second object (720) on the display (115).

[0200] According to one embodiment, the side view may indicate the vertical movement direction of the first appliance (200), and the top view may indicate the horizontal movement direction of the first appliance (200).

[0201] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit this disclosure. For example, a component expressed in the singular should be understood as a concept including a plural component unless the context clearly implies only the singular. Each of the phrases used in this disclosure, such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C,” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. It should be understood that the term “and / or” used in this disclosure encompasses any possible combination of one or more of the listed items. Terms such as “comprising,” “having,” and “consisting of” used in this disclosure are intended merely to specify the existence of the features, components, parts, or combinations thereof described in this disclosure, and the use of such terms is not intended to exclude the existence or addition of one or more other features, components, parts, or combinations thereof. Expressions such as “first,” “second,” used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0202] The expression “configured to” as used in this disclosure may be appropriately substituted depending on the context, for example, with “suitable for,” “capable of,” “designed to,” “modified to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only that which is “specially designed” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” together with other devices or components. For example, the phrase “device configured (or set) to perform A, B, and C” may mean a device dedicated to performing the said operation, or a general-purpose device capable of performing various operations including said operation.

[0203] Meanwhile, terms such as "upper side," "lower side," and "front-rear direction" used in this disclosure are defined based on the drawings, and the shape and position of each component are not limited by these terms.

[0204] Although the foregoing description in this disclosure has focused on specific embodiments, this disclosure is not limited to such specific embodiments and should be understood to encompass all various modifications, equivalents, and / or substitutions of various embodiments.

Claims

1. In an electronic device, Display (115); communication module (190); At least one processor (120) including a processing circuit; and The electronic device comprises a memory (130) including at least one storage medium for storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor (120), cause: Based on the selection of one first home appliance (200) among at least one home appliance, an augmented reality displaying the first home appliance (200) and the surrounding environment of the first home appliance (200) is displayed on the display (115), and Based on information corresponding to the operating intensity of the first home appliance (200), a first object (620) corresponding to the operating intensity of the first home appliance is displayed in the augmented reality, and Causing to transmit a signal to the first home appliance (200) to increase or decrease the operating intensity of the first home appliance (200) based on user input that increases or decreases the size or volume of the first object (620). Electronic device.

2. In Paragraph 1, The above electronic device further includes a camera module (180), When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device: Identify at least one home appliance in an image captured by the above camera module (180), and Causing to identify one of the at least one home appliance as the first home appliance (200) based on user input, Electronic device.

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device: Based on information corresponding to at least one of the operation direction or operation distance of the first home appliance (200), causing a second object (720) corresponding to at least one of the operation direction or operation distance of the first home appliance to be displayed in the augmented reality. Electronic device.

4. In Paragraph 3, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device: Causing to transmit a signal to the first home appliance (200) to adjust at least one of the operation direction or operation distance of the first home appliance (200) based on user input moving the second object (720), Electronic device.

5. In Paragraph 3 or 4, When the above instructions are executed individually or collectively by the at least one processor (120), the electronic device: Causing at least one of the side view and top view corresponding to the position of the second object (720) to be displayed on the display (115), Electronic device.

6. In any one of paragraphs 1 through 5, The first home appliance (200) above is an indoor unit including a blower, and An electronic device in which information corresponding to the above operating intensity is information corresponding to at least one of the rotational speed of the blower and the refrigerant temperature.

7. In any one of paragraphs 1 through 5, The first appliance (200) above includes a fan that forms air convection, and The information corresponding to the above operating intensity is information corresponding to the rotational speed of the fan, Electronic device.

8. In any one of paragraphs 1 through 5, The above-mentioned first household appliance is a lighting device including a light source, and An electronic device in which information corresponding to the above operating intensity is information corresponding to the output of the light source unit.

9. In any one of paragraphs 1 through 8, The above first object (620) is, An electronic device that is superimposed and displayed on the first home appliance (200) above.

10. As a method for controlling an electronic device, A step of displaying augmented reality based on the selection of the first home appliance (200); A step of displaying a first object (620) in the augmented reality based on the operating intensity of the first home appliance (200); A step of expanding or reducing the size or volume of the first object (620) based on user input; The method comprises the step of transmitting a signal to the first home appliance (200) to increase or decrease the operating intensity of the first home appliance (200) based on the expansion or reduction of the size or volume of the first object (620). method.

11. In Paragraph 10, The method further comprises the step of identifying at least one home appliance in an image captured by the camera module (180) of the electronic device. method.

12. In Paragraph 10 or 11, The method further includes the step of requesting information corresponding to the operating intensity from the first home appliance (200) to determine the size of the first object (620). method.

13. In any one of paragraphs 10 through 12, The step of displaying the first object (620) in the augmented reality based on the operating intensity of the first home appliance (200) further includes the step of determining the size or volume of the first object (620) based on the operating intensity of the first home appliance (200). method.

14. In any one of paragraphs 10 through 13, The method further includes the step of displaying a second object (720) in the augmented reality based on information corresponding to at least one of the operating direction or operating distance of the first appliance (200). method.

15. In any one of paragraphs 10 through 14, The method further includes the step of transmitting a signal to the first home appliance (200) to adjust at least one of the operation direction or operation distance of the first home appliance (200) based on user input that moves the second object (720). method.

Images