Electronic device and method for displaying and controlling home appliance

Abstract

An electronic device is provided. The electronic device may include: a display including a display area; and at least one processor electrically connected with the display. The at least one processor may: display a 3D map including at least one of indoor space state information including an environmental state of an indoor space and home appliance state information including an operating state of at least one home appliance disposed in the indoor space; and controlling an operation of the at least one home appliance based on a user input to a predetermined area on the 3D map.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for displaying and controlling household appliances disposed in an indoor space. Background Art

[0002] The development of wireless communication technology has promoted the development of smart device control technology, which is used to control indoor devices (such as televisions (TVs), air conditioners, refrigerators, air purifiers, smart plugs, or lighting devices) on a user's portable terminal.

[0003] In this regard, cloud services for linking built-in Internet of Things (IoT) devices, home services, and IoT networks disposed in an indoor space to a smart device via a user's portable terminal, and user interfaces related to the cloud services have been developed.

[0004] A user can easily control various smart devices disposed in an indoor space by controlling the smart device through the user interface of the portable terminal.

[0005] The above information is presented as background information only to assist in understanding the present disclosure. It is not determined nor asserted that any of the above is available as prior art with respect to the present disclosure. Summary of the Invention

[0006] Solution to the Problem

[0007] Aspects of the present disclosure are to at least address the above problems and / or disadvantages and to at least provide the advantages described below. Accordingly, an aspect of the present disclosure is to provide an electronic device and method for displaying and controlling household appliances disposed in an indoor space based on input data, the input data including three-dimensional (3D) data of the household appliances and the indoor space.

[0008] Additional aspects will be set forth in part in the following description, and in part will become apparent from the following description, or may be learned by practice of the presented embodiments.

[0009] According to an aspect of the present disclosure, there is provided an electronic device. The electronic device includes: a display including a display area; and at least one processor electrically connected to the display. The at least one processor displays a 3D map including at least one of indoor space state information or household appliance state information, the indoor space state information including an environmental state of the indoor space, and the household appliance state information including an operating state of at least one household appliance disposed in the indoor space. The at least one processor controls an operation of at least one household appliance based on a user input to a predetermined area on the 3D map.

[0010] In an embodiment of the present disclosure, the at least one processor displays visual information about at least one household appliance in the form of a 3D animation on a 3D map based on the household appliance status information.

[0011] In an embodiment of the present disclosure, the visual information about at least one household appliance includes at least one of the following: the on / off state of the light of at least one household appliance, the output air direction, the output air speed, the output temperature, the door open / close state, or the moving state.

[0012] In an embodiment of the present disclosure, the at least one processor displays at least one subtask for performing a predetermined operation of at least one household appliance on the 3D map.

[0013] In an embodiment of the present disclosure, at least one household appliance includes a first object and a second object that perform at least one subtask for performing a predetermined operation. The predetermined operation includes: performing a first subtask by the first object; and performing a second subtask associated with the first subtask by the second object.

[0014] In an embodiment of the present disclosure, the at least one processor sequentially displays on the 3D map an image showing the operation sequence of the first object and the second object, an enlarged image of the first object, a pop-up window including text describing the first subtask, an enlarged image of the second object, and a pop-up window including text describing the second subtask.

[0015] In an embodiment of the present disclosure, the predetermined operation includes: moving at least one household appliance in an indoor space; detecting a target object included in the indoor space by at least one household appliance; capturing at least one image of the target object by at least one household appliance; and displaying the target object image captured by at least one household appliance.

[0016] In an embodiment of the present disclosure, the at least one processor sequentially displays an image of at least one household appliance moving in the indoor space, an image of at least one household appliance detecting a target object included in the indoor space, an image of at least one household appliance capturing the target object, and an image of the target object captured by at least one household appliance.

[0017] In an embodiment of the present disclosure, the at least one processor controls at least one household appliance performing at least one subtask based on a predetermined operation on the 3D map.

[0018] In an embodiment of the present disclosure, the predetermined operation includes: determining the driving environment of at least one household appliance; and setting at least one of the power setting, position, or operation mode of at least one household appliance to a pre-stored operation configuration value.

[0019] In an embodiment of the present disclosure, at least one processor receives user input regarding the operating environment of at least one household appliance. The at least one processor controls at least one of the power settings, position, or operating mode of the at least one household appliance based on an operation configuration value corresponding to the operating environment.

[0020] In an embodiment of the present disclosure, the at least one processor displays indoor space status information and household appliance status information on a 3D map using at least one of text, numerical values, colors, icons, graphics, or contour lines.

[0021] In an embodiment of the present disclosure, the indoor space status information includes at least one of the air quality, temperature, humidity, lighting, or noise of the indoor space.

[0022] In an embodiment of the present disclosure, the at least one processor generates a history of the indoor space status based on changes in the indoor space status information over a reference or predetermined period of time. The at least one processor displays the history of the indoor space status for a specified period of time (e.g., hourly).

[0023] In an embodiment of the present disclosure, the household appliance status information includes at least one of the operating mode, remaining time, energy usage, temperature, remaining battery level, or position of the at least one household appliance.

[0024] In an embodiment of the present disclosure, the at least one processor detects an operation error in the at least one household appliance based on the household appliance status information. The at least one processor displays the operation error and action guidance.

[0025] According to another aspect of the present disclosure, a method for displaying and controlling household appliances is provided. The method includes: displaying a 3D map that includes indoor space status information or household appliance status information, where the indoor space status information includes the environmental status of the indoor space and the household appliance status information includes the operating status of at least one household appliance disposed in the indoor space; and controlling the operation of the at least one household appliance based on user input to a predetermined area on the 3D map.

[0026] In an embodiment of the present disclosure, displaying the 3D map includes: displaying visual information regarding the at least one household appliance in the form of a 3D animation on the 3D map based on the household appliance status information.

[0027] In an embodiment of the present disclosure, displaying the 3D map includes: displaying at least one subtask for performing a predetermined operation of the at least one household appliance on the 3D map.

[0028] In an embodiment of the present disclosure, controlling the at least one household appliance includes: controlling the at least one household appliance that performs at least one subtask based on a predetermined operation on the 3D map.

[0029] Advantages of the Invention

[0030] According to various embodiments of the present disclosure, the electronic device and method of the present disclosure display a 3D map including an indoor space and household appliances based on input data. In addition, the electronic device and method of the present disclosure display visual information about household appliances in the form of 3D animations on the 3D map.

[0031] Therefore, the electronic device and method of the present disclosure visually display the positions and operating states of household appliances in the indoor space on the 3D map.

[0032] In addition, according to various embodiments of the present disclosure, the electronic device and method of the present disclosure display subtasks of household appliances based on a predetermined operation sequence. In addition, the electronic device and method of the present disclosure control the household appliances that execute the subtasks based on a predetermined operation.

[0033] Therefore, the electronic device and method of the present disclosure enhance the usability and convenience of household appliances by sequentially displaying and controlling the associated operations for executing a predetermined operation of the household appliances.

[0034] Through the following detailed description of various embodiments of the present disclosure disclosed in conjunction with the accompanying drawings, other aspects, advantages, and significant features of the present disclosure will become apparent to those skilled in the art. Brief Description of the Drawings

[0035] Figure 1 Shows household appliances according to an embodiment of the present disclosure;

[0036] Figure 2 Is a block diagram showing an electronic device according to an embodiment of the present disclosure;

[0037] Figure 3 Shows a 3D map according to an embodiment of the present disclosure;

[0038] Figure 4 Shows a 3D map including indoor space state information according to an embodiment of the present disclosure;

[0039] Figure 5 Shows a 3D map including household appliance state information according to an embodiment of the present disclosure;

[0040] Figure 6 Shows the display of indoor space state information and household appliance state information on a 3D map according to an embodiment of the present disclosure;

[0041] Figure 7a , Figure 7b and Figure 7c Show the control of the operation of household appliances on a 3D map according to various embodiments of the present disclosure;

[0042] Figure 8 shows visual information regarding a household appliance according to an embodiment of the present disclosure;

[0043] Figure 9a 、 Figure 9b and Figure 9c show operations in which an electronic device sequentially displays subtasks for performing a predetermined operation according to various embodiments of the present disclosure;

[0044] Figure 10a 、 Figure 10b and Figure 10c show operations in which an electronic device sequentially displays subtasks for performing a predetermined operation according to various embodiments of the present disclosure;

[0045] Figure 11a and Figure 11b show operations in which an electronic device controls a household appliance based on a predetermined operation according to various embodiments of the present disclosure;

[0046] Figure 12 show operations in which an electronic device displays a history of an indoor space state according to an embodiment of the present disclosure;

[0047] Figure 13 show operations in which an electronic device displays household appliance state information according to an embodiment of the present disclosure;

[0048] Figure 14 show operations in which an electronic device detects an operation error in a household appliance according to an embodiment of the present disclosure; and

[0049] Figure 15 show a method of displaying and controlling a household appliance according to an embodiment of the present disclosure.

[0050] Note that, throughout the drawings, like reference numerals are used to describe the same or similar elements, features, and structures. Detailed Description

[0051] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of various embodiments of the present disclosure defined by the claims and their equivalents. The following description includes various specific details to assist in the understanding, but these specific details are to be regarded as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of known functions and structures may be omitted for clarity and conciseness.

[0052] The terms and words used in the following description and claims are not limited to their literal meanings, but are used solely by the inventors to achieve a clear and consistent understanding of the present disclosure. Thus, it should be clear and apparent to those skilled in the art that the following description of the various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0053] It should be understood that, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" include plural referents. Thus, for example, a reference to "a component surface" includes a reference to one or more of such surfaces.

[0054] As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include all possible combinations of the items listed together in the corresponding phrase.

[0055] As used herein, terms such as "first" and "second" or "1st" and "2nd" may be used simply to distinguish the corresponding components from other components and do not otherwise limit the components (e.g., in terms of importance or order).

[0056] It should be understood that if an element (e.g., a first element) is referred to as being "coupled", "coupled to", "connected" or "connected to" another element (e.g., a second element) using or without using the terms "operatively" or "communicatively", it means that the element can be coupled to the other element directly (e.g., wired), wirelessly, or via a third element.

[0057] It will also be understood that the terms "comprises" and / or "having" as used herein denote the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0058] It should be understood that when a component is referred to as being "connected to", "coupled to", "supported on" or "in contact with" another component, the components can be connected, coupled, supported or in contact with each other directly or via a third component.

[0059] Throughout the specification, when a component is located "on" another component, the first component can be directly on the second component, or other components can be located between the first component and the second component.

[0060] The term "and / or" can mean a combination of the listed multiple related components or any one of the components.

[0061] In the following, the working principle and embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0062] Figure 1 A household appliance according to an embodiment of the present disclosure is shown.

[0063] Referring to Figure 1 , the household appliance 10 may include a communication module capable of communicating with at least one household appliance, user device 2, or server 3, a user interface for receiving user input and / or outputting information to the user, at least one processor for controlling the operation (or task) of the household appliance 10, and at least one memory for storing a program for controlling the operation of the household appliance 10.

[0064] The household appliance 10 may be at least one of various types of household appliances. For example, as shown in the figure, the household appliance 10 may include at least one of a refrigerator 11, a dishwasher 12, an electric stove 13, an electric oven 14, an air conditioner 15, a steam wardrobe 16, a washing machine 17, a dryer 18, and a microwave oven 19, but is not limited thereto. The household appliance 10 may also include various types of household appliances not shown in the figure, such as a robotic vacuum cleaner, a vacuum cleaner, and a television. In addition to the above household appliances, according to an embodiment, the household appliance 10 may also include a device capable of connecting to other household appliances, user device 2, or server 3 and performing the operations described below.

[0065] The server 3 may include a communication module capable of communicating with another server, household appliance 10, or user device 2, at least one processor capable of processing data received from another server, household appliance 10, or user device 2, and at least one memory capable of storing a program for processing data or processed data. The server 3 may be implemented as various computing devices, such as a workstation, cloud, data drive, and data station. The server 3 may be implemented as one or more servers physically or logically partitioned based on functions, detailed configurations of functions, data, etc., and may send and receive data between servers and process the sent and received data.

[0066] The server 3 can perform functions such as managing user accounts, registering home appliances 10 to user accounts, and managing or controlling the registered home appliances 10. For example, a user can access the server 3 through the user device 2 to create a user account. The user account can be identified by an ID and password set by the user. The server 3 can register the home appliance 10 to the user account according to a predetermined process. For example, the server 3 can register, manage, and control the home appliance 10 by verifying / authenticating the identification information (e.g., serial number or MAC address) of the home appliance 10 to the user account. The user device 2 can include a communication module capable of communicating with the home appliance 10 or the server 3, a user interface for receiving user input or outputting information to the user, at least one processor for controlling the operation of the user device 2, and at least one memory for storing a program for controlling the operation of the user device 2.

[0067] The user device 2 can be carried by the user or can be set in the user's home or office. The user device 2 can include, but is not limited to, a personal computer, a terminal, a mobile phone, a smart phone, a handheld device, or a wearable device.

[0068] A program (i.e., an application) for controlling the home appliance 10 can be stored in the memory of the user device 2. The application can be pre-installed on the user device 2 or can be downloaded from an external server.

[0069] The user can access the server 3 through the application installed on the user device 2 to create a user account, and can communicate with the server 3 based on the logged-in user account to register the home appliance 10.

[0070] For example, when the user programs / operates the home appliance 10 according to the instructions provided by the application installed on the user device 2 to connect to the server 3, the identification information (e.g., serial number or MAC address) of the home appliance 10 associated with the user account can be stored in the server 3, thereby allowing the home appliance 10 to be registered to the user account.

[0071] The user can use the application installed on the user device 2 to control the home appliance 10. For example, when the user uses the application installed on the user device 2 to log in to the user account, the home appliances 10 registered in the user account are displayed, and when a control command for the home appliance 10 is input, the control command can be transmitted to the home appliance 10 through the server 3.

[0072] The network can include both a wired network and a wireless network. The wired network can include a cable network, a telephone network, etc., and the wireless network can include all networks that send and receive signals through radio waves. The wired network and the wireless network can be connected to each other.

[0073] The network may include a wide area network (WAN) (such as the Internet), a local area network (LAN) formed around an access point (AP), and a short-range wireless network that does not pass through the AP. The short-range wireless network may include, but is not limited to, Bluetooth TM (IEEE802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Direct, Near Field Communication (NFC), and Z-wave.

[0074] The AP may connect the home appliance 10 or the user device 2 to the WAN to which the server 3 is connected. The home appliance 10 or the user device 2 may be connected to the server 3 via the WAN.

[0075] The AP may use wireless communication (such as Wi-Fi (Wi-Fi TM , IEEE 802.11), Bluetooth (Bluetooth TM , IEEE802.15.1), Zigbee (IEEE 802.15.4), etc.) to communicate with the home appliance 10 or the user device 2, and may use wired communication to access the WAN, but is not limited thereto.

[0076] According to various embodiments of the present disclosure, the home appliance 10 may be directly connected to the user device 2 or the server 3 without passing through the AP.

[0077] The home appliance 10 may be connected to the user device 2 or the server 3 via a long-range wireless network or a short-range wireless network.

[0078] For example, the home appliance 10 may be connected to the user device 2 via a short-range wireless network (e.g., Wi-Fi Direct).

[0079] As another example, the home appliance 10 may use a long-range wireless network (e.g., a cellular communication module) to be connected to the user device 2 or the server 3 via the WAN.

[0080] As another example, the home appliance 10 may use wired communication to access the WAN, and may be connected to the user device 2 or the server 3 via the WAN.

[0081] When the home appliance 10 is capable of using wired communication to access the WAN, it may operate as an access repeater. Therefore, the home appliance 10 may connect another home appliance to the WAN to which the server 3 is connected. In addition, another home appliance may connect the home appliance 10 to the WAN to which the server 3 is connected.

[0082] The home appliance 10 can send information about its operation or status to other home appliances, the user device 2, or the server 3 via a network. For example, when receiving a request from the server 3, if a specific event occurs in the home appliance 10, periodically or in real time, the home appliance 10 can send information about its operation or status to other home appliances, the user device 2, or the server 3. When receiving information about the operation or status of the home appliance 10, the server 3 can update the stored information about the operation or status of the home appliance 10 and send the updated information about the operation and status of the home appliance 10 to the user device 2 via the network. Here, the updated information can include various operations for changing the existing information, such as adding new information to the existing information and replacing the existing information with new information.

[0083] The home appliance 10 can obtain various information from other home appliances, the user device 2, or the server 3 and provide the obtained information to the user. For example, the home appliance 10 can obtain information related to the functions of the home appliance 10 (e.g., recipes, washing methods, etc.) and various environmental information (e.g., weather, temperature, humidity, etc.) from the server 3 and output the obtained information through the user interface.

[0084] The home appliance 10 can operate according to a control command received from other home appliances, the user device 2, or the server 3. For example, when the home appliance 10 obtains prior approval from the user to operate according to the control command of the server 3, the home appliance 10 can operate according to the control command received from the server 3 even without user input. Here, the control command received from the server 3 can include a control command input by the user through the user device 2 or a control command based on a predetermined condition, but is not limited thereto.

[0085] The user device 2 can send information about the user to the home appliance 10 or the server 3 through the communication module. For example, the user device 2 can send information about the user's location, the user's health status, the user's preferences, the user's schedule, etc. to the server 3. The user device 2 can send information about the user to the server 3 according to the prior approval of the user.

[0086] The home appliance 10, the user device 2, or the server 3 can determine a control command using a technology such as artificial intelligence. For example, the server 3 can receive information about the operation or status of the home appliance 10, receive information about the user of the user device 2, process the information using a technology such as artificial intelligence, and send the processing result or control command to the home appliance 10 or the user device 2 based on the processing result.

[0087] Figure 2 is a block diagram showing an electronic device according to an embodiment of the present disclosure. For example, the electronic device 201 can correspond toFigure 1 the user equipment 2 in

[0088] Referring to Figure 2 , the electronic device 201 may communicate with an external electronic device (not shown) through a short - range wireless communication network, or may communicate with at least one of an external electronic device or a server (e.g., Figure 1 the server 3 in

[0089] The processor 220 may execute, for example, software (e.g., the program 240) to control at least one other component (e.g., a hardware or software component) coupled to the processor 220 of the electronic device 201, and may perform various data processing or calculations. According to an embodiment of the present disclosure, as at least a part of the data processing or calculation, the processor 220 may store a command or data received from another component (e.g., the sensor module 276 or the communication module 290) in the volatile memory 232, process the command or data stored in the volatile memory 232, and store the resulting data in the non - volatile memory 234. According to an embodiment of the present disclosure, the processor 220 may include a main processor 221 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 223 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when the electronic device 201 includes the main processor 221 and the auxiliary processor 223, the auxiliary processor 223 may be configured to use less power than the main processor 221 or be dedicated to a specified function. The auxiliary processor 223 may be implemented separately from the main processor 221 or as part of the main processor 221.

[0090] When the main processor 221 is in an inactive (e.g., sleep) state, the auxiliary processor 223 (instead of the main processor 221) may control at least some of the functions or states related to at least one of the components of the electronic device 201 (e.g., the display module 260, the sensor module 276, or the communication module 290), or when the main processor 221 is in an active state (e.g., running an application), the auxiliary processor 223 may control at least some of the functions or states related to at least one of the components of the electronic device 201 (e.g., the display module 260, the sensor module 276, or the communication module 290) together with the main processor 221. According to an embodiment of the present disclosure, the auxiliary processor 223 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 280 or the communication module 290) that is functionally related to the auxiliary processor 223. According to an embodiment of the present disclosure, the auxiliary processor 223 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated via machine learning. Such learning may be performed by, for example, the electronic device 201 in which the artificial intelligence model is executed or by a separate server. The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more thereof, but is not limited thereto. The artificial intelligence model may additionally or alternatively include a software structure in addition to the hardware structure.

[0091] The memory 230 may store various data used by at least one component of the electronic device 201 (e.g., the processor 220 or the sensor module 276). The various data may include input / output data of software (e.g., the program 240) and commands related thereto. The memory 230 may include a volatile memory 232 or a non-volatile memory 234. The non-volatile memory 234 may include an internal memory 236 and an external memory 238.

[0092] The program 240 may be stored in the memory 230 as software, and may include an operating system (OS) 242, middleware 244, or an application 246.

[0093] The input module 250 may receive commands or data used by other components of the electronic device 201 (e.g., the processor 220) from an external source (e.g., a user) of the electronic device 201. The input module 250 may include a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0094] The sound output module 255 may output a sound signal to the outside of the electronic device 201. The sound output module 255 may include a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing a record. The receiver may be used to receive an incoming call. According to an embodiment of the present disclosure, the receiver may be implemented separately from the speaker or as part of the speaker.

[0095] The display module 260 may visually provide information to the outside (e.g., a user) of the electronic device 201. For example, the electronic device 201 may include a display having a display area, and the display may visually provide information to the outside. The display module 260 may include a display, a holographic device, or a projector, and a control circuit for controlling at least one corresponding device. According to an embodiment of the present disclosure, the display module 260 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0096] The audio module 270 may convert sound into an electrical signal and vice versa. According to an embodiment of the present disclosure, the audio module 270 may obtain sound through the input module 250, or output sound through the sound output module 255 or an external electronic device (e.g., a speaker or headphones) directly or wirelessly connected to the electronic device 201.

[0097] The sensor module 276 may detect an operating state of the electronic device 201 (e.g., power or temperature) or an environmental state outside the electronic device 201 (e.g., a user state), and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment of the present disclosure, the sensor module 276 may include a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0098] The interface 277 may support one or more specified protocols for physically or wirelessly connecting the electronic device 201 to an external electronic device. According to an embodiment of the present disclosure, the interface 277 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0099] The connection terminal 278 may include a connector through which the electronic device 201 may be physically connected to an external electronic device. According to an embodiment of the present disclosure, the connection terminal 278 may include an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0100] The haptic module 279 may convert an electrical signal into a mechanical stimulus (e.g., vibration or motion) or an electrical stimulus that can be recognized by a user via his sense of touch or kinesthesia. According to an embodiment of the present disclosure, the haptic module 279 may include a motor, a piezoelectric element, or an electrical stimulator.

[0101] The camera module 280 may capture a still image or a moving image. According to an embodiment of the present disclosure, the camera module 280 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0102] The power management module 288 may manage the power provided to the electronic device 201. According to an embodiment of the present disclosure, the power management module 288 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0103] The battery 189 may supply power to at least one component of the electronic device 201. According to an embodiment of the present disclosure, the battery 289 may include a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0104] The communication module 290 may support establishing a direct or physical (e.g., wired) communication channel or a wireless communication channel between the electronic device 201 and an external electronic device (e.g., an electronic device or a server), and perform communication through the established communication channel. The communication module 290 may include one or more communication processors, which may operate independently of the processor 220 (e.g., an application processor (AP)), and support direct (e.g., wired) communication or wireless communication. According to an embodiment of the present disclosure, the communication module 290 may include a wireless communication module 292 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 294 (e.g., a LAN communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with an external electronic device via a first network (e.g., a short-range communication network, such as Bluetooth TM , Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network (e.g., a long-range communication network, such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 292 may use the subscriber information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the subscriber identity module 296 to identify or authenticate the electronic device 201 in a communication network (e.g., the first network or the second network).

[0105] The wireless communication module 292 may support 5G networks and next-generation communication technologies following 4G networks, such as New Radio (NR) access technology. The NR access technology may support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 292 may support high frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transfer rates. The wireless communication module 292 may support various technologies for ensuring performance in high frequency bands, such as beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 292 may support various requirements specified in the electronic device 201, an external electronic device, or a network system. According to an embodiment of the present disclosure, the wireless communication module 292 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), a loss coverage for implementing mMTC (e.g., 164 dB or less), or a U-plane latency for implementing URLLC (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or a round trip of 1 ms or less).

[0106] The antenna module 297 may transmit signals or power to the outside (e.g., an external electronic device) or receive signals or power from the outside (e.g., an external electronic device). According to an embodiment of the present disclosure, the antenna module 297 may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment of the present disclosure, the antenna module 297 may include a plurality of antennas (e.g., an antenna array). In this case, for example, the communication module 290 may select at least one antenna suitable for a communication scheme used in a communication network (e.g., a first network or a second network) from among the plurality of antennas. Subsequently, signals or power may be transmitted or received between the communication module 290 and an external electronic device via the selected at least one antenna. According to an embodiment of the present disclosure, other components (e.g., a radio frequency integrated circuit (RFIC)) other than the radiator may be further formed as part of the antenna module 297.

[0107] According to various embodiments of the present disclosure, the antenna module 297 may form a millimeter wave antenna module. According to an embodiment of the present disclosure, the millimeter wave antenna module may include a PCB, an RFIC disposed on a first surface (e.g., a bottom surface) of the PCB or adjacent to the first surface and capable of supporting a specified high frequency band (e.g., a millimeter wave band), and a plurality of antennas (e.g., an array antenna) disposed on a second surface (e.g., a top surface or a side surface) of the PCB or adjacent to the second surface and capable of transmitting or receiving signals of the specified high frequency band.

[0108] At least some of the above components may be coupled to each other via an inter-peripheral communication scheme (e.g., a bus, general-purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and transmit signals (e.g., commands or data).

[0109] According to an embodiment of the present disclosure, instructions or data may be sent or received between the electronic device 201 and external electronic devices via a server coupled to a second network. Each of the external electronic devices may be a device of the same or different type as the electronic device 201. According to an embodiment of the present disclosure, all or some of the operations to be performed at the electronic device 201 may be performed at one or more external electronic devices. For example, if the electronic device 201 automatically performs a function or service or should perform a function or service in response to a request from a user or another device, the electronic device 201 may request the one or more external electronic devices to perform at least part of the function or service, rather than running the function or service, or in addition to running the function or service, the electronic device 201 may also request the one or more external electronic devices to perform at least part of the function or service. The one or more external electronic devices that receive the request may perform the requested at least part of the function or service, or perform additional functions or additional services related to the request, and transmit the result of the execution to the electronic device 201. The electronic device 201 may provide the result as at least part of a reply to the request with or without further processing of the result. For this purpose, for example, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology may be used. The electronic device 201 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment of the present disclosure, the external electronic device may include an IoT device. The server may be an intelligent server using machine learning and / or neural networks. According to an embodiment of the present disclosure, the external electronic device or the server may be included in the second network. The electronic device 201 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology or IoT-related technology.

[0110] For example, the processor 220 may display and control household appliances disposed in an indoor space based on input data including 3D data of the household appliances and the indoor space.

[0111] For example, the processor 220 may display a 3D map including the indoor space and the household appliances based on the input data. In addition, the processor 220 may display visual information about the household appliances in the form of 3D animations on the 3D map.

[0112] For example, the processor 220 may display subtasks of a household appliance sequentially based on a predetermined operation order. In addition, the processor 220 may control the household appliance that executes the subtask based on a predetermined operation.

[0113] Reference will be made below Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7a , Figure 7b , Figure 7c , Figure 8 , Figure 9a , Figure 9b , Figure 9c , Figure 10a , Figure 10b , Figure 10c , Figure 11a , Figure 11b , Figure 12 , Figure 13 and Figure 14 Exemplarily describe the operation of the electronic device 201 displaying and controlling a household appliance according to an embodiment of the present disclosure.

[0114] Figure 3 A 3D map according to an embodiment of the present disclosure is shown.

[0115] Referring to Figure 3 , the electronic device may display a 3D map 300 including an indoor space and a household appliance based on input data. For example, the input data may include 3D data of the indoor space and the household appliance. For example, the 3D map 300 may include a 3D image of the indoor space 310, which includes at least one divided space. For example, the 3D map 300 may include 3D images of household appliances 320a to 320l disposed in a predetermined divided space of the indoor space 310.

[0116] For example, the indoor space may be a predetermined indoor space including at least one divided space, such as an apartment, an office, a restaurant, a café, a hospital, and a school. The indoor space may include at least one household appliance.

[0117] For example, the household appliance may include at least one of a wall-mounted air conditioner 320a, an air purifier 320b, a floor-standing air conditioner 320c, a robotic vacuum cleaner 320d, a TV 320e, a steam wardrobe 320f, a washing machine 320g, a dryer 320h, an induction cooker 320i, a range hood 320j, a dishwasher 320k, and a refrigerator 320l. However, the types of household appliances in the present disclosure are not limited to the above examples, and in addition to the above household appliances, various other household appliances may also be included.

[0118] According to an embodiment of the present disclosure, an electronic device may obtain input data that maps an indoor space and at least one household appliance to a 3D image.

[0119] For example, the electronic device may receive input data that maps an indoor space and at least one household appliance to a 3D image from an external server (e.g., the server 3 in Figure 1 ). The external server may store 3D spatial information of the indoor space. The external server may store device information of at least one household appliance and user account information. In other words, the electronic device may receive input data in which the indoor space and the household appliance are pre-mapped from the external server.

[0120] As another example, a memory of the electronic device (e.g., the memory 230 in Figure 2 ) may store input data that maps an indoor space and at least one household appliance to a 3D image. A processor of the electronic device (e.g., the processor 220 in Figure 2 ) may obtain the input data from the memory. In other words, the electronic device may execute the input data pre-stored in the memory by using the processor.

[0121] As another example, the electronic device may obtain floor plan data of the indoor space and generate a 3D image of the indoor space. The electronic device may obtain position data of the household appliance and map a 3D image of the household appliance onto the 3D image of the indoor space. In other words, the electronic device may generate input data based on the floor plan data of the indoor space and the position data of the household appliance.

[0122] The electronic device may increase or decrease the size of a predetermined area in the 3D map 300 based on a user input. For example, the electronic device may increase the size and display at least one indoor space in the 3D map 300. For example, the electronic device may display the entire indoor space by reducing the 3D map 300.

[0123] The electronic device may focus on at least one household appliance on the 3D map 300 based on a user input. For example, the electronic device may increase the size and display a predetermined household appliance on the 3D map 300. For example, the electronic device may align a predetermined household appliance on the 3D map 300 to the center of the display area.

[0124] The electronic device may rotate the 3D map 300 based on a user input. For example, the electronic device may control the display angle of the 3D map 300 based on a user input. In other words, the user may select the direction and angle of the 3D map 300 displayed on the electronic device.

[0125] According to an embodiment of the present disclosure, an electronic device may change a method of displaying an indoor space and at least one household appliance. For example, the electronic device may provide function buttons for selecting at least one of a card-type image (not shown), a two-dimensional (2D) map (not shown), and a 3D map 300. For example, the electronic device may provide a map view switching button and may switch the card-type image to the 3D map 300 based on a user's selection of the map view switching button. For example, the electronic device may provide a map view switching button and may switch the 2D map to the 3D map 300 based on a user's selection of the map view switching button.

[0126] Figure 4 A 3D map including indoor space state information according to an embodiment of the present disclosure is shown. Figure 5 A 3D map including household appliance state information according to an embodiment of the present disclosure is shown. Figure 6 Displaying indoor space state information and household appliance state information on a 3D map according to an embodiment of the present disclosure is shown.

[0127] Referring to Figure 4 、 Figure 5 and Figure 6 , the electronic device may display a 3D map including at least one of indoor space state information and household appliance state information. For example, the 3D map may include at least one of indoor space state information and household appliance state information.

[0128] According to an embodiment of the present disclosure, the electronic device may display indoor space state information corresponding to a state of an indoor space 410 on a 3D map. Specifically, the electronic device may display the indoor space state information on the 3D map using at least one of text, numerical values, colors, icons, graphics, and contour lines. For example, the indoor space state information may include at least one of air quality, temperature, humidity, lighting, and noise of the indoor space 410.

[0129] Specifically, the indoor space 410 may include at least one partitioned space (e.g., 411, 412, 413, and 414). The electronic device may display the indoor space state information of each partitioned space. For example, the electronic device may receive the indoor space state information from at least one household appliance via a network. For example, the electronic device may receive the indoor space state information from a separate sensor installed in the indoor space 410 via a network.

[0130] The electronic device may use at least one of text, numerical values, colors, icons, graphics, and contour lines to display at least one of air quality, temperature, humidity, lighting, and noise of each partitioned space. As Figure 4As shown, the electronic device may display the average temperature (e.g., 21°C) and average humidity (e.g., 45%) of the first partitioned space 411 using numerical values and icons. For example, the electronic device may display the temperature distribution of the first partitioned space 411 using colors and contour lines. Similarly, the electronic device may use at least one of text, numerical values, colors, icons, graphics, and contour lines to display the average temperature, average humidity, and temperature distribution of each of the second partitioned space 412, third partitioned space 413, and fourth partitioned space 414.

[0131] According to an embodiment of the present disclosure, the electronic device may display home appliance status information corresponding to the status of at least one home appliance on a 3D map. Specifically, the electronic device may use at least one of text, numerical values, colors, icons, graphics, and contour lines to display the home appliance status information on the 3D map. For example, the home appliance status information may include at least one of the operation mode, remaining time, energy consumption, temperature, remaining battery level, and location of at least one home appliance.

[0132] Specifically, the electronic device may display the home appliance status information based on the real-time operation status of each home appliance set at a predetermined position in the indoor space 410. For example, the electronic device may receive home appliance status information including the operation status of the home appliance from the home appliance via a network.

[0133] As Figure 5 shown, the electronic device may display the home appliance status information based on the current operation status of the home appliance. For example, the electronic device may display the air quality of the partitioned space detected by the air purifier in text form on the 3D map. For example, the electronic device may display the desired temperature setting of the wall-mounted air conditioner in numerical form on the 3D map. For example, the electronic device may display the desired temperature setting of the floor-standing air conditioner in numerical form on the 3D map. For example, the electronic device may display the remaining battery level of the robotic vacuum cleaner in numerical form on the 3D map. For example, the electronic device may display the remaining time of a specific operation of the washing machine in numerical form on the 3D map.

[0134] Meanwhile, the indoor space status information and home appliance status information displayed on the 3D map are not limited to Figure 4 and Figure 5 the examples shown, and may include various methods other than the above display methods. For example, the electronic device may use at least one of text, numerical values, colors, icons, graphics, and contour lines to display the indoor space status information and home appliance status information on the 3D map. As Figure 6As shown, indoor space status information and household appliance status information can be displayed using at least one of the following: an icon including facial expressions (a), text indicating status (b), a bar graph (c), a temperature value (d), a remaining time value (e), and a contour line (f).

[0135] Figure 7a , Figure 7b and Figure 7c An operation of controlling a home appliance on a 3D map according to various embodiments of the present disclosure is illustrated.

[0136] Reference Figure 7a , Figure 7b and Figure 7c , the electronic device can control the operation of at least one home appliance based on a user input to a predetermined area on the 3D map.

[0137] According to an embodiment of the present disclosure, the electronic device may determine the home appliance to be controlled based on the user input of at least one home appliance on the 3D map. For example, the electronic device may determine the home appliance to be controlled based on the user selection of the 3D image of at least one home appliance. For example, the electronic device may highlight the edge of the home appliance selected based on the user input. The electronic device may display the current state of the home appliance selected based on the user input. For example, the electronic device may display the current state of the home appliance selected by the user.

[0138] The electronic device may display an operation control window of the selected home appliance. For example, the electronic device may display an operation control window indicating an operation mode and an operation guide of the selected home appliance. The electronic device may operate a home appliance selected through a network based on user input. For example, the electronic device may operate the selected home appliance based on the user's operation mode selection. The electronic device may display home appliance status information about the operated home appliance.

[0139] Reference Figure 7a , the electronic device can control the operation of a home appliance (eg, a floor air conditioner) based on a user input to a predetermined area on a 3D map. For example, the electronic device can display a home appliance (eg, a floor air conditioner) selected based on the user input, with the edge highlighted.

[0140] For example, Figure 7a As shown in part (a) of FIG. 1 , the electronic device may display the power setting (e.g., on) of the vertical air conditioner. Figure 7a As shown in part (b) of , the electronic device may display an operation control window indicating at least one of an operation mode (eg, cleaning), a set temperature (eg, 21.5° C.), and a power setting (eg, off) of the vertical air conditioner.

[0141] Reference Figure 7b , the electronic device may control the operation of a home appliance (e.g., a washing machine) based on an input to a predetermined area on the 3D map. For example, the electronic device may highlight the edge of the home appliance (e.g., a washing machine) selected based on the user input.

[0142] For example, as shown in part (a) of Figure 7b , the electronic device may display the current state of the washing machine (e.g., standby mode). For example, as shown in part (b) of Figure 7b , the electronic device may display an operation control window that indicates the operation mode (e.g., quick wash) and operation guidance (e.g., send to the washing machine) of the washing machine. For example, as shown in part (c) of Figure 7b , the electronic device may operate the washing machine based on the user's operation mode selection (e.g., select quick wash). The electronic device may display the home appliance status information of the operated washing machine (e.g., 1 hour 30 minutes remaining).

[0143] Reference Figure 7c , the electronic device may control the operation of a home appliance (e.g., a robotic vacuum cleaner) based on a user input to a predetermined area on the 3D map. For example, the electronic device may highlight the edge of the home appliance (e.g., a robotic vacuum cleaner) selected based on the user input.

[0144] For example, as shown in part (a) of Figure 7c , the electronic device may display the power setting of the robotic vacuum cleaner (e.g., on). For example, as shown in part (b) of Figure 7c , the electronic device may display an operation control window that indicates the operation mode (e.g., thorough cleaning) and operation guidance (e.g., start vacuuming) of the robotic vacuum cleaner. For example, as shown in part (c) of Figure 7c , the electronic device may operate the robotic vacuum cleaner based on the user's operation mode selection (e.g., thorough cleaning). The electronic device may display the home appliance status information about the operated robotic vacuum cleaner (e.g., cleaning time and cleaning area).

[0145] Figure 8 Shows the visual information of a home appliance according to an embodiment of the present disclosure.

[0146] Reference Figure 8 , the electronic device may display visual information about at least one home appliance in 3D animation form on the 3D map based on the home appliance status information. Specifically, Figure 8 part (a) of Figure 8Part (b) shows a 3D image of a home appliance reflecting (or showing) visual information.

[0147] According to an example, an electronic device may receive home appliance status information from a home appliance through a network, and display visual information corresponding to the home appliance status information. For example, the electronic device may store visual information about each home appliance status information of at least one home appliance. The electronic device may receive home appliance status information, and display visual information corresponding to the home appliance status information among the stored visual information. For example, the electronic device may receive visual information about each home appliance status information of at least one home appliance from an external server. The electronic device may receive home appliance status information from a home appliance, and display visual information corresponding to the home appliance status information among the visual information received from the external server.

[0148] The visual information about at least one household appliance may include at least one of a light on / off state, an output wind direction (e.g., airflow direction), an output wind speed (e.g., the strength or speed of the airflow), an output temperature, a door open / closed state, and a moving state (e.g., moving from one location to a different location) of the at least one household appliance. Figure 8 As shown in part (b) of , the electronic device can display visual information in the form of 3D animation, and the visual information includes at least one of the light on / off state, output wind direction, output wind speed, and output temperature of at least one household appliance (e.g., a floor-standing air conditioner or a wall-mounted air conditioner). In addition, the electronic device can display visual information including the moving state of at least one household appliance (e.g., a robot vacuum cleaner) in the form of 3D animation. In addition, the electronic device can display visual information including at least one of the suction and exhaust of at least one household appliance (e.g., an air purifier) ​​in the form of 3D animation. In addition, the electronic device can display visual information including the door open / close state of at least one household appliance (e.g., a refrigerator) in the form of 3D animation.

[0149] As described above, the electronic device according to the present disclosure can display a 3D map including an indoor space and home appliances based on input data, and display visual information about home appliances in the form of 3D animation on the 3D map, thereby intuitively displaying the position and operating status of the home appliances in the indoor space on the 3D map.

[0150] Figure 9a , Figure 9b and Figure 9c An operation of sequentially displaying subtasks for performing a predetermined operation by an electronic device according to various embodiments of the present disclosure is illustrated.

[0151] Reference Figure 9a , Figure 9b and Figure 9c, at least one processor may sequentially display, on a 3D map, at least one subtask for performing a predetermined operation of at least one household appliance. The predetermined operation may include a complex operation performed as a series of subtasks of at least one household appliance. For example, the predetermined operation may include at least one of a cooking operation, a washing operation, or a cleaning operation performed as a series of subtasks of at least one household appliance.

[0152] According to an embodiment of the present disclosure, at least one household appliance may include at least one object (e.g., 920a, 920b, 920c, and 920d) for performing a predetermined operation. For example, at least one household appliance may include a first object 920a and a second object 920b that perform at least one subtask for performing a predetermined operation. The predetermined operation may include performing a first subtask by the first object 920a and performing a second subtask associated with the first subtask by the second object 920b.

[0153] The electronic device may sequentially display, on the 3D map, an image showing the operation order of the first object and the second object, an enlarged image of the first object, a pop-up window including text describing the first subtask, an enlarged image of the second object, and a pop-up window including text describing the second subtask.

[0154] Referring to Figure 9a , the electronic device may display, on the 3D map, an image in which the operation order (e.g., 1, 2, 3, and 4) of at least one object (e.g., 920a, 920b, 920c, and 920d) is shown.

[0155] Referring to Figure 9b , the electronic device may display an enlarged image of the first object 920a. For example, the electronic device may display an image in which the first object 920a is emphasized and enlarged. The electronic device may display a pop-up window including text describing the first subtask. For example, the electronic device may display, in the pop-up window PUW1, guidance information for the first subtask step 1 of the first object 920a.

[0156] Referring to Figure 9c , the electronic device may display an enlarged image of the second object 920b. For example, the electronic device may display an image in which the second object 920b is emphasized and enlarged. The electronic device may display a pop-up window including text describing the second subtask. For example, the electronic device may display, in the pop-up window PUW2, guidance information for the second subtask step 2 of the second object 920b associated with the first subtask step 1 of the first object 920a.

[0157] In this way, the electronic device may sequentially display a predetermined operation performed as a series of subtasks (e.g., step 1 and step 2) of at least one household appliance.

[0158] Figure 10a , Figure 10b and Figure 10c An operation of sequentially displaying subtasks for performing a predetermined operation by an electronic device according to various embodiments of the present disclosure is illustrated.

[0159] Reference Figure 10a , Figure 10b and Figure 10c , at least one processor may sequentially display at least one subtask of at least one household appliance for performing a predetermined operation on the 3D map. The predetermined operation may include a sequential operation performed as at least one subtask of at least one household appliance capable of moving and photographing (e.g., recording a still image or a moving image). For example, the predetermined operation may include at least one of a pet search operation and a safety operation performed as at least one subtask of at least one household appliance capable of moving and photographing.

[0160] According to an example, the predetermined operation may include: moving at least one home appliance in the indoor space; detecting a target object included in the indoor space by at least one home appliance; photographing the target object by at least one home appliance; and displaying an image of the target object photographed by at least one home appliance.

[0161] The electronic device can sequentially display an image of at least one home appliance moving in the indoor space, an image of at least one home appliance detecting a target object included in the indoor space, an image of at least one home appliance photographing the target object, and an image of the target object photographed by at least one home appliance.

[0162] Reference Figure 10a , the electronic device can display an image of at least one household appliance moving in the indoor space. For example, the electronic device can display an image of at least one household appliance 1020 (eg, a robot vacuum cleaner) moving in real time in the indoor space to perform a predetermined operation (eg, a pet search operation).

[0163] Reference Figure 10b , the electronic device may display an image of at least one home appliance 1020 detecting a target object (TO) included in the indoor space, and an image of at least one home appliance 1020 photographing the target object. For example, the electronic device may display that at least one home appliance 1020 that is moving stops moving and photographs the target object after detecting a target object (e.g., a cat) in at least one partitioned space (e.g., a master bedroom) of the indoor space.

[0164] Reference Figure 10c, the electronic device may display a moving image of a target object captured by at least one household appliance 1020. For example, the electronic device may display a pop-up window that includes a moving image of a target object captured by at least one household appliance 1020 and a notification message for a separated space (e.g., the master bedroom) where the target object is detected.

[0165] In this way, the electronic device may sequentially display a predetermined operation performed as at least one subtask of at least one household appliance 1020 capable of moving and capturing images.

[0166] Figure 11a and Figure 11b illustrates an operation of controlling a household appliance by an electronic device based on a predetermined operation according to an embodiment of the present disclosure.

[0167] Referring to Figure 11a and Figure 11b , the electronic device may control at least one household appliance that performs at least one subtask based on a predetermined operation on a 3D map. The predetermined operation may be an operation of configuring the household appliance to a set value optimized for an operating environment of an indoor space. For example, the predetermined operation may be an automated operation that includes at least one of a movie operation, a weather operation, and a game operation.

[0168] Specifically, the predetermined operation may include determining an operating environment of at least one household appliance and setting at least one of a power setting, a position, and an operation mode of the at least one household appliance to a pre-stored operation configuration value.

[0169] Referring to Figure 11a , the electronic device may display pre-stored operation configuration values (e.g., off, closed, or on) of at least one household appliance for a predetermined operation on a 3D map. The electronic device may receive a user input for an operating environment of at least one household appliance. For example, the electronic device may receive a user input for selecting an automated operation. For example, the electronic device may receive a user input for selecting a movie operation from among automated operations.

[0170] Referring to Figure 11b , the electronic device may control at least one of a power setting, a position, and an operation mode of at least one household appliance based on an operation configuration value corresponding to the operating environment. For example, in a movie operation, the electronic device may turn off at least one household appliance that is irrelevant or not related to watching a movie. For example, in a movie operation, the electronic device may darken the indoor space by closing electronic blinds. For example, in a movie operation, the electronic device may reduce the illumination brightness or turn off the illumination. For example, in a movie operation, the electronic device may turn on the TV. For example, in a movie operation, the electronic device may set the operation mode of the TV to a movie mode.

[0171] Accordingly, the electronic device of the present disclosure can enhance the usability and convenience of home appliances by sequentially displaying and controlling associated operations for performing predetermined operations of home appliances.

[0172] Figure 12 An operation of an electronic device displaying a history of an indoor space state according to an embodiment of the present disclosure is shown.

[0173] Referring to Figure 12 , the electronic device can display indoor space state information corresponding to the state of the indoor space on a 3D map. For example, the indoor space state information may include at least one of air quality, temperature, humidity, lighting, and noise of the indoor space.

[0174] According to an example, the electronic device can generate a history of the indoor space state based on changes in the indoor space state information over a reference or predetermined period (e.g., 12 hours). At least one processor can display the history of the indoor space state for a specified period (e.g., hourly).

[0175] For example, as Figure 12 shown, the electronic device can store changes in the indoor space state information (e.g., air quality of the indoor space) during a reference or predetermined period (e.g., 12 hours). The electronic device can generate a history of the indoor space state (e.g., air quality history) based on changes in the air quality of the indoor space during the reference or predetermined period. For example, the electronic device can display the indoor space state information (e.g., air quality of the indoor space at a predetermined time point) at a predetermined time point based on a user input.

[0176] Figure 13 An operation of an electronic device displaying home appliance state information according to an embodiment of the present disclosure is shown.

[0177] Referring to Figure 13 , the electronic device can display home appliance state information corresponding to the state of at least one home appliance on a 3D map. For example, the home appliance state information may include at least one of an operation mode, remaining time, energy usage, temperature, remaining battery level, and location of at least one home appliance.

[0178] Specifically, the electronic device can display the operation mode, remaining time, energy usage, temperature, remaining battery level, and location of the home appliance on the 3D map using at least one of text, numerical values, colors, icons, graphics, and contour lines.

[0179] The electronic device may calculate statistical data on the home appliance status information for the indoor space based on the home appliance status information. The electronic device may aggregate the home appliance status information for each of at least one home appliance and display the home appliance status information for the indoor space as one piece of data. For example, as Figure 13 shown, the electronic device may aggregate the energy usage of at least one home appliance and calculate statistical data on the total energy usage consumed in the indoor space based on the aggregated energy usage. The electronic device may use at least one of text, numerical values, colors, icons, graphics, and contour lines to display the statistical data on the total energy usage.

[0180] Figure 14 Illustrated is an operation of detecting an operation error in a home appliance by an electronic device according to an embodiment of the present disclosure.

[0181] Referring to Figure 14 , the electronic device may detect an operation error in at least one home appliance based on the home appliance status information. For example, the electronic device may detect an operation error including at least one of abnormal high-temperature operation, abnormal power application, abnormal position change, abnormal energy consumption, and abnormal battery consumption of at least one home appliance (e.g., a refrigerator).

[0182] The electronic device may display the operation error and action guidance. For example, the electronic device may display a pop-up window including an operation error notification. The electronic device may display action guidance corresponding to the operation error in the pop-up window. The electronic device may display an access link for connecting to a service center in the pop-up window.

[0183] Figure 15 Illustrated is a method of displaying and controlling a home appliance according to an embodiment of the present disclosure.

[0184] Referring to Figure 15 , a method of displaying and controlling a home appliance according to the present disclosure may include: displaying a 3D map including at least one of indoor space status information and home appliance status information, the indoor space status information including an environmental status of the indoor space, and the home appliance status information including an operation status of at least one home appliance. A method of displaying and controlling a home appliance according to the present disclosure may include: controlling the operation of at least one home appliance based on a user input to a predetermined area on the 3D map.

[0185] According to an embodiment of the present disclosure, in operation 1510, the electronic device 201 may display a 3D map including at least one of indoor space status information corresponding to the status of the indoor space and home appliance status information corresponding to the status of at least one home appliance.

[0186] For example, the electronic device 201 may obtain input data including 3D data of an indoor space and at least one household appliance.

[0187] For example, the display 3D map may display visual information about at least one household appliance in the form of 3D animation on the 3D map based on the household appliance status information.

[0188] For example, the display 3D map may sequentially display at least one subtask for performing a predetermined operation of at least one household appliance on the 3D map.

[0189] According to the example, in operation 1520, the electronic device 201 may control the operation of at least one household appliance based on a user input to a predetermined area on the 3D map.

[0190] For example, controlling at least one household appliance may control at least one household appliance that performs at least one subtask based on a predetermined operation on the 3D map.

[0191] As described above, the method for displaying and controlling a household appliance according to the present disclosure may display a 3D map including an indoor space and a household appliance based on input data, and display visual information of the household appliance in the form of 3D animation on the 3D map, thereby intuitively displaying the position and operation status of the household appliance in the indoor space on the 3D map.

[0192] In addition, the method for displaying and controlling a household appliance according to the present disclosure may enhance the usability and convenience of the household appliance by sequentially displaying and controlling related operations for performing a predetermined operation of the household appliance.

[0193] However, since this has been described above, it will not be described again.

[0194] The electronic device according to various embodiments of the present disclosure may be one of various types of electronic devices. The electronic device may include, for example, an electronic device, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to the above-mentioned electronic devices.

[0195] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described herein to specific embodiments, and include various changes, equivalents or substitutions of the corresponding embodiments. The term "and / or" used herein should be understood to cover any and all possible combinations of one or more of the listed items. The terms "comprising", "having" and "including" used herein are only used to indicate the presence of the features, components, parts or combinations thereof described herein, but the use of these terms does not exclude the possibility of the presence or addition of one or more other features, components, parts or combinations thereof. Each phrase used herein, such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C", may cover all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as "first" and "second" or "1st" and "2nd" may be used simply to distinguish the corresponding components from another component, and do not otherwise limit the components (such as importance or order).

[0196] The term "component" or "module" used herein may include a unit implemented in hardware, software or firmware, and may be used interchangeably with other terms such as "logic", "logic block", "component" or "circuit". A component or module may be a single integrated component or its smallest unit or part suitable for performing one or more functions. For example, according to an embodiment of the present disclosure, a "component" or "module" may be implemented in the form of an application specific integrated circuit (ASIC).

[0197] As used in various embodiments of the present disclosure, the term "if" may be interpreted as "when", "at the time of", "in response to determining" or "in response to detecting" according to the context. Similarly, "if A is determined" or "if A is detected" may be interpreted as "when A is determined" or "in response to determining A", or "at the time of detecting A" or "in response to detecting A" according to the context.

[0198] The program executed by the electronic device described herein may be implemented as a hardware component, a software component and / or a combination thereof. The program may be executed by any system capable of executing computer-readable instructions.

[0199] The software may include a computer program, code, instructions, or a combination of one or more thereof, and may configure the processing device when the processing device operates as needed, or may independently or jointly direct the processing device. The software may be implemented as a computer program including instructions stored in a computer-readable storage medium. The computer-readable storage medium may include, for example, a magnetic storage medium (e.g., read-only memory (ROM), random access memory (RAM), floppy disk, hard disk, etc.) and an optically-readable medium (e.g., CD-ROM or digital versatile disc (DVD)). In addition, the computer-readable storage medium may be distributed to computer systems connected via a network, and the computer-readable code may be stored and executed in a distributed manner. The computer program may be distributed via an application store (e.g., PlayStore TM ), directly between two UEs (e.g., smart phones), or online (e.g., downloaded or uploaded). If distributed online, at least a part of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (e.g., the memory of a manufacturer's server, an application store's server, or a relay server).

[0200] According to various embodiments of the present disclosure, each of the above components (e.g., modules or programs) may include a single entity or multiple entities. Some of the multiple entities may be separately provided in different components. According to various embodiments of the present disclosure, one or more of the above components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments of the present disclosure, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as the corresponding components in the multiple components before integration. According to various embodiments of the present disclosure, the operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or may be executed in a different order or one or more operations may be omitted, or one or more other operations may be added.

[0201] Although the present disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device for displaying and controlling household appliances, the electronic device comprising: A display including a display area; and At least one processor electrically connected to the display, wherein the at least one processor is configured to: Display a three-dimensional (3D) map, the 3D map including at least one of indoor space status information or household appliance status information, the indoor space status information including the environmental status of the indoor space, and the household appliance status information including the operating status of at least one household appliance provided in the indoor space, and Based on a user input to a predetermined area on the 3D map, control the operation of the at least one household appliance.

2. The electronic device according to claim 1, wherein, The at least one processor is further configured to: Based on the household appliance status information, display visual information about the at least one household appliance in the form of a 3D animation on the 3D map.

3. The electronic device according to claim 2, wherein, The visual information about the at least one household appliance includes at least one of the following: the on / off state of the lamp of the at least one household appliance, the output air direction, the output air speed, the output temperature, the door open / close state, or the moving state.

4. The electronic device according to claim 1, wherein, The at least one processor is further configured to: Display at least one subtask for the at least one household appliance to perform a predetermined operation on the 3D map.

5. The electronic device according to claim 4, Among them, The at least one household appliance includes a first object and a second object that perform at least one subtask for performing the predetermined operation, and wherein the predetermined operation includes: Performing a first subtask by the first object, and Performing a second subtask associated with the first subtask by the second object.

6. The electronic device according to claim 5, wherein, The at least one processor is further configured to: Sequentially display an image showing the operation sequence of the first object and the second object, an enlarged image of the first object, a pop-up window including text describing the first subtask, an enlarged image of the second object, and a pop-up window including text describing the second subtask on the 3D map.

7. The electronic device according to claim 4, wherein, The predetermined operation includes: Moving the at least one household appliance in the indoor space; Detecting a target object included in the indoor space by the at least one household appliance; Taking at least one image of the target object by the at least one household appliance; and Displaying the at least one image of the target object taken by the at least one household appliance.

8. The electronic device according to claim 4, wherein, The at least one processor is further configured to: Sequentially display an image of the at least one household appliance capable of moving in the indoor space, an image of the at least one household appliance detecting a target object in the indoor space, an image of the at least one household appliance taking an image of the target object, and an image of the target object taken by the at least one household appliance.

9. The electronic device according to claim 1, wherein, The at least one processor is further configured to: Control the at least one household appliance performing at least one subtask based on a predetermined operation on the 3D map.

10. The electronic device according to claim 9, wherein, The predetermined operation includes: Determining the operating environment of the at least one household appliance; and Set at least one of the power setting, position, or operation mode of the at least one household appliance to a pre-stored operation configuration value.

11. The electronic device according to claim 9, wherein, The at least one processor is further configured to: Receive user input for the operating environment of the at least one household appliance, and Based on the operation configuration value corresponding to the operating environment, control at least one of the power setting, position, or operation mode of the at least one household appliance.

12. The electronic device according to claim 1, wherein, The at least one processor is further configured to: display at least one of the indoor space state information or the household appliance state information on the 3D map using at least one of text, numerical value, color, icon, graphic, or contour line.

13. The electronic device according to claim 1, wherein, The indoor space state information includes at least one of the air quality, temperature, humidity, lighting, or noise of the indoor space.

14. The electronic device according to claim 1, wherein, The at least one processor is further configured to: Generate a history of the indoor space state based on the change of the indoor space state information within a reference time period, and Display the history of the indoor space state within a specified time period.

15. The electronic device according to claim 1, wherein, The household appliance state information includes at least one of the operation mode, remaining time, energy usage, temperature, remaining battery level, or position of the at least one household appliance.

Images