Electronic device and control method thereof

Abstract

An electronic device is disclosed. The electronic device comprises: a communication interface; a memory in which driving map information including identification information corresponding to each of one or more home appliances in a driving space is stored; and one or more processors, wherein the one or more processors: confirm whether a home appliance is in a first captured image, when the first captured image for the driving space is received through the communication interface; acquire identification information about the confirmed home appliance on the basis of the driving map information and the first captured image, when it is confirmed that the home appliance is in the first captured image; acquire, on the basis of the confirmed identification information, function information provided by the confirmed home appliance; identify, on the basis of the acquired function information, UI information for providing a function of the confirmed home appliance together with the first captured image; and transmit, to a user terminal, the identified UI information through the communication interface.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device that, when a home appliance existing in a home is discovered through a photographed image, provides information related to the function of the discovered home appliance through a user terminal.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. IoT technology, which allows multiple devices within a home to be connected to a network and exchange data, is actively being developed. Data exchange between multiple IoT devices within a home can provide users with high-quality services.

[0003] In this case, a method is proposed in which the user can easily recognize home appliances (e.g., air conditioners or refrigerators) existing in the home by using cameras included in home appliances existing in the driving space, and control the functions of home appliances existing in the home through a single electronic device.

[0004] An electronic device according to one embodiment of the present disclosure may include a communication interface, a memory storing driving map information including identification information corresponding to each of at least one home appliance existing in a driving space and at least one instruction, and at least one processor that provides UI (User Interface) information using a photographed image received through the communication interface by executing the at least one instruction.

[0005] The one or more processors can, when a first captured image of the driving space is received through the communication interface, check whether a home appliance is present in the received first captured image.

[0006] The one or more processors may, when it is determined that a home appliance exists in the first captured image, obtain identification information of the determined home appliance based on the driving map information and the first captured image.

[0007] The one or more processors can obtain function information provided by the identified home appliance based on the obtained identification information.

[0008] The one or more processors can identify UI information for providing a function of the identified home appliance together with the first captured image based on the acquired function information.

[0009] The one or more processors may transmit the identified UI information to a user terminal via the communication interface.

[0010] A method for controlling an electronic device according to an embodiment of the present disclosure may include, when a first captured image of a driving space is received, a step of checking whether a home appliance exists in the received first captured image.

[0011] The above control method may include, when it is confirmed that a home appliance exists in the first photographed image, a step of obtaining driving map information including identification information corresponding to each of at least one home appliance existing in the driving space and identification information of the confirmed home appliance based on the first photographed image.

[0012] The above control method may include a step of obtaining function information provided by the identified home appliance based on the obtained identification information.

[0013] The above control method may include a step of identifying UI (User Interface) information for providing a function of the identified home appliance together with the first photographed image based on the acquired function information.

[0014] The above control method may include a step of transmitting the identified UI information to a user terminal.

[0015] In a non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device according to one embodiment of the present disclosure, cause the electronic device to perform an operation, the operation may include, when a first captured image of a driving space is received, a step of checking whether a home appliance is present in the received first captured image.

[0016] The above operation may include, when it is confirmed that a home appliance exists in the first captured image, a step of obtaining driving map information including identification information corresponding to each of at least one home appliance existing in the driving space and identification information of the confirmed home appliance based on the first captured image.

[0017] The above operation may include a step of obtaining function information provided by the identified home appliance based on the obtained identification information.

[0018] The above operation may include a step of identifying UI (User Interface) information for providing a function of the identified home appliance together with the first captured image based on the acquired function information.

[0019] The above operation may include a step of transmitting the identified UI information to a user terminal.

[0020] FIG. 1 is a drawing schematically illustrating a control method of an electronic device according to one embodiment.

[0021] Figure 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0022] Figure 3 is a flowchart for explaining a control method of an electronic device according to one embodiment.

[0023] FIG. 4 is a diagram illustrating a method for identifying UI information according to one embodiment.

[0024] FIG. 5 is a drawing for explaining a method for providing a second captured image according to one embodiment.

[0025] FIG. 6 is a drawing for explaining a method for providing a second captured image according to one embodiment.

[0026] FIG. 7 is a drawing for explaining a method for providing UI information according to one embodiment.

[0027] FIGS. 8A and 8B are diagrams illustrating a method for controlling a robot based on user movement information according to one embodiment.

[0028] FIGS. 9A to 9C are diagrams for explaining a method for identifying UI information when a plurality of cameras are included in a home appliance according to one embodiment.

[0029] FIG. 10 is a drawing for explaining a method for providing a captured image corresponding to one of a plurality of home appliances according to one embodiment.

[0030] FIG. 11 is a diagram illustrating a method for providing a second captured image to a user according to one embodiment.

[0031] FIG. 12 is a diagram illustrating a method for providing a second captured image to a user according to one embodiment.

[0032] Figure 13 is a block diagram showing a detailed configuration of an electronic device according to one embodiment.

[0033] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0034] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0035] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0036] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0037] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0038] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0039] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be coupled via another component (e.g., a third component).

[0040] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0042] FIG. 1 is a drawing schematically illustrating a control method of an electronic device according to one embodiment.

[0043] According to FIG. 1, according to one embodiment, an electronic device (100) can provide UI (User Interface) information to a user through a user terminal (10) using a photographed image taken of the interior of a home.

[0044] For example, when an image captured of a home (or a driving space) is received, the electronic device (100) can identify a home appliance included in the received image. The electronic device (100) can identify information related to the function of the home appliance included in the image captured, and transmit a UI (20) containing information related to the identified function to the user terminal (10).

[0045] For example, if an air conditioner and a refrigerator are included in the captured image, the electronic device (100) can identify information related to the function of the air conditioner and information related to the function of the refrigerator. Based on the information related to the function of the identified air conditioner, the electronic device (100) can identify a guide UI (21) for providing the function of the air conditioner, and based on the information related to the function of the identified refrigerator, can identify a guide UI (22) for providing the function of the refrigerator. The electronic device (100) can transmit a UI (20) including the identified guide UIs (21 and 22) to the user terminal (10). In this case, the UI (20) can include a captured image of the driving space.

[0046] Hereinafter, various embodiments will be described in which a user can intuitively control multiple home appliances within a driving space by providing a UI for providing functions of home appliances included in a captured image together with the captured image when the home appliances are included in the captured image.

[0047] Figure 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0048] According to FIG. 2, the electronic device (100) may include a communication interface (110), a memory (120), and one or more processors (130).

[0049] The electronic device (100) may be implemented as a device that processes data like a server and communicates with external devices according to one embodiment, but is not limited thereto. For example, the electronic device (100) may be implemented as a cloud server that is connected to a plurality of IoT devices for communication, but is not limited thereto, and the electronic device (100) may be implemented as various devices such as a smart TV, a tablet, a monitor, a smart phone, a desktop computer, a laptop computer, etc. The electronic device (100) according to one embodiment of the present disclosure is not limited to the above-described devices, and the electronic device (100) may be implemented as an electronic device (100) having the functions of two or more of the above-described devices.

[0050] The external device (not shown) may be implemented as an IoT device installed in a home according to one embodiment, but is not limited thereto. For example, the IoT device may be implemented in the form of an oven, a microwave, an air conditioner, a washing machine, a TV, or a refrigerator, and may be a variety of electronic devices that communicate with the electronic device (100), but is not limited thereto.

[0051] An external device (not shown) may be implemented as an external device (not shown) having two or more functions of the above-described devices.

[0052] Meanwhile, the electronic device (100) can communicate with an external device (not shown) in various ways. According to one embodiment, a communication module for communication with an external device (not shown) may be implemented in the same manner. For example, the electronic device (100) can communicate with an external device (not shown) using a Bluetooth module.

[0053] The communication interface (110) can input and output various types of data. For example, the communication interface (110) can transmit and receive various types of data to and from an external device (not shown) (e.g., a source device), an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive) through a communication method such as AP-based Wi-Fi (Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0054] For example, the communication interface (110) may include a BLE (Bluetooth Low Energy) module. BLE refers to Bluetooth technology that enables transmission and reception of low-power, low-capacity data in the 2.4 GHz frequency band with a range of about 10 m. However, the present invention is not limited thereto, and the communication interface (110) may also include a Wi-Fi communication module. That is, the communication interface (110) may include at least one of a BLE (Bluetooth Low Energy) module or a Wi-Fi communication module.

[0055] The memory (120) can store data required for various embodiments. The memory (120) may be implemented in the form of a memory embedded in the electronic device (100) or may be implemented in the form of a memory that can be detachably attached to the electronic device (100) depending on the purpose of data storage. For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be detachably attached to the electronic device (100).

[0056] Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)). In addition, in the case of memory that can be detachably attached to the electronic device (100), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented.

[0057] According to one embodiment, the memory (120) may store driving map information including identification information corresponding to each of at least one home appliance existing in the driving space. Here, the identification information means information for identifying at least one home appliance existing in the driving space, and according to one example, the identification information may include at least one of model information of the home appliance, an identifier, location information of the home appliance on the driving map, and function information of the home appliance.

[0058] Here, each home appliance existing in the driving space may be communicatively connected to the electronic device (100) and may transmit and receive information through the communication interface (110). The identifier of the home appliance refers to unique information corresponding to at least one home appliance within the driving space. The electronic device (100) may distinguish and recognize at least one home appliance through identification information including the identifier of the home appliance, and may transmit and receive information with at least one home appliance. The functional information of the home appliance refers to information regarding the service provided by the home appliance. As an example, the functional information of the home appliance may include information regarding the function of providing images through a camera included in the home appliance. However, the present invention is not limited thereto.

[0059] One or more processors (130) (hereinafter, “processors”) are electrically connected to the communication interface (110) and the memory (120) to control the overall operation of the electronic device (100). The processor (130) may be composed of one or more processors. Specifically, the processor (130) may perform operations of the electronic device (100) according to various embodiments of the present disclosure by executing at least one instruction stored in the memory (120).

[0060] According to one embodiment, the processor (130) may be implemented as a digital signal processor (DSP), a microprocessor, a graphics processing unit (GPU), an artificial intelligence (AI) processor, a neural processing unit (NPU), or a time controller (TCON) for processing a digital image signal. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, or may be defined by the relevant terms. In addition, the processor (130) may be implemented as a system on chip (SoC) or large scale integration (LSI) having a built-in processing algorithm, or may be implemented in the form of an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0061] According to one embodiment, the processor (130) may receive a photographed image of a driving space from an external device (not shown) via a communication interface (110). According to one example, the processor (130) may be communicatively connected to a robot (not shown) and may receive a first photographed image from the robot (not shown) via the communication interface (110). Here, the first photographed image refers to a photographed image acquired via a camera included in the robot (not shown).

[0062] According to one embodiment, the processor (130) may determine whether a home appliance exists based on a captured image received through the communication interface (110). For example, when the processor (130) receives a first captured image of a driving space from a robot (not shown) through the communication interface (110), the processor (130) may determine whether a home appliance exists in the received first captured image. For example, the memory (120) may store a neural network model trained to identify objects (e.g., a refrigerator, an air conditioner, etc.) included in the image, and the processor (130) may input the first captured image received through the communication interface (110) into the trained neural network model to determine whether a home appliance exists in the first captured image. However, the present invention is not limited thereto, and the processor (130) may of course analyze the first captured image through a preset algorithm to determine whether an object included in the first captured image exists.

[0063] According to one embodiment, the processor (130) may obtain identification information of a home appliance identified in a captured image. According to one example, if the presence of a home appliance in the first captured image is confirmed, the processor (130) may obtain identification information of the identified home appliance based on the driving map information and the first captured image.

[0064] For example, if a home appliance is identified in the first captured image, the processor (130) can identify the home appliance in the first captured image using the location of the robot (not shown) that provided the first captured image, the location of the home appliance in the first captured image, and driving map information, and obtain identification information corresponding to the identified home appliance. This will be described in detail with reference to FIG. 4.

[0065] According to one embodiment, the processor (130) may obtain function information provided by the identified home appliance based on the obtained identification information. According to one example, the identification information may include at least one of model information, identifier, location information on a driving map, and function information of at least one home appliance existing in the driving space. The processor (130) may obtain identification information of a home appliance existing in a first captured image among the identification information of at least one home appliance existing in the driving space stored in the memory (120), thereby obtaining function information of the home appliance.

[0066] For example, let's assume that the home appliance present in the first captured image is an air conditioner. The memory (120) may store identification information about the air conditioner present in the driving space, and the identification information about the air conditioner may include model information of the air conditioner, an identifier of the air conditioner, location information of the air conditioner on the driving map, and information about the function of the air conditioner (e.g., output mode of the air conditioner, power on / off, video provision function through a camera included in the air conditioner, etc.).

[0067] According to one embodiment, the processor (130) may identify UI (User Interface) information for providing a function of the home appliance identified together with the first captured image based on the acquired function information. According to one example, the processor (130) may identify a guide UI corresponding to each of at least one function of the home appliance included in the first captured image, and based on this, may identify UI information overlapping the identified guide UI with the location of the home appliance within the first captured image. This will be described in detail with reference to FIG. 4.

[0068] According to one embodiment, the processor (130) may transmit the identified UI information to the user terminal via the communication interface (110). According to one example, when UI information including a guide UI corresponding to each of at least one function of a home appliance included in the first captured image is identified, the processor (130) may transmit the identified UI information to the user terminal.

[0069] Figure 3 is a flowchart for explaining a control method of an electronic device according to one embodiment.

[0070] According to FIG. 3, first, according to one embodiment, the control method can identify whether a first captured image for a driving space is received (S310).

[0071] Next, according to one embodiment, when a first captured image of a driving space is received (S310: Y), the control method can check whether a home appliance exists in the received first captured image (S320). According to one example, the processor (130) can receive the first captured image from a robot (not shown). The memory (120) may store a neural network model trained to identify objects included in the image, and the processor (130) can input the first captured image received through the communication interface (110) into the trained neural network model to check whether a home appliance exists in the first captured image. Alternatively, according to one example, the processor (130) can check whether a home appliance exists in the first captured image through a preset algorithm.

[0072] Next, according to one embodiment, if it is confirmed that a home appliance exists in the first captured image (S320: Y), the control method can obtain identification information of the home appliance identified based on the first captured image and driving map information including identification information corresponding to each of at least one home appliance existing in the driving space (S330). According to one example, the processor (130) inputs the first captured image into the learned neural network model, and if it is confirmed that an air conditioner is included in the first captured image, the processor (130) can obtain identification information of the air conditioner based on the driving map information and the first captured image.

[0073] For example, the processor (130) may identify at least one air conditioner existing in the driving map based on driving map information, and may identify an air conditioner existing in the first captured image among the at least one air conditioner based on the location of the air conditioner in the first captured image. In this case, the location information of the robot (not shown) that provided the first captured image may be used. The processor (130) may identify a home appliance that matches the air conditioner included in the captured image on the driving map based on the location information of the robot (not shown) and the first captured image. When a home appliance that matches the air conditioner included in the captured image on the driving map is identified, the processor (130) may obtain identification information corresponding to the identified home appliance among the identification information stored in the memory (120) as identification information of the air conditioner included in the captured image, which will be described in detail with reference to FIG. 4.

[0074] Next, according to one embodiment, the control method may acquire functional information provided by the identified home appliance based on the acquired identification information (S340). According to one example, the identification information may include functional information, and the processor (130) may acquire functional information included in the identification information corresponding to the air conditioner present in the first captured image.

[0075] Next, according to one embodiment, the control method may identify UI (User Interface) information for providing a function of the home appliance identified together with the first captured image based on the acquired function information (S350). According to one example, when the function information of the air conditioner is acquired, the processor (130) may identify UI (User Interface) information in which a guide UI corresponding to each of at least one function of the air conditioner is overlapped on the first captured image based on the function information of the air conditioner. In this case, the guide UI may be displayed at the location of the air conditioner within the first captured image.

[0076] Next, according to one embodiment, the control method can transmit the identified UI information to a user terminal (not shown) (S360).

[0077] According to the example described above, the electronic device (100) can identify home appliances using captured images received from a robot (not shown) and provide the user with UI (User Interface) information for providing the functions of the identified home appliances. Accordingly, user convenience can be enhanced.

[0078] FIG. 4 is a diagram illustrating a method for identifying UI information according to one embodiment.

[0079] According to FIG. 4, according to one embodiment, when a first captured image is acquired from a robot (420), the processor (130) can acquire identification information of the home appliance based on the acquired first captured image. Here, the first captured image refers to an image acquired through a camera included in the robot (420).

[0080] For example, a first captured image acquired through a camera included in a robot (420) may include a first object and a second object. The robot (420) may transmit the acquired first captured image to an electronic device (400). When the first captured image is received from the robot (420) through the communication interface (110), the processor (130) may identify the first object and the second object within the received first captured image.

[0081] For example, when a first object and a second object are identified in a first captured image, the processor (130) may obtain identification information of the first object and the second object based on driving map information stored in the memory (120).

[0082] For example, the processor (130) can identify the locations of the first object and the second object in the first captured image, respectively. Subsequently, the processor (130) can identify home appliances that match the first object and the second object, respectively, on the driving map. In this case, the processor (130) can utilize the location information of the robot (420). For example, the processor (130) can receive the first captured image from the robot (420) through the communication interface (110), along with the location information of the robot (420) and the shooting direction information of the camera at the time of acquiring the captured image of the robot (420). The processor (130) identifies home appliances on the driving map that match the first object and the second object included in the first shooting image on the driving map based on the location information and shooting direction information of the robot (420), as an air conditioner (430) and a refrigerator (440), respectively, and obtains identification information of the air conditioner (430) and identification information of the refrigerator (440), respectively, as identification information corresponding to the first object and the second object, based on the information stored in the memory (120).

[0083] For example, the processor (130) may identify UI information for providing a function of a home appliance included in a first captured image based on function information included in the acquired identification information.

[0084] For example, assume that an air conditioner (430) is identified on the left side of the first captured image and a refrigerator (440) is identified on the right side. The processor (130) may identify a guide UI (430-1) for providing a function corresponding to the air conditioner (430) at the location of the air conditioner (430) in the first captured image based on function information included in the identification information of the air conditioner (430), and may identify a guide UI (440-1) for providing a function corresponding to the refrigerator (440) at the location of the refrigerator (440) in the first captured image based on function information included in the identification information of the refrigerator (440). The processor (130) may transmit UI information (411) including a plurality of identified guide UIs (430-1 and 440-1) and the first captured image to the user terminal (410) through the communication interface (110). In this case, multiple guide UIs (430-1 and 440-1) can be provided to the user by overlapping the first captured image.

[0085] According to the above-described example, if a home appliance is included in a captured image acquired while the robot (420) is driving, the electronic device (400) can provide a UI for providing functions of the home appliance included in the captured image to the user along with the captured image. Accordingly, the user can freely control multiple home appliances within the driving space through the user terminal, thereby increasing usability. In addition, since the home appliance can be operated using the captured image acquired through the robot (420), the user can operate the home appliance more intuitively.

[0086] FIG. 5 is a drawing for explaining a method for providing a second captured image according to one embodiment.

[0087] According to FIG. 5, according to one embodiment, the control method may first determine whether driving information is received from the robot (S510). Here, the driving information may include location information of the robot, movement direction information of the robot, and movement speed information of the robot. According to one example, the processor (130) may receive the driving information of the robot together with the first captured image from the robot, and in this case, the driving information of the robot may include time information corresponding to the acquisition time of the first captured image, location information of the robot corresponding to the acquisition time of the first captured image, movement direction information of the robot, and movement speed information of the robot.

[0088] Next, according to one embodiment, when driving information is received from the robot (S510: Y), the control method can identify whether a home appliance exists within a threshold distance from the robot based on the received driving information (S520). According to one example, the processor (130) can identify a home appliance existing within a threshold distance from the robot based on driving map information stored in the memory (120) and location information of the robot. In this case, the processor (130) can also identify a home appliance existing within a threshold distance from the robot's driving path among home appliances existing within a threshold distance from the robot based on movement direction information of the robot.

[0089] Meanwhile, according to an example, the processor (130) may calculate a critical distance. If a home appliance within a driving space includes a camera, the processor (130) may calculate the critical distance based on the time required for camera initialization, the predicted time until the home appliance is exposed to the camera included in the robot, and the maximum speed information of the robot. For example, if the robot is driving at maximum speed, the processor (130) may calculate a value greater than the first distance that is the same as the time required for camera initialization and the predicted time until the home appliance is exposed to the camera included in the robot, as the critical distance. The processor (130) may store the calculated critical distance in the memory (120).

[0090] Cameras embedded in home appliances require time to initialize. If the camera initialization is completed after the home appliance is exposed to the first captured image, as described below, the processor (130) may experience problems providing the user with the captured image contained in the home appliance. Therefore, to provide the user with appropriate functionality, the camera embedded in the home appliance must be initialized before the home appliance is exposed to the first captured image.

[0091] Next, according to one embodiment, the control method may transmit request information for initializing a camera included in a home appliance within the threshold distance to the home appliance within the threshold distance based on the received driving information (S520: Y). According to one example, if the home appliance is within the threshold distance, the processor (130) may identify whether the home appliance includes a camera based on identification information of the home appliance. If the processor (130) identifies that the home appliance within the threshold distance includes a camera, the processor (130) may transmit request information for initializing the camera of the home appliance to the home appliance through the communication interface (110). When the home appliance receives the request information from the electronic device (100), the home appliance may perform initialization of the camera of the home appliance.

[0092] Next, according to one embodiment, the control method may receive a second captured image acquired through a camera from a home appliance existing within a threshold distance and transmit the second captured image to a user terminal (S540). Here, the second captured image refers to a captured image acquired through a camera included in the home appliance. For example, in the case of a refrigerator including a camera, the second captured image may be acquired through a camera included in the refrigerator. According to one example, when a refrigerator exists within a threshold distance from the robot, the processor (130) may receive a second captured image acquired through a camera included in the refrigerator through the communication interface (110). The processor (130) may transmit the received second captured image to the user terminal through the communication interface (110).

[0093] According to the above-described example, the electronic device (100) can identify a home appliance within a threshold distance from the robot based on driving information. The electronic device (100) can then provide the user with a captured image acquired through a camera included in the identified home appliance. This can increase user satisfaction. Meanwhile, the electronic device (100) can control the home appliance so that the second captured image can be acquired without issue by initializing the camera included in the home appliance within the threshold distance.

[0094] FIG. 6 is a drawing for explaining a method for providing a second captured image according to one embodiment.

[0095] According to FIG. 6, according to one embodiment, the control method may first determine whether a second captured image is received (S610). According to one example, the processor (130) may receive the second captured image from a home appliance within the driving space via the communication interface (110). Here, the second captured image refers to a captured image acquired via a camera included in the home appliance. Meanwhile, according to one example, the processor (130) may receive the second captured image from a home appliance located within a threshold distance from the robot. For example, the processor (130) may receive the second captured image from a refrigerator located within a threshold distance from the robot via the communication interface (110).

[0096] Next, according to one embodiment, when a second captured image is received (S610: Y), the control method may store the second captured image in the memory (120) (S620). According to one example, when a second captured image is received from a home appliance within a threshold distance from the robot, the processor (130) may buffer the received second captured image. For example, the processor (130) may buffer a second captured image received from a refrigerator in the memory (120).

[0097] Next, according to one embodiment, the control method can check whether a home appliance exists in the first photographed image (S630).

[0098] Next, according to one embodiment, the control method may transmit the stored second captured image to the user terminal if a home appliance is present in the first captured image (S630: Y). According to one example, if the processor (130) identifies that a refrigerator is included in the first captured image received from the robot, the processor (130) may transmit a second captured image corresponding to the refrigerator as a captured image stored in the memory (120) to the user terminal through the communication interface (110).

[0099] According to one embodiment, if a home appliance is present in the first captured image, the processor (130) may identify UI information for providing a second captured image corresponding to the home appliance. When the identified UI information is transmitted to the user terminal and a user input for reproducing the second captured image is received from the user terminal, the processor (130) may transmit the second captured image corresponding to the home appliance to the user terminal via the communication interface (110).

[0100] According to the above-described example, if there is a home appliance within a threshold distance from the robot, the electronic device (100) can receive and buffer a second captured image from the home appliance before the home appliance is exposed to the first captured image captured by the robot. Accordingly, if the home appliance is exposed to the first captured image, the second captured image can be transmitted directly to the user terminal without a separate waiting time.

[0101] FIG. 7 is a drawing for explaining a method for providing UI information according to one embodiment.

[0102] According to one embodiment, the processor (130) may transmit UI information including the first captured image and the second captured image to the user terminal (700).

[0103] According to an example, the processor (130) may transmit UI information (710) in which the second captured image overlaps the position of the home appliance present in the first captured image to the user terminal through the communication interface (110). According to an example, the overlapped UI information (710) may include a second captured image corresponding to each of at least one home appliance present in the first captured image, and may also include an exterior image of the home appliance corresponding to each of the second captured images.

[0104] For example, let's assume that an oven and a refrigerator are present in a first captured image. If the oven is present on the relatively left side of the first captured image, the processor (130) can identify UI information (710) that a second captured image (720 or a second captured image corresponding to the oven) received from the oven overlaps the position of the oven in the first captured image. In this case, the second captured image (720) corresponding to the oven can be overlapped with the first captured image together with an exterior image of the oven.

[0105] In addition, when a refrigerator is present on the relatively right side of the first captured image, the processor (130) can identify UI information (710) in which a second captured image (730 or a second captured image corresponding to the refrigerator) received from the refrigerator overlaps the location of the refrigerator in the first captured image. In this case, the second captured image (730) corresponding to the refrigerator can be overlapped with the first captured image together with an exterior image of the refrigerator. The processor (130) can transmit UI information (710) in which the second captured image (720) corresponding to the oven and the captured image (730) corresponding to the refrigerator overlap the locations of the home appliances present in the first captured image to the user terminal (700) through the communication interface (110).

[0106] Meanwhile, according to one embodiment, the processor (130) may transmit UI information including the first captured image and the second captured image to the user terminal (700) based on a user input. According to one example, when a user input for displaying a second captured image corresponding to at least one home appliance included in the first captured image is received from the user terminal (700), the processor (130) may transmit UI information (710) in which the second captured image is overlapped within the first captured image to the user terminal.

[0107] For example, if multiple home appliances exist in the first captured image, the processor (130) may display a second captured image of one of the multiple home appliances. For example, if a user input for displaying a second captured image corresponding to an oven is received, the processor (130) may identify UI information overlapping only the second captured image corresponding to the oven among the multiple home appliances present in the first captured image and transmit the overlapping UI information to the user terminal.

[0108] FIGS. 8A and 8B are diagrams illustrating a method for controlling a robot based on user movement information according to one embodiment.

[0109] First, according to FIG. 8A, according to an embodiment, the control method can determine whether user movement information is received from a user terminal (S810). Here, the user movement information refers to the direction rotation information of the user terminal sensed through a movement detection sensor (e.g., a gyro sensor, an acceleration sensor, etc.) included in the user terminal. For example, the user movement information may include at least one of the left-right rotation angle or the up-down rotation angle of the user terminal. Meanwhile, as illustrated in FIG. 8B, according to an example, the user terminal may be implemented as at least one of a smartphone or a VR (Virtual Reality) device.

[0110] Next, according to one embodiment, when user movement information is received from a user terminal (S810: Y), the control method can identify whether the user movement size is greater than or equal to a threshold value based on the received information (S820).

[0111] Next, according to one embodiment, if the control method determines that the user movement size is greater than or equal to a threshold value (S820: Y), the robot can identify control information to correspond to the received user movement (S830). According to one example, if the processor (130) determines that the user has rotated 30º left and right based on the received user input (for example, if the user has rotated his head 30º from left to right), the processor (130) can identify control information for rotating the robot's head 30º left and right.

[0112] Next, according to one embodiment, the control method can transmit the identified control information to the robot (S840).

[0113] According to FIG. 8b, according to one embodiment, the electronic device (800) can control the robot (830) based on user movement information received from the user terminals (810 and 820).

[0114] For example, when a user turns his / her head 30º from left to right, the user terminals (810 and 820) can identify corresponding user movement information and transmit the information to the electronic device (800). When the processor (130) receives user movement information indicating that the user has turned 30º left and right (or left to right) from the user terminals (810 and 820), the processor (130) can identify control information for rotating the robot's head 30º left and right based on the received information. The processor (130) can transmit the identified control information to the robot (830) through the communication interface (110), and the robot (830) can control the driving unit of the robot (830) so that the robot's head rotates 30º left and right based on the received control information.

[0115] Alternatively, for example, if the user turns his / her head 30º from downward to upward (or, if the gaze that was relatively directed toward the floor turns toward the sky), the user terminals (810 and 820) can identify corresponding user movement information and transmit it to the electronic device (800). When the processor (130) receives user movement information indicating that the user has turned 30º upward and downward (or downward and upward) from the user terminals (810 and 820), the processor (130) can identify control information for rotating the robot's head 30º upward and downward based on the received information. The processor (130) can transmit the identified control information to the robot (830) through the communication interface (110), and the robot (830) can control the driving unit of the robot (830) to rotate the robot's head 30º upward and downward based on the received control information.

[0116] FIGS. 9A to 9C are diagrams for explaining a method for identifying UI information when a plurality of cameras are included in a home appliance according to one embodiment.

[0117] First, according to FIG. 9A, according to one embodiment, the control method may transmit a second captured image corresponding to a first camera among a plurality of cameras included in a home appliance to a user terminal (S910). According to one example, it is assumed that at least one home appliance (e.g., a refrigerator) included in the first captured image acquired from the robot includes a plurality of cameras. If a refrigerator is included in the first captured image, the processor (130) may receive a second captured image corresponding to the first camera among the plurality of cameras included in the refrigerator from the refrigerator, overlap the second captured image with the first captured image, and transmit the second captured image to the user terminal.

[0118] Alternatively, according to an example, when a user input for displaying a second captured image corresponding to a first camera among a plurality of cameras included in a refrigerator is received from a user terminal, the processor (130) may transmit the second captured image to the user terminal by overlapping the first captured image based on the user input.

[0119] Next, according to one embodiment, the control method can determine whether a user input for selecting a second camera among a plurality of cameras is received from the user terminal (S920). According to one example, it is assumed that a second captured image corresponding to a first camera among a plurality of cameras corresponding to a refrigerator is displayed. The processor (130) can identify whether a user input for selecting a second camera different from the first camera among the plurality of cameras included in the refrigerator is received from the user terminal.

[0120] Next, according to one embodiment, when a user input for selecting a second camera among a plurality of cameras is received (S920: Y), the control method can transmit a second captured image corresponding to the second camera to the user terminal based on the received user input (S930).

[0121] For example, the processor (130) may receive a user input for selecting a second camera among a plurality of cameras included in the refrigerator. For example, the processor (130) may receive a user input for changing a currently provided second captured image (a second captured image corresponding to the first camera) to a second captured image corresponding to the second camera, and based on this, the processor may receive a second captured image corresponding to the second camera from the refrigerator.

[0122] Alternatively, according to an example, the processor (130) may transmit a second captured image corresponding to the second camera to the user terminal based on user movement information. For example, assume that the second camera is located relatively higher than the first camera in the refrigerator. If the processor (130) receives user movement information from the user terminal indicating that the user's head has rotated from a downward direction to an upward direction by a preset value or more, the processor (130) may identify this as a user input for selecting the second camera among the multiple cameras included in the refrigerator. Subsequently, the processor (130) may receive a second captured image corresponding to the second camera located relatively higher from the refrigerator based on this and transmit the second captured image to the user terminal through the communication interface (110).

[0123] Meanwhile, according to one embodiment, the processor (130) may store second captured images corresponding to each of the plurality of cameras included in the home appliance in the memory (120). According to one example, when the processor (130) identifies that the refrigerator is located within a threshold distance from the robot, the processor (130) may receive second captured images corresponding to each of the plurality of cameras included in the refrigerator from the refrigerator and store them in the memory (120). When a user input for selecting a second camera among the plurality of cameras included in the refrigerator is received from the user terminal through the communication interface (110), the processor (130) may transmit a second captured image corresponding to the second camera among the second captured images stored in the memory (120) to the user terminal.

[0124] Accordingly, the electronic device (100) can provide the user with a second captured image immediately without much loading time even when a user input for changing a camera included in the home appliance is received, thereby improving user satisfaction.

[0125] According to FIGS. 9b and 9c, in one embodiment, when a home appliance existing in a first captured image includes multiple cameras, the processor (130) can identify UI information including a second captured image corresponding to one of the multiple cameras based on a user input.

[0126] For example, as illustrated in FIG. 9b, a refrigerator (90) may have multiple cameras and second captured images (91 to 93) corresponding to each of the multiple cameras. For example, assume that a UI including a second captured image (92) corresponding to a first camera among the multiple cameras corresponding to the refrigerator is provided to the user. The processor (130) may receive a user input from the user to display a second captured image (91) corresponding to the second camera. For example, the processor (130) may receive user movement information from the user terminal indicating that the user's head has rotated from a downward direction to an upward direction by a preset value or more. As illustrated in FIG. 9c, the processor (130) may identify UI information for changing the second captured image provided to the user to a second captured image (91) corresponding to the second camera based on the received user movement information.

[0127] FIG. 10 is a drawing for explaining a method for providing a captured image corresponding to one of a plurality of home appliances according to one embodiment.

[0128] According to FIG. 10, first, according to one embodiment, the control method can determine whether a user input for selecting one of a plurality of home appliances present in a driving space is received (S1010). As an example, as illustrated in FIG. 12, assume a case where a plurality of home appliances (1220 to 1240) are present in a driving space (1200). The processor (130) can receive a user input for selecting one of the plurality of home appliances (1220 to 1240) from a user terminal via a communication interface (110).

[0129] For example, let us assume that UI information including a second photographed image corresponding to a current refrigerator (1230) is being provided to the user. That is, let us assume that a first photographed image acquired through a camera corresponding to a robot (1210) includes a refrigerator (1230). The processor (130) may receive a user input for selecting an oven (1220) located relatively to the left based on the photographing direction of the camera corresponding to the robot (1210) from a user terminal through a communication interface (110). For example, the processor (130) may receive user movement information indicating that the user has rotated more than a preset value in the left-right direction (or from the left-to-right direction) from the user terminal, and may identify a user input for selecting an oven (1220) from among a plurality of home appliances existing in a driving space based on the received user movement information and driving map information.

[0130] However, it is not limited thereto, and as an example, the processor (130) may receive a selection input from a user terminal to select an oven among multiple home appliances.

[0131] Next, according to one embodiment, when a user input for selecting one of a plurality of home appliances is received (S1010: Y), the control method can transmit a second captured image corresponding to one of the selected home appliances to the user terminal based on the received user input (S1020).

[0132] For example, as illustrated in FIG. 12, when a plurality of home appliances exist in a driving space and a user input for selecting an oven (1220) is received, the processor (130) may receive a second captured image corresponding to the oven (1220) from the oven (1220) based on the received user input. In this case, the shooting direction of the camera corresponding to the robot (1210) may also be changed so that the oven (1220) is included in the first captured image based on the user input for selecting the oven (1220). The processor (130) may identify UI information for overlapping the received second captured image corresponding to the oven (1220) with the first captured image, and transmit the same to the user terminal.

[0133] According to the above-described example, the electronic device (100) can provide a second captured image corresponding to the home appliance desired by the user, and the user can view the second captured image corresponding to the desired home appliance with a simple operation. Accordingly, user satisfaction is enhanced.

[0134] FIG. 11 is a diagram illustrating a method for providing a second captured image to a user according to one embodiment.

[0135] According to FIG. 11, first, according to one embodiment, the control method may determine whether a second captured image corresponding to each of the plurality of home appliances is received from each of the plurality of home appliances (S1110). According to one example, the processor (130) may identify a home appliance including a camera among the plurality of home appliances present in the driving space and transmit a signal requesting a second captured image corresponding to each of the home appliances including the camera.

[0136] Next, according to one embodiment, when a second captured image corresponding to each of a plurality of home appliances is received (S1110: Y), the control method may store each of the received second captured images in the memory (120). According to one example, the processor (130) may transmit a signal requesting a second captured image corresponding to each of the home appliances including a camera to each home appliance, and based on the signal, may receive the second captured image from the home appliance including a camera. The processor (130) may store (or buffer) the received second captured images corresponding to each of the plurality of home appliances in the memory (120).

[0137] Next, according to one embodiment, the control method can determine whether a user input for selecting one of a plurality of home appliances is received (S1130). As an example, as illustrated in FIG. 12, assume that there are a plurality of home appliances (1220 to 1240) within a driving space (1200). The processor (130) can receive a user input for selecting an oven (1220) from among the plurality of home appliances (1220 to 1240) from a user terminal via a communication interface (110).

[0138] Next, according to one embodiment, when a user input for selecting one of a plurality of home appliances is received (S1130: Y), the control method may transmit a second captured image corresponding to one of the stored second captured images to the user terminal (S1140). According to one example, when a user input for selecting an oven (1220) among the plurality of home appliances is received, the processor (130) may identify a second captured image corresponding to the oven (1220) among the second captured images corresponding to each of the plurality of home appliances stored in the memory (120), and transmit the same to the user terminal. For example, the processor (130) may identify UI information including the second captured image corresponding to the oven (1220) stored in the memory (120) and the second captured image received in real time from the oven (1220), and transmit the same to the user terminal.

[0139] According to the above-described example, the electronic device (100) can buffer second captured images corresponding to each of a plurality of home appliances present in the driving space, and when a user input is received, the second captured images can be provided immediately without a separate loading time. Accordingly, user satisfaction is enhanced and usability is increased.

[0140] However, the present invention is not limited thereto, and according to an embodiment, the processor (130) may receive and store a second captured image corresponding to a home appliance within a critical distance from the robot among a plurality of home appliances within the driving space. According to an example, as illustrated in FIG. 12, when the home appliances within a critical distance from the robot (1210) are a refrigerator (1230) and an oven (1220), the processor (130) may receive a second captured image corresponding to the refrigerator (1230) and the oven (1220) and store the second captured image in the memory (120).

[0141] FIG. 13 is a block diagram showing a detailed configuration of an electronic device according to one embodiment.

[0142] According to FIG. 13, the electronic device (100') may include a communication interface (110), a memory (120), one or more processors (130), a display (140), a microphone (150), a speaker (160), at least one sensor (170), and a user interface (180). Among the configurations illustrated in FIG. 13, a detailed description of configurations that overlap with those illustrated in FIG. 2 will be omitted.

[0143] The display (140) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc. The processor (130) can control the display (140) to output an output image obtained according to the various embodiments described above. Here, the output image can be a high-resolution image of 4K or 8K or higher.

[0144] Meanwhile, according to another embodiment, the electronic device (100') may not include a display (140). The electronic device (100') may be connected to an external display device and may transmit images or content stored in the electronic device (100') to the external display device. Specifically, the electronic device (100') may transmit images or content to the external display device together with a control signal for controlling the display of the images or content on the external display device.

[0145] Here, the external display device can be connected to the electronic device (100') via a communication interface (110) or an input / output interface (not shown). For example, the electronic device (100') may not include a display, such as a set-top box (STB). In addition, the electronic device (100') may include only a small display that can display only simple information, such as text information. Here, the electronic device (100') can transmit images or contents to the external display device via the communication interface (110) wired or wirelessly, or transmit them to the external display device via the input / output interface (not shown).

[0146] A microphone (150) may refer to a module that acquires sound and converts it into an electrical signal, and may be a condenser microphone, a ribbon microphone, a moving coil microphone, a piezoelectric element microphone, a carbon microphone, or a MEMS (Micro Electro Mechanical System) microphone. In addition, it may be implemented in an omnidirectional, bidirectional, unidirectional, subcardioid, supercardioid, or hypercardioid manner.

[0147] The speaker (160) may be composed of a tweeter for reproducing high-frequency sounds, a midrange for reproducing mid-frequency sounds, a woofer for reproducing low-frequency sounds, a subwoofer for reproducing ultra-low-frequency sounds, an enclosure for controlling resonance, and a crossover network for dividing the frequency of an electric signal input to the speaker into bands.

[0148] The speaker (160) can output an audio signal to the outside of the electronic device (100'). The speaker (160) can output multimedia playback, recording playback, various notification sounds, voice messages, etc. The electronic device (100') may include an audio output device such as the speaker (160), but may also include an output device such as an audio output terminal. In particular, the speaker (160) can provide acquired information, information processed and produced based on acquired information, response results to user voice, operation results, etc. in voice form.

[0149] At least one sensor (170, hereinafter referred to as a sensor) may include a plurality of sensors of various types. The sensor (170) may measure a physical quantity or detect an operating state of an electronic device (100') and convert the measured or detected information into an electrical signal. The sensor (170) may include a camera, and the camera may include a lens that focuses visible light or other optical signals reflected by an object and received onto an image sensor, and an image sensor that may detect visible light or other optical signals. Here, the image sensor may include a 2D pixel array divided into a plurality of pixels. Alternatively, at least one sensor (170) may include a temperature sensor or an infrared sensor.

[0150] The user interface (180) is a configuration for the electronic device (100') to interact with the user. For example, the user interface (180) may include, but is not limited to, at least one of a touch sensor, a motion sensor, a button, a jog dial, a switch, a microphone, or a speaker.

[0151] According to the above-described example, if a home appliance is included in a captured image acquired while the robot is driving, the electronic device (100') can provide a UI for providing functions of the home appliance included in the captured image to the user along with the captured image. Accordingly, the user can freely control multiple home appliances within the driving space through the user terminal, thereby increasing usability. In addition, since the home appliances can be operated using the captured image acquired through the robot, the user can operate the home appliances more intuitively.

[0152] Meanwhile, the methods according to the various embodiments of the present disclosure described above may be implemented in the form of applications that can be installed on existing electronic devices. Alternatively, the methods according to the various embodiments of the present disclosure described above may be performed using a deep learning-based trained neural network (or a deep learned neural network), i.e., a learning network model. Furthermore, the methods according to the various embodiments of the present disclosure described above may be implemented only through a software upgrade or a hardware upgrade of an existing electronic device. Furthermore, the various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device or an external server of the electronic device.

[0153] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The machine is a device that can call commands stored from the storage medium and operate according to the called commands, and may include a display device (e.g., display device (A)) according to the disclosed embodiments. When a command is executed by a processor, the processor can perform a function corresponding to the command directly or by using other components under the control of the processor. The command may include code provided or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0154] Furthermore, according to one embodiment, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily provided in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0155] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0156] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices; communication interface; A memory storing driving map information and at least one instruction including identification information corresponding to each of at least one home appliance existing in the driving space; and By executing at least one instruction, one or more processors are provided to provide UI (User Interface) information using the photographed image received through the communication interface; One or more of the above processors, When a first photographed image of the driving space is received through the above communication interface, the presence of a home appliance in the received first photographed image is checked, If it is confirmed that a home appliance exists in the first captured image, identification information of the confirmed home appliance is obtained based on the driving map information and the first captured image, Based on the above-mentioned acquired identification information, function information provided by the identified home appliance is acquired, Based on the acquired function information, UI information for providing the function of the confirmed home appliance together with the first captured image is identified, An electronic device that transmits the identified UI information to a user terminal through the communication interface.

2. In paragraph 1, One or more of the above processors, When driving information is received from the robot through the above communication interface, the presence of a home appliance within a critical distance from the robot is identified based on the received driving information, When it is identified that a home appliance exists within the above threshold distance, request information for initialization of a camera included in the home appliance within the above threshold distance is transmitted to the home appliance within the above threshold distance through the communication interface, An electronic device that receives a second captured image acquired through the camera from a home appliance existing within the critical distance through the communication interface and transmits the image to the user terminal.

3. In paragraph 2, One or more of the above processors, When the second captured image is received, the second captured image is stored in the memory, An electronic device that transmits the stored second captured image to the user terminal when it is confirmed that a home appliance exists in the first captured image.

4. In paragraph 2, One or more of the above processors, An electronic device that transmits UI information in which the second photographed image overlaps the location of a home appliance existing in the first photographed image to the user terminal through the communication interface.

5. In paragraph 1, One or more of the above processors, When user movement information is received from the user terminal through the communication interface, it is identified whether the user movement size is greater than a threshold value based on the received information, If it is identified as being above the above threshold value, the robot identifies control information to correspond to the received user movement, An electronic device that transmits the identified control information to the robot through the communication interface.

6. In paragraph 2, One or more of the above processors, Transmitting a second captured image corresponding to a first camera among the multiple cameras included in the above home appliance to the user terminal, An electronic device that, when a user input for selecting a second camera among the plurality of cameras is received from the user terminal through the communication interface, transmits a second captured image corresponding to the second camera to the user terminal based on the received user input.

7. In paragraph 6, One or more of the above processors, An electronic device that, when user movement information for selecting the second camera is received, transmits a second captured image corresponding to the second camera to the user terminal based on the received user movement information.

8. In paragraph 6, One or more of the above processors, Store the second captured image corresponding to each of the plurality of cameras included in the above home appliance in the memory, An electronic device that, when a user input for selecting a second camera among the plurality of cameras is received from the user terminal through the communication interface, transmits a second captured image corresponding to the second camera among the stored second captured images to the user terminal.

9. In paragraph 2, One or more of the above processors, An electronic device that, when a user input for selecting one of a plurality of home appliances existing in the driving space is received through the communication interface, transmits a second captured image corresponding to the selected one to the user terminal based on the received user input.

10. In paragraph 9, One or more of the above processors, When a second photographed image corresponding to each of the plurality of home appliances is received from each of the plurality of home appliances, each of the received second photographed images is stored in the memory, An electronic device that, when a user input for selecting one of the plurality of home appliances is received, transmits a second captured image corresponding to one of the stored second captured images to the user terminal.

11. In a method for controlling an electronic device; When a first captured image of a driving space is received, a step of checking whether a home appliance exists in the received first captured image; When it is confirmed that a home appliance exists in the first photographed image, a step of obtaining driving map information including identification information corresponding to each of at least one home appliance existing in the driving space and identification information of the confirmed home appliance based on the first photographed image; A step of obtaining function information provided by the identified home appliance based on the obtained identification information; A step of identifying UI (User Interface) information for providing the function of the confirmed home appliance together with the first photographed image based on the acquired function information; and A control method, comprising: a step of transmitting the identified UI information to a user terminal.

12. In paragraph 11, When driving information is received from a robot, a step of identifying whether a home appliance exists within a threshold distance from the robot based on the received driving information; If it is identified that a home appliance exists within the above threshold distance, a step of transmitting request information for initialization of a camera included in the home appliance within the above threshold distance to the home appliance within the above threshold distance; and A control method further comprising: a step of receiving a second captured image acquired through the camera from a home appliance existing within the critical distance and transmitting the image to the user terminal.

13. In paragraph 12, The above transmitting step is, When the second shooting image is received, a step of storing the second shooting image in memory; and A control method, comprising: a step of transmitting the stored second captured image to the user terminal when it is confirmed that a home appliance exists in the first captured image.

14. In paragraph 12, The step of transmitting the identified UI information to the user terminal is: An electronic device that transmits UI information in which the second photographed image overlaps the location of the home appliance existing in the first photographed image to the user terminal.

15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising: When a first captured image of a driving space is received, a step of checking whether a home appliance exists in the received first captured image; When it is confirmed that a home appliance exists in the first photographed image, a step of obtaining driving map information including identification information corresponding to each of at least one home appliance existing in the driving space and identification information of the confirmed home appliance based on the first photographed image; A step of obtaining function information provided by the identified home appliance based on the obtained identification information; A step of identifying UI (User Interface) information for providing the function of the confirmed home appliance together with the first photographed image based on the acquired function information; and A non-transitory computer-readable recording medium comprising: a step of transmitting the identified UI information to a user terminal.

Images